Conversation
…ng US (LLMO-4258) During LLMO onboarding, prompt generation always sent region: 'US' to DRS regardless of the site's actual market. Now reads the region from the normalized V2 brands table (populated by Brandalf) and falls back to 'US' only when no region is available. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…258) - Add test for listBrands error → falls back to US region - Fix NL test to match on baseSiteId to cover both find() branches Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
This PR will trigger a patch release when merged. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
solaris007
left a comment
There was a problem hiding this comment.
Hey @dzehnder,
Strengths
- Correct architectural direction - reading region from the V2 normalized brands table (the system of record, populated by Brandalf) instead of hardcoding is the right move. (
llmo-onboarding.js:1418-1435) - Solid defensive structure - the region lookup is isolated in its own try/catch with fallback to
'US', so a PostgREST failure or missing data can never break onboarding. (llmo-onboarding.js:1420-1434) - Null-safe access with
context.dataAccess?.services?.postgrestClientand theif (postgrest)guard prevents NPEs when PostgREST is unavailable. (llmo-onboarding.js:1421-1422) - The default stub
listBrands: sinon.stub().resolves([])in the test helper ensures all existing tests pass without changes. (llmo-onboarding.test.js:260) - Good coverage of both the happy path (region from brands table) and error fallback (PostgREST failure).
Issues
Important (Should Fix)
1. Timing: the fix is ineffective during initial onboarding
llmo-onboarding.js:1418-1435
Trace the execution order inside performLlmoOnboarding:
- Step A (~line 1361):
upsertBrandcreates/updates the brand with noregionfield, resulting inregions: []being written to the DB - Step B (~line 1383):
triggerBrandalfOnboardingJobsubmits an async DRS job - Brandalf will eventually populate the brand's regions, but that takes minutes - Step C (this new code, line 1423):
listBrandsreads the brand back - finds the brand from Step A withregion: [], somatchingBrand?.region?.length > 0is false - falls back to'US' - Step D:
submitOnboardingPromptGenerationJobfires withregion: 'US'- same as before the fix
During initial onboarding, Brandalf has not completed by the time prompt generation is submitted. The region will always be empty. The fallback will always trigger. The stated problem (Dutch customers getting English prompts) is not fixed for the initial onboarding path.
On re-onboarding, upsertBrand at Step A also writes regions: [], overwriting whatever Brandalf previously populated - so the fix is also ineffective there.
Options to address:
- (a) Derive region from an already-available signal earlier in the flow (site TLD, org metadata, site locale config) and pass it into the initial
upsertBrandcall - (b) Defer prompt generation until Brandalf completes (event-driven via SNS/callback rather than submitting both jobs sequentially) - cleanest architecturally
- (c) Have DRS resolve region at job processing time from the brands table rather than baking it into job parameters at submission time
- (d) If this fix is intentionally scoped to a narrow re-trigger scenario only, document that in a comment and update the PR description
2. Brand matching priority - baseSiteId should take precedence over name
llmo-onboarding.js:1424-1426
brands.find((b) => b.name === brandName.trim() || b.baseSiteId === site.getId()) uses OR, so a name collision with a different brand in the same org could return that brand's region. The baseSiteId match is a direct link to the site being onboarded and is the more precise signal. Prefer it:
const matchingBrand = brands.find((b) => b.baseSiteId === site.getId())
|| brands.find((b) => b.name === brandName.trim());3. Over-fetching: listBrands joins 5 child tables for a single-field lookup
llmo-onboarding.js:1423
listBrands uses BRAND_SELECT which joins brand_aliases, brand_social_accounts, brand_earned_sources, competitors, and brand_sites. All that data is fetched and mapped through mapDbBrandToV2, then discarded - only region is used. A targeted PostgREST query would be far lighter:
const { data } = await postgrest
.from('brands')
.select('regions')
.eq('organization_id', orgId)
.eq('site_id', siteId)
.eq('status', 'active')
.maybeSingle();4. No validation of region value against known region codes
llmo-onboarding.js:1428
brandRegion is read from the database and passed to DRS without verifying it is a valid region code. The regions reference table already exists (listRegions() in brands-storage.js), but the onboarding code does not consult it. An unexpected value (freeform string, the 'gl' global marker, or a very long value) would be forwarded to DRS and could produce wrong-language prompts. Validate against the reference table or a static allowlist of supported regions, falling back to 'US' with a warning if invalid.
5. Missing test coverage for realistic edge cases
test/controllers/llmo/llmo-onboarding.test.js
Three production-realistic scenarios lack tests:
- Brand matched by name (not baseSiteId) - the current test only exercises the baseSiteId path
- Brand found but region is empty (
region: []) - the most common state during initial onboarding postgrestClientis undefined - environments where PostgREST is not yet configured
These are quick to add given the existing test scaffolding.
Minor (Nice to Have)
6. Log message lacks stack trace
llmo-onboarding.js:1433 - log.warn only logs regionError.message. If the catch fires due to an unexpected TypeError, the stack trace would be essential. Consider log.warn('Failed to read brand region, defaulting to US', regionError).
7. Multi-region ambiguity undocumented
llmo-onboarding.js:1428 - [brandRegion] = matchingBrand.region picks the first element. For a brand with ['NL', 'BE'], this silently picks 'NL'. A comment explaining the heuristic (e.g., "first region is the primary market") would help future readers.
8. Inconsistent variable naming
llmo-onboarding.js:1421 - The new code uses postgrest while every other reference in this file uses postgrestClient.
Recommendations
- Consider whether
upsertBrandat line 1360 should preserve existing regions rather than implicitly resetting them. That would make the region lookup work correctly on re-onboarding. - Long-term, if region is only reliably available after Brandalf completes, consider an event-driven design where Brandalf's completion triggers prompt generation - this removes the temporal coupling entirely.
- If the over-fetching is not addressed in this PR, consider a follow-up to add a targeted
getBrandRegionhelper rather than repurposinglistBrands.
Assessment
Ready to merge? No - needs discussion on scope
The code is safe and will not break anything - the fallback behavior is solid. However, the timing interaction with upsertBrand and the async Brandalf job means the fix does not achieve its stated goal during initial onboarding or re-onboarding, which is the scenario described in the ticket. Before merging, the team should either address the timing gap, or reframe the PR scope to document which scenario this actually fixes. The matching priority, over-fetching, and missing test coverage are secondary but worth addressing.
…constraint (LLMO-4258, LLMO-4294) Remove premature prompt generation from onboarding flow. Brandalf hasn't completed at onboarding time so the seed brand has no region, causing prompts to always use the 'US' fallback. Prompt generation is now deferred to DRS which triggers it after Brandalf completes and syncs brands with correct regions. Also fix type: 'url' -> type: 'base' in 3 brand URL definitions to satisfy the brands table constraint (LLMO-4294). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
alinarublea
left a comment
There was a problem hiding this comment.
Security & Architecture Review — PR #2198
Reviewer perspective: Security specialist / Distinguished engineer
Verdict: Approve
Executive Summary
This PR removes ~75 lines of premature prompt generation logic from the LLMO onboarding flow, deferring it to DRS post-Brandalf completion. It directly addresses the timing bug identified in the previous review (option b). The type: 'url' → type: 'base' fix mentioned in the PR description was already merged to main separately and is no longer in this diff — the PR description is stale on that point.
Security Assessment: No Issues
-
Attack surface reduction — Removing
submitOnboardingPromptGenerationJobandbuildPromptGenerationMetadataeliminates a direct code path from SpaceCat to DRS prompt generation. One fewer outbound RPC from the onboarding hot path means one fewer place where parameter injection, SSRF viabaseUrl, or model override attacks (model: 'gpt-5-nano') could be exploited. Net positive. -
No new trust boundaries introduced — The DRS companion PR (adobe-rnd/llmo-data-retrieval-service#1308) takes over prompt generation, but SpaceCat already trusts DRS for Brandalf jobs. No new cross-service trust relationship is created.
-
No secrets, credentials, or auth changes — Clean removal of business logic only.
-
No input validation regressions — The removed code only consumed internal state (
siteConfig,brandName,baseURL), so removing it doesn't weaken any validation path.
Architecture Assessment
What's good:
- Correct resolution of the timing bug. The previous review identified that
upsertBrandwritesregions: [], then Brandalf is async, so the prompt gen job always fell back to'US'. Deferring to post-Brandalf is the cleanest fix — it removes temporal coupling rather than papering over it. - Clean deletion. Both the function definitions and their sole call site are removed. No dead imports left behind.
buildOnboardingMetadatais still used by the Brandalf metadata builder, so it's correctly retained. - Tests assert the negative.
expect(...secondCall).to.be.nullandexpect(...submitJob).to.not.have.been.calledcorrectly verify the behavior change rather than just deleting assertions.
Observations (non-blocking):
-
Tombstone comment at line 137 —
// submitOnboardingPromptGenerationJob removed — prompt generation is now triggered by DRS...Comments describing what used to be here become noise within a few months. The function is gone;git blame/git logis the right tool for archaeology. Consider replacing with nothing, or at most a one-liner about the current contract. -
Observability gap — The removed code had explicit error handling with Slack notification (
:warning: Failed to start DRS prompt generation (will need manual trigger)). With prompt gen now deferred to DRS, if DRS fails to trigger it after Brandalf completes, there is no visibility from SpaceCat's side. Consider:- Does DRS have equivalent alerting for failed prompt generation?
- Is there a reconciliation mechanism to detect sites that completed onboarding but never got prompts generated?
- Not a blocker for this PR, but worth a follow-up ticket.
-
PR description drift — The description prominently features the LLMO-4294
type: 'url'→type: 'base'fix, but that change is not in this diff (it already landed onmainvia another PR). Consider updating the PR title/description to reflect only what's actually in scope.
Test Coverage Assessment
- Removed 3 prompt-gen-related tests: appropriate since the code path no longer exists
- Updated 2 assertions to verify prompt gen is not called: correct negative testing
- Removed the "DRS prompt generation failure" test (~94 lines): appropriate since the error path no longer exists
- Remaining Brandalf-related error tests are retained and still exercise relevant code paths
No gaps identified.
Summary
| Category | Rating |
|---|---|
| Security | No issues |
| Correctness | Sound — directly addresses the root cause |
| Test coverage | Adequate |
| Code hygiene | Good (minor tombstone comment) |
| Operational risk | Low — follow up on DRS-side observability |
🤖 Generated with Claude Code
implemented requested changes. Alina re-reviewed in your absence
|
🎉 This PR is included in version 1.445.2 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Summary
type: 'url'→type: 'base'constraint violation blocking all V2 brand creationProblem
LLMO-4258: Wrong language prompts
During onboarding, prompt generation fired immediately with hardcoded
region: 'US'— before Brandalf had a chance to populate the brand's actual region. Even with a region lookup, the timing was wrong (reviewer feedback: Brandalf is async and hasn't completed yet).Fix: Remove prompt generation from the onboarding flow entirely. DRS now triggers
prompt_generation_base_urlautomatically after Brandalf completes and syncs brands (with correct regions) to SpaceCat. See companion PR.LLMO-4294: Brand creation constraint violation
The
upsertBrandcall passedtype: 'url'in the brand's URLs array, but thebrand_sitestable only allows'base'or'localized'. This silently failed, meaning NO brand entity was created for ANY V2 onboarding.Fix: Change
type: 'url'→type: 'base'in 3 places (lines 218, 278, 1366).Changes
src/controllers/llmo/llmo-onboarding.js:listBrandsimporttype: 'url'→type: 'base'in 3 placestest/controllers/llmo/llmo-onboarding.test.js:'url'to'base'Companion PR
Test plan
type: 'base'used for brand URLs🤖 Generated with Claude Code