feat(compliance): add vendor_metric optimization-goal storyboard coverage#4939
feat(compliance): add vendor_metric optimization-goal storyboard coverage#4939bokelley wants to merge 1 commit into
Conversation
…rage (#4933) Adds storyboard media_buy_seller/vendor_metric_optimization_flow exercising the 3.1 vendor_metric goal contract: positive acceptance when all preconditions are met, plus two negative paths for capability mismatch and reporting-coherence failure. Updates training-agent publishers.ts + product-factory.ts to surface vendor_metric_optimization on products. https://claude.ai/code/session_01VtKu8sAUEWAHRPMiZJBgHw
|
Taking this out of draft. This looks like merge-track compliance coverage rather than a parking-lot draft; leaving it for normal review. |
|
Acknowledged — noted that you've moved this to normal review. No action from triage; this PR stands as documented. Generated by Claude Code |
|
Closing as superseded. The substantive changes from this PR are already on main: vendor_metric_optimization_flow exists in static compliance source and released bundles, and the training-agent product factory/publisher/type support is present. This branch now conflicts with main and no longer needs to merge separately. |
There was a problem hiding this comment.
LGTM. Compliance-gap closure on the 3.1 vendor_metric contract, additive only, two negative paths exercising the right MUST preconditions.
Things I checked
- Schema conformance. Storyboard wire shapes match
static/schemas/source/core/optimization-goal.json:183-225(goal structure) andstatic/schemas/source/core/vendor-metric-optimization.json:1-40(capability declaration).target.kind: "threshold_rate"+ numericvalue,committed_metrics[{scope: "vendor", ...}], and the(vendor.domain, metric_id)key threading across discovery / capability / commitment / optimization are all correct. - Error codes.
ad-tech-protocol-expertconfirmedINVALID_REQUESTis the right code for both negative paths. Path B's distinction (reporting-coherence failure on the goal vs.committed_metricsover-proposal →TERMS_REJECTEDperpackage-request.json:107) is sound — that prose is load-bearing. - Discovery SHOULD. Documented in the narrative, not asserted as a failing path. Right call per the explicit SHOULD-level language in
optimization-goal.json:183. - Symmetry.
attentionvendor.example/attention_scorethreaded throughreporting_capabilities.vendor_metrics,vendor_metric_optimization.supported_metrics, the goal, and the package's vendor-scopecommitted_metrics. Same(vendor, metric_id)on all four surfaces. - Training-agent TS.
vendorMetricOptimizationshape (server/src/training-agent/types.ts:98-105) matches the schema. Spread inproduct-factory.ts:565follows themetric_optimizationprecedent on the line directly above it.publishers.ts:33-43adds the symmetric reporting + optimization entries forpinnacle_news— that pairing is required for the positive path's reporting-coherence precondition, not duplication. - Changeset. Empty-frontmatter form matches repo convention for non-package-bump changes (precedent:
.changeset/2383-test-vector-index-scope-update.md). code-reviewer: clean.ad-tech-protocol-expert: sound-with-caveats, no wire-shape drift.
Follow-ups (non-blocking — file as issues)
- Path A confounding variable.
reject_unsupported_capabilitysubmits anemissions_scoregoal and acommitted_metricsentry for the same metric.emissions_scoreis also absent from the product's reportingvendor_metrics, so a conformant seller could legitimately emitTERMS_REJECTEDon thecommitted_metricsover-proposal (perpackage-request.json:107) before ever reaching the optimization-goal capability check — and would correctly fail this storyboard. Either dropcommitted_metricsfrom path A, or seed a publisher case where a metric is invendor_metricsbut not insupported_metricsso the capability check is isolated. - Unexercised MUST:
target.kindnot insupported_targets. Already acknowledged in the PR body. MUST-reject pervendor-metric-optimization.json. File and link. - Schema silence on error codes. Neither
optimization-goal.json:183norvendor-metric-optimization.jsonpins the rejection code toINVALID_REQUEST. The storyboard's choice is defensible and consistent with how other capability-table checks are framed, but the storyboard would rest on firmer ground if the schema named the code in the MUST-reject prose.
Minor nits (non-blocking)
supported_targetstype drift risk.server/src/training-agent/types.ts:99hard-codesArray<'cost_per' | 'threshold_rate'>. Derive fromNonNullable<Product['vendor_metric_optimization']>['supported_metrics'][number]['supported_targets']once the SDK exposes it.- Path A narrative. A one-line note that the duplicated
committed_metricsis intentional (to isolate the capability check) would help future readers — though the Follow-up above would obviate the comment if taken.
Approving.
Closes #4933
Adds storyboard coverage for the 3.1
vendor_metricoptimization goal contract, which was missing from the compliance suite. A seller could previously pass the existingvendor_metric_accountabilitystoryboard without proving it enforces the 3.1 optimization-goal acceptance/rejection rules.Changes
New storyboard
static/compliance/source/protocols/media-buy/scenarios/vendor_metric_optimization_flow.yaml:kind: "vendor_metric"goal accepted when(vendor, metric_id)is invendor_metric_optimization.supported_metrics[],target.kindis insupported_targets, and package has a matching vendor-scopecommitted_metrics[]entryINVALID_REQUEST) when(vendor, metric_id)is not in the product'ssupported_metricsINVALID_REQUEST) when optimization goal is capability-valid but the package has no vendor-scopecommitted_metricsentry (reporting-coherence precondition failure — distinct from aTERMS_REJECTEDcommitted_metricsover-proposal)metric_idin vendor's catalog) documented in narrative; not asserted as failing since the spec is SHOULD-level in 3.1Training-agent fixtures (
server/src/training-agent/):types.ts: addedvendorMetricOptimizationoptional field toPublisherProfilepublishers.ts: addedattention_scoretopinnacle_newsvendorMetricsandvendorMetricOptimization.supported_metricsproduct-factory.ts: spreadvendor_metric_optimizationonto the product build alongside the existingmetric_optimizationpatternNon-breaking justification: entirely additive — new storyboard file, new optional TypeScript field, new publisher capability entry; no existing storyboard, schema, or publisher profile is modified.
Pre-PR review:
key:toname:incontext_outputs); nit noted (TypeScript type could be tighter, low risk)INVALID_REQUESTconfirmed correct for both negative paths; path B narrative clarified to distinguish reporting-coherence precondition failure fromTERMS_REJECTEDcommitted_metricsover-proposal; omittingrequires_capabilityis correct sincevendor_metric_optimizationis product-level onlyNits (not fixed): one unexercised MUST-level path (
target.kindnot insupported_targets) surfaced by review — flagged for follow-up issue rather than scope-expanding this PR.Session: https://claude.ai/code/session_01VtKu8sAUEWAHRPMiZJBgHw
Generated by Claude Code