From 0655b230bdbb3ea6c44c7e5848360034c8edd33b Mon Sep 17 00:00:00 2001 From: stacknil Date: Sat, 9 May 2026 13:22:41 +0800 Subject: [PATCH] [codex] Add policy decision explainability guide --- tools/sbom-diff-and-risk/README.md | 22 +++- .../docs/github-actions-consumer-example.md | 6 +- .../docs/policy-decision-explainability.md | 103 ++++++++++++++++++ .../sbom-diff-and-risk/docs/policy-schema.md | 4 + .../pypi-production-publishing-decision.md | 62 ++++++++--- .../sbom-diff-and-risk/docs/report-schema.md | 3 + .../sbom-diff-and-risk/docs/reviewer-brief.md | 3 +- .../docs/reviewer-evidence-pack.md | 54 ++++++--- .../examples/github-actions-consumer.yml | 6 +- 9 files changed, 218 insertions(+), 45 deletions(-) create mode 100644 tools/sbom-diff-and-risk/docs/policy-decision-explainability.md diff --git a/tools/sbom-diff-and-risk/README.md b/tools/sbom-diff-and-risk/README.md index 9a4ea3e..175fb18 100644 --- a/tools/sbom-diff-and-risk/README.md +++ b/tools/sbom-diff-and-risk/README.md @@ -10,11 +10,25 @@ It uses conservative heuristics for change intelligence. By default it does not This project has two different provenance stories: -For a concise reviewer-facing overview, start with [docs/reviewer-brief.md](docs/reviewer-brief.md). For reproducible review evidence and verification commands, use [docs/reviewer-evidence-pack.md](docs/reviewer-evidence-pack.md). For machine-readable JSON output shape, see [docs/report-schema.md](docs/report-schema.md). For CI consumption of summary-only output, see [docs/summary-json-ci-cookbook.md](docs/summary-json-ci-cookbook.md). -For a consumer-facing GitHub Actions example, see [docs/github-actions-consumer-example.md](docs/github-actions-consumer-example.md). +For a concise reviewer-facing overview, start with +[docs/reviewer-brief.md](docs/reviewer-brief.md). For reproducible review +evidence and verification commands, use +[docs/reviewer-evidence-pack.md](docs/reviewer-evidence-pack.md). For +machine-readable JSON output shape, see +[docs/report-schema.md](docs/report-schema.md). For policy decision +explainability fields, see +[docs/policy-decision-explainability.md](docs/policy-decision-explainability.md). +For CI consumption of summary-only output, see +[docs/summary-json-ci-cookbook.md](docs/summary-json-ci-cookbook.md). +For a consumer-facing GitHub Actions example, see +[docs/github-actions-consumer-example.md](docs/github-actions-consumer-example.md). -1. If you want to verify `sbom-diff-and-risk` itself, start with [docs/verification.md](docs/verification.md). -2. If you want to use `sbom-diff-and-risk` to analyze third-party dependency provenance, start with [Dependency provenance analysis](#dependency-provenance-analysis-opt-in) and [Dependency provenance reporting](#dependency-provenance-reporting). +1. If you want to verify `sbom-diff-and-risk` itself, start with + [docs/verification.md](docs/verification.md). +2. If you want to use `sbom-diff-and-risk` to analyze third-party dependency + provenance, start with + [Dependency provenance analysis](#dependency-provenance-analysis-opt-in) + and [Dependency provenance reporting](#dependency-provenance-reporting). ## Scope diff --git a/tools/sbom-diff-and-risk/docs/github-actions-consumer-example.md b/tools/sbom-diff-and-risk/docs/github-actions-consumer-example.md index 8fad5c6..314af1b 100644 --- a/tools/sbom-diff-and-risk/docs/github-actions-consumer-example.md +++ b/tools/sbom-diff-and-risk/docs/github-actions-consumer-example.md @@ -50,15 +50,15 @@ jobs: GH_TOKEN: ${{ github.token }} run: | mkdir -p .tooling/sbom-diff-risk - gh release download v0.6.0 \ + gh release download v0.7.0 \ --repo stacknil/scientific-computing-toolkit \ - --pattern "sbom_diff_and_risk-0.6.0-py3-none-any.whl" \ + --pattern "sbom_diff_and_risk-0.7.0-py3-none-any.whl" \ --dir .tooling/sbom-diff-risk - name: Install sbom-diff-risk run: | python -m pip install \ - .tooling/sbom-diff-risk/sbom_diff_and_risk-0.6.0-py3-none-any.whl + .tooling/sbom-diff-risk/sbom_diff_and_risk-0.7.0-py3-none-any.whl - name: Compare dependency evidence run: | diff --git a/tools/sbom-diff-and-risk/docs/policy-decision-explainability.md b/tools/sbom-diff-and-risk/docs/policy-decision-explainability.md new file mode 100644 index 0000000..c95ccaa --- /dev/null +++ b/tools/sbom-diff-and-risk/docs/policy-decision-explainability.md @@ -0,0 +1,103 @@ +# Policy decision explainability + +This page explains the machine-readable policy decision metadata emitted in +JSON reports. It is intended for reviewers and CI consumers who need to +understand why a local policy rule produced a block, warning, or suppression. + +The fields described here are explainability metadata for local policy +decisions. They are not dependency safety verdicts, CVE results, or proof that a +package is safe or unsafe. + +## Where the fields appear + +Policy decision explanation fields appear only on policy finding objects, such +as: + +- `policy_evaluation.blocking_violations` +- `policy_evaluation.warning_violations` +- `policy_evaluation.suppressed_violations` +- `blocking_findings` +- `warning_findings` +- `suppressed_findings` +- provenance policy impact sections + +Risk findings in `risks` remain the analyzer's local heuristic findings. They +do not receive policy-decision metadata unless policy evaluation maps them into +policy findings. + +## Field contract + +- `decision_reason`: Stable reason code for the policy decision. +- `policy_rule`: Policy rule id that produced the decision. +- `severity_source`: Source of the active severity, such as `block_on`, + `warn_on`, `default_block`, or `default_warn`; `null` when there is no active + severity. +- `matched_threshold`: Configured threshold or allowlist value involved in the + decision, when applicable. +- `observed_value`: Observed local value that was compared to the policy rule, + when applicable. + +The full JSON report shape is documented in [report-schema.md](report-schema.md). +Policy configuration fields and supported rules are documented in +[policy-schema.md](policy-schema.md). + +## Example interpretations + +A policy finding with: + +```json +{ + "decision_reason": "added_package_count_exceeded_threshold", + "policy_rule": "max_added_packages", + "severity_source": "block_on", + "matched_threshold": 0, + "observed_value": 1 +} +``` + +means the local policy compared an observed added-package count of `1` against a +configured threshold of `0`, and the matching rule was active through +`block_on`. + +A policy finding with: + +```json +{ + "decision_reason": "risk_finding_matched_policy_rule", + "policy_rule": "new_package", + "severity_source": "warn_on", + "matched_threshold": null, + "observed_value": "new_package" +} +``` + +means a local heuristic risk finding matched the `new_package` policy rule, and +the matching rule was active through `warn_on`. + +## CI and review usage + +Consumers can use these fields to group policy findings by rule, explain why a +local gate failed, or build small job summaries. For example, a CI step can read +`blocking_findings`, print each `policy_rule` and `decision_reason`, and fail +only because the tool already returned a policy failure exit code. + +Use `summary.policy` for compact counts and status. Use policy finding +explanation fields when a reviewer needs to inspect why the status was +`warn` or `fail`. + +## Compatibility notes + +- The fields are additive JSON metadata for policy findings. +- `summary.policy` is unchanged and remains the compact count/status surface. +- Absence of policy findings means policy evaluation did not produce findings + for that section. +- Absence of policy explanation fields outside policy finding objects is + expected. +- Consumers should treat unrecognized future fields as additive report data. + +## Non-claims + +- The fields do not resolve CVEs. +- The fields do not claim a package is safe or unsafe. +- The fields do not add network behavior. +- The fields do not replace human review of local policy choices. diff --git a/tools/sbom-diff-and-risk/docs/policy-schema.md b/tools/sbom-diff-and-risk/docs/policy-schema.md index 81d74ec..fed0f4e 100644 --- a/tools/sbom-diff-and-risk/docs/policy-schema.md +++ b/tools/sbom-diff-and-risk/docs/policy-schema.md @@ -106,6 +106,10 @@ Version `3` supports every version `1` and `2` rule id plus: - Scorecard evidence remains an auxiliary trust signal. A high score is not proof of safety, and missing Scorecard data is not proof of risk. +For the JSON policy finding explanation fields emitted after policy evaluation, +see +[policy-decision-explainability.md](policy-decision-explainability.md). + ## Version 1 example ```yaml diff --git a/tools/sbom-diff-and-risk/docs/pypi-production-publishing-decision.md b/tools/sbom-diff-and-risk/docs/pypi-production-publishing-decision.md index 123579d..38a4cd5 100644 --- a/tools/sbom-diff-and-risk/docs/pypi-production-publishing-decision.md +++ b/tools/sbom-diff-and-risk/docs/pypi-production-publishing-decision.md @@ -4,14 +4,21 @@ This page records the PR 5 production PyPI gate for `sbom-diff-and-risk`. ## PR 5 decision -Production PyPI publishing is **deferred, but conditionally allowed after the prerequisites below are complete**. In short: production PyPI is currently deferred. - -PR 5 does not add an enabled production publishing workflow and does not publish to production PyPI. The successful TestPyPI Trusted Publishing dry-run proves that the package metadata can render on TestPyPI and that the TestPyPI OIDC path can work, but it is not automatic proof that production PyPI publishing is ready. +Production PyPI publishing is **deferred, but conditionally allowed after the +prerequisites below are complete**. In short: production PyPI is currently +deferred. + +PR 5 does not add an enabled production publishing workflow and does not publish +to production PyPI. The successful TestPyPI Trusted Publishing dry-run proves +that the package metadata can render on TestPyPI and that the TestPyPI OIDC path +can work, but it is not automatic proof that production PyPI publishing is +ready. The production gate is intentionally conservative because: - the production PyPI project does not currently exist under the intended name -- the package metadata has moved to `0.5.0` for the GitHub Release, but production PyPI publishing has not been enabled +- the package metadata has moved beyond the TestPyPI dry-run version, but + production PyPI publishing has not been enabled - the first production upload should be a deliberate release version, not an old dry-run version - the production PyPI pending publisher or trusted publisher has not been configured - the production GitHub environment has not yet been confirmed @@ -26,18 +33,26 @@ As checked on April 26, 2026: - `https://test.pypi.org/pypi/sbom-diff-and-risk/json` returned `200` - TestPyPI reports `sbom-diff-and-risk` version `0.4.1` -This means the intended production project name is not currently visible on production PyPI, while the TestPyPI dry-run project exists. Treat the production name as available for this decision, but re-check immediately before configuration because PyPI can reserve, prohibit, or receive new projects at any time. The first production upload should use a production PyPI pending publisher unless the project is created by a maintainer before the publishing workflow is enabled. +This means the intended production project name is not currently visible on +production PyPI, while the TestPyPI dry-run project exists. Treat the production +name as available for this decision, but re-check immediately before +configuration because PyPI can reserve, prohibit, or receive new projects at any +time. The first production upload should use a production PyPI pending publisher +unless the project is created by a maintainer before the publishing workflow is +enabled. ## First production version Do not publish `0.4.1` to production PyPI casually. -The first production PyPI version should be `0.5.0` only if v0.5 is approved as the first production package release. Otherwise, defer to a later GitHub release tag. +The first production PyPI version should be an explicitly approved current or +future GitHub release tag. Do not backfill an older release casually. For the first production upload: - the GitHub tag should be `v` -- `tools/sbom-diff-and-risk/pyproject.toml` should declare the matching `` +- `tools/sbom-diff-and-risk/pyproject.toml` should declare the matching + `` - the GitHub release and release assets should be available for the same tag - the production PyPI workflow should run from the matching tag ref - the production PyPI upload should use the checked distributions from that workflow run @@ -55,15 +70,24 @@ Configure the production PyPI publisher to match this identity exactly: | Trusted Publisher workflow name field | `sbom-diff-and-risk-pypi.yml` | | GitHub environment | `pypi` | -If production PyPI still has no project for this name, configure a pending publisher for a new project. If the project exists by the time production publishing is implemented, add the trusted publisher to the existing project instead. - -Do not create or document a PyPI API token for this workflow. Production upload should use Trusted Publishing / OIDC only. +If production PyPI still has no project for this name, configure a pending +publisher for a new project. If the project exists by the time production +publishing is implemented, add the trusted publisher to the existing project +instead. + +Do not create or document a PyPI API token for this workflow. Production upload +should use Trusted Publishing / OIDC only. PyPI-side setup should use these paths: -- for a new production project, create a pending publisher on production PyPI for project `sbom-diff-and-risk` with the owner, repository, workflow, and environment values above -- for an existing production project, open that project on production PyPI and add a trusted publisher with the same owner, repository, workflow, and environment values -- leave the environment field as `pypi`; if the PyPI publisher omits the environment, it will not match the future publish job identity +- for a new production project, create a pending publisher on production PyPI + for project `sbom-diff-and-risk` with the owner, repository, workflow, and + environment values above +- for an existing production project, open that project on production PyPI and + add a trusted publisher with the same owner, repository, workflow, and + environment values +- leave the environment field as `pypi`; if the PyPI publisher omits the + environment, it will not match the future publish job identity - do not add a PyPI API token, PyPI password, or GitHub publishing secret as a fallback ## Prerequisites before enabling production publishing @@ -71,7 +95,8 @@ PyPI-side setup should use these paths: Before adding `.github/workflows/sbom-diff-and-risk-pypi.yml`, maintainers should complete all of these checks: - confirm the intended production package name still resolves as expected on production PyPI -- choose the first production version, likely `0.5.0` or a later release tag +- choose the first production version as an explicitly approved current or + future release tag - update `pyproject.toml` to that version - create or verify the matching GitHub tag and release assets - create the GitHub environment named `pypi` @@ -90,7 +115,7 @@ The future production workflow should: - require an explicit boolean input such as `publish_to_pypi` - require a confirmation string such as `publish sbom-diff-and-risk to production PyPI` - require an expected version input and assert that it matches `pyproject.toml` -- require the run ref to be a version tag such as `refs/tags/v0.5.0` +- require the run ref to be a version tag such as `refs/tags/v` - build the wheel and source distribution once - run `python -m twine check dist/*` - upload the checked distributions as a workflow artifact @@ -99,11 +124,14 @@ The future production workflow should: - grant `id-token: write` only to the publish job - avoid production upload on ordinary push or pull request events -The publish step should use `pypa/gh-action-pypi-publish@release/v1` without a `repository-url` override so it targets production PyPI. +The publish step should use `pypa/gh-action-pypi-publish@release/v1` without a +`repository-url` override so it targets production PyPI. ## Provenance boundaries -Production PyPI Trusted Publishing provenance, GitHub workflow artifact attestations, and GitHub Release asset verification answer related but different questions. +Production PyPI Trusted Publishing provenance, GitHub workflow artifact +attestations, and GitHub Release asset verification answer related but +different questions. PyPI Trusted Publishing provenance answers: diff --git a/tools/sbom-diff-and-risk/docs/report-schema.md b/tools/sbom-diff-and-risk/docs/report-schema.md index a8f4526..4061013 100644 --- a/tools/sbom-diff-and-risk/docs/report-schema.md +++ b/tools/sbom-diff-and-risk/docs/report-schema.md @@ -71,6 +71,9 @@ Explanation fields appear only on policy finding objects. Risk findings in policy-decision metadata unless a policy evaluation maps them into policy findings. +For reviewer-facing examples and interpretation guidance, see +[policy-decision-explainability.md](policy-decision-explainability.md). + ## Summary contract `summary` is the stable, compact entry point for automation that needs counts diff --git a/tools/sbom-diff-and-risk/docs/reviewer-brief.md b/tools/sbom-diff-and-risk/docs/reviewer-brief.md index 88062ee..78c8de8 100644 --- a/tools/sbom-diff-and-risk/docs/reviewer-brief.md +++ b/tools/sbom-diff-and-risk/docs/reviewer-brief.md @@ -4,7 +4,7 @@ `sbom-diff-and-risk` is a local CLI for comparing two SBOMs or dependency manifests and producing deterministic review artifacts: JSON, Markdown, and SARIF. It is built for conservative supply-chain review, not for vulnerability scanning or package reputation scoring. -Current released version: `v0.6.0`. +Current released version: `v0.7.0`. ## Why this project matters @@ -28,6 +28,7 @@ Dependency review often needs evidence that is stable enough for code review, CI | What does the tool do? | `README.md`, examples, tests, and generated sample reports. | | How can a reviewer reproduce the core evidence? | [reviewer-evidence-pack.md](reviewer-evidence-pack.md) for demo, release, TestPyPI, and SARIF verification paths. | | What is the stable JSON shape? | [report-schema.md](report-schema.md) documents the machine-readable report structure and `summary` contract. | +| How are policy findings explained? | [policy-decision-explainability.md](policy-decision-explainability.md) documents the policy decision metadata in JSON reports. | | Are default runs offline? | CLI docs, tests for no-enrichment behavior, and explicit enrichment flags. | | Can code scanning consume the output? | `docs/github-code-scanning.md` and `examples/sample-sarif.sarif`. | | Can the tool's own artifacts be verified? | `docs/self-provenance.md` for workflow artifact attestations. | diff --git a/tools/sbom-diff-and-risk/docs/reviewer-evidence-pack.md b/tools/sbom-diff-and-risk/docs/reviewer-evidence-pack.md index 3a9297d..ec3d0b9 100644 --- a/tools/sbom-diff-and-risk/docs/reviewer-evidence-pack.md +++ b/tools/sbom-diff-and-risk/docs/reviewer-evidence-pack.md @@ -6,7 +6,7 @@ This page is a reproducible evidence checklist for reviewing `sbom-diff-and-risk `sbom-diff-and-risk` is a local-first deterministic CLI for comparing SBOMs and dependency manifests. It is designed to produce stable review evidence for dependency changes. -Current released version: `v0.6.0`. +Current released version: `v0.7.0`. Core identity: @@ -53,7 +53,8 @@ Compare-Object (Get-Content examples/sample-report.md) (Get-Content outputs/repo No differences means the sample path reproduced the committed example output. -`examples/sample-summary.json` is the summary-only artifact for the same run and is expected to match `examples/sample-report.json`'s `summary` object. +`examples/sample-summary.json` is the summary-only artifact for the same run +and is expected to match `examples/sample-report.json`'s `summary` object. Generate the strict-policy SARIF sample: @@ -73,22 +74,34 @@ Compare-Object (Get-Content examples/sample-sarif.sarif) (Get-Content outputs/re The SARIF sample is intentionally conservative. It covers selected high-signal findings and explicit policy violations, not every enrichment fact. -For consumers of the JSON output, see [report-schema.md](report-schema.md). It documents the stable `summary` contract, including conditional `summary.policy` and `summary.enrichment` fields. +For consumers of the JSON output, see [report-schema.md](report-schema.md). It +documents the stable `summary` contract, including conditional +`summary.policy` and `summary.enrichment` fields. -For CI dashboard, job-summary, and local-threshold examples that consume `outputs/summary.json`, see [summary-json-ci-cookbook.md](summary-json-ci-cookbook.md). +For policy finding interpretation, see +[policy-decision-explainability.md](policy-decision-explainability.md). It +documents the policy decision metadata used to explain local blocks, warnings, +and suppressions. + +For CI dashboard, job-summary, and local-threshold examples that consume +`outputs/summary.json`, see +[summary-json-ci-cookbook.md](summary-json-ci-cookbook.md). ## Release Verification Path -Start with the GitHub Release for the version under review. For `v0.6.0`, inspect the release and assets: +Start with the GitHub Release for the version under review. For `v0.7.0`, +inspect the release and assets: ```powershell -gh release view v0.6.0 --repo stacknil/scientific-computing-toolkit --json tagName,name,isDraft,isPrerelease,assets,url +gh release view v0.7.0 ` + --repo stacknil/scientific-computing-toolkit ` + --json tagName,name,isDraft,isPrerelease,assets,url ``` Expected release assets: -- `sbom_diff_and_risk-0.6.0-py3-none-any.whl` -- `sbom_diff_and_risk-0.6.0.tar.gz` +- `sbom_diff_and_risk-0.7.0-py3-none-any.whl` +- `sbom_diff_and_risk-0.7.0.tar.gz` - `sbom-diff-and-risk-SHA256SUMS.txt` The checksum manifest checks local downloaded distribution bytes before or alongside provenance verification: @@ -110,33 +123,40 @@ Get-Content .\sbom-diff-and-risk-SHA256SUMS.txt | ForEach-Object { } ``` -Checksum verification confirms local byte integrity against the release manifest; it does not replace workflow artifact attestations or immutable-release verification. The attestation subject remains the built wheel and source distribution. +Checksum verification confirms local byte integrity against the release +manifest; it does not replace workflow artifact attestations or +immutable-release verification. The attestation subject remains the built wheel +and source distribution. -For workflow-built artifacts downloaded from a trusted workflow run, verify artifact attestations with the signer workflow: +For workflow-built artifacts downloaded from a trusted workflow run, verify +artifact attestations with the signer workflow: ```powershell -gh attestation verify path/to/sbom_diff_and_risk-0.6.0-py3-none-any.whl ` +gh attestation verify path/to/sbom_diff_and_risk-0.7.0-py3-none-any.whl ` --repo stacknil/scientific-computing-toolkit ` --signer-workflow stacknil/scientific-computing-toolkit/.github/workflows/sbom-diff-and-risk-ci.yml ``` ```powershell -gh attestation verify path/to/sbom_diff_and_risk-0.6.0.tar.gz ` +gh attestation verify path/to/sbom_diff_and_risk-0.7.0.tar.gz ` --repo stacknil/scientific-computing-toolkit ` --signer-workflow stacknil/scientific-computing-toolkit/.github/workflows/sbom-diff-and-risk-ci.yml ``` -`gh release verify` and `gh release verify-asset` are conditional on immutable releases. Use them only when the repository release is immutable and GitHub has generated release attestations: +`gh release verify` and `gh release verify-asset` are conditional on immutable +releases. Use them only when the repository release is immutable and GitHub has +generated release attestations: ```powershell -gh release view v0.6.0 --repo stacknil/scientific-computing-toolkit --json isImmutable,assets,url +gh release view v0.7.0 --repo stacknil/scientific-computing-toolkit --json isImmutable,assets,url ``` -If `isImmutable` is true, release verification can check the release record and downloaded release assets: +If `isImmutable` is true, release verification can check the release record and +downloaded release assets: ```powershell -gh release verify v0.6.0 --repo stacknil/scientific-computing-toolkit -gh release verify-asset v0.6.0 path/to/sbom_diff_and_risk-0.6.0-py3-none-any.whl --repo stacknil/scientific-computing-toolkit +gh release verify v0.7.0 --repo stacknil/scientific-computing-toolkit +gh release verify-asset v0.7.0 path/to/sbom_diff_and_risk-0.7.0-py3-none-any.whl --repo stacknil/scientific-computing-toolkit ``` If `isImmutable` is false, use the workflow artifact attestation path as the primary artifact verification story. diff --git a/tools/sbom-diff-and-risk/examples/github-actions-consumer.yml b/tools/sbom-diff-and-risk/examples/github-actions-consumer.yml index a8c550a..383612e 100644 --- a/tools/sbom-diff-and-risk/examples/github-actions-consumer.yml +++ b/tools/sbom-diff-and-risk/examples/github-actions-consumer.yml @@ -31,15 +31,15 @@ jobs: GH_TOKEN: ${{ github.token }} run: | mkdir -p .tooling/sbom-diff-risk - gh release download v0.6.0 \ + gh release download v0.7.0 \ --repo stacknil/scientific-computing-toolkit \ - --pattern "sbom_diff_and_risk-0.6.0-py3-none-any.whl" \ + --pattern "sbom_diff_and_risk-0.7.0-py3-none-any.whl" \ --dir .tooling/sbom-diff-risk - name: Install sbom-diff-risk run: | python -m pip install \ - .tooling/sbom-diff-risk/sbom_diff_and_risk-0.6.0-py3-none-any.whl + .tooling/sbom-diff-risk/sbom_diff_and_risk-0.7.0-py3-none-any.whl - name: Compare dependency evidence run: |