Umbrella: Harden untrusted PR CI + Codex agent security model

[lmeyerov]

Goal

Harden CI + agent workflows for untrusted third-party PRs against prompt-injection and supply-chain attacks, while preserving contributor velocity through staged rollout.

Threat Model

• Confused deputy in CI: privileged automation processes attacker-controlled PR inputs.
• Privileged trigger misuse (pull_request_target/workflow_run) when untrusted code is checked out/executed.
• Supply-chain compromise via mutable action tags/dependency ingestion.
• Self-hosted runner persistence risk.
• AI prompt-injection (direct and indirect) with excessive agent permissions.

Trust Tiers

• Tier 0: untrusted scan (pull_request, no secrets, read-only token).
• Tier 1: maintainer metadata ops (optional; currently out of scope unless a concrete automation need appears).
• Tier 2: promoted privileged CI (maintainer-approved, isolated runner, short-lived creds).

Child Issues

• 1) PR-B: Explicit workflow/job token permissions + credential handling

  • Add explicit least-privilege permissions: in CI workflows
  • Elevate only per job where required
  • Add persist-credentials: false on untrusted checkouts

• 2) PR-A: Repo/org Actions policy hardening (parallel)

  • Status: merged prep PR: PR-A prep: Explicit permissions in publish-pypi workflow #1143 (2026-04-18)
  • Completed settings (2026-04-17): default GITHUB_TOKEN read-only, disable Actions PR approval, require approval for all external contributors
  • Merged prep: publish-pypi.yml explicit contents: read + id-token: write

• 3) PR-D: GPU/self-hosted runner isolation + promotion gate (temporary lockdown)

  • Status: merged: PR-D: Temporarily disable ci-gpu gpu_public lane with fail-fast guard #1141 (2026-04-18)
  • ci-gpu.yml now fails fast and keeps gpu_public execution disabled pending re-enable criteria

• 4) PR-C: Re-enable promoted GPU lane in-place (no workflow split)

  • Status: deferred (no GPU runner path planned; revisit only if GPU/self-hosted strategy returns)
  • Keep using existing ci-gpu.yml; do not add separate maintainer-ops / promoted workflow files unless needed
  • Re-enable execution only with explicit maintainer promotion gate
  • Verify/document runner ephemerality/reset guarantees before re-enabling any self-hosted execution

• 5) PR-E: Actions/dependency supply-chain hardening (scheduled scope)

  • Status: merged: PR-E: Add actionlint + zizmor workflow security scanning #1144 (2026-04-19)
  • Added workflow scanner gate (workflow-security.yml) with actionlint + zizmor
  • actionlint now selects latest stable release older than 6 days; zizmor uses PIP_EXCLUDE_NEWER: 6 days
  • Add workflow security scanning (actionlint, zizmor, optionally Scorecard)
  • Keep dependency review required for manifest/lockfile changes
  • Tighten/maintain Actions allowlist and exception tracking

• 6) PR-F: Trusted release hardening

  • Status: completed (2026-04-25)
  • Sub-PRs:
    • PR-F1: Trusted publish gate for PyPI workflow #1150 (PR-F1): trusted publish gate in publish-pypi.yml with environment approval + trusted-ref verification
    • PR-F1b: Align publish docs and legacy PyPI links #1157 (PR-F1b): release docs/metadata alignment (safe tag push + pypi.org links)
    • PR-F2: Tighten trusted-publish OIDC context and policy docs #1163 (PR-F2): OIDC claim/policy tightening for publish/deploy paths
    • PR-F3: Add SBOM and provenance baseline to publish pipeline #1184 (PR-F3): SBOM + provenance attestations in release path
    • PR-F4: Add release verification docs for attestations + SBOM evidence #1206 (PR-F4): consumer verification docs (PyPI attestations + SBOM retrieval)
  • Follow-up: docs: add a Security section (rehome release verification guide) #1208 (rehome verification guide into a dedicated docs Security section)

• 7) PR-G: Provider safelist (agent-credential audit descoped)

  • Status: completed 2026-04-27. Both in-scope workstreams (7a + 7b) landed; 7c descoped (rationale below).
  • Scoping decision: do not SHA-pin Tier 2 (pypa/*) or Tier 3 (actions/*, github/*). Trusting these orgs' account security in exchange for fast CVE patching is the better tradeoff for this team size. Historical attack pattern is on smaller third-party orgs (tj-actions, reviewdog), which is what we deprecate via Tier 1.
  • 7a) Deprecate Tier 1 untrusted third-party actions + add provider safelist to CI — merged: PR-G 7a: deprecate Tier 1 third-party actions + zizmor provider safelist #1221 (2026-04-27), tracked in PR-G 7a: deprecate Tier 1 third-party Actions + SHA-pin remaining #1215
    • Threat: mutable-tag supply-chain compromise from third-party orgs with weaker account security
    • Tier 1 deprecations: styfle/cancel-workflow-action → native concurrency: block; dorny/paths-filter → inline git diff shell job
    • Provider safelist: configured .github/zizmor.yml unpinned-uses rule with allowlist for actions/*, github/*, pypa/*. Anything outside must be SHA-pinned and explicitly listed.
    • Workflow-security gate threshold lowered from --min-severity high to medium so policy violations actually fail CI.
    • Bonus: closed 5 PR-B credential-persistence gaps (persist-credentials: false on LFS / fetch-depth checkouts). Bonus: fixed a silent fail-closed BLOCKER in the changes job (force-pushed branch with orphaned event.before, or rebased PR base — were silently skipping all gated tests). Both surfaced via the multi-wave /review skill on the PR; see plans/1130-pr-g-7a/ audit trail.
  • 7b) Enable repo Actions allowlist matching the safelist — applied 2026-04-27
    • GitHub UI: Settings → Actions → General → Allow specified actions and reusable workflows
    • Active allowlist: "Allow actions created by GitHub" (covers actions/* and github/*) + explicit github/*, pypa/*. Stale dorny/paths-filter@v3 and styfle/cancel-workflow-action@0.11.0 entries removed (deprecated by 7a).
    • "Require actions to be pinned to a full-length commit SHA" deliberately NOT enabled — would force SHA pinning of trusted orgs, contrary to the scoping decision above.
    • Layered defense now in place with 7a: repo setting blocks at runner level; zizmor unpinned-uses catches in PRs before merge.
  • 7c) Agent-credential exposure audit — descoped 2026-04-27. Different threat surface from CI hardening (local agent runtimes ≠ untrusted-PR-in-CI). Without a concrete trigger (new agent integration, incident, compliance ask), the deliverable would be a write-for-writing's-sake doc nobody references. Revisit only when one of those triggers appears.

Descoped (from prior PR-G)

The original PR-G bundled three items that did not pass a "what attack does this stop?" test. Removed from active scope:

• Incident response playbook — useful only if tied to tabletop practice; for a small team the runbook is not what you reach for during a real incident. Revisit if/when a compliance driver appears.
• Measurable CI security metrics — only valuable when a non-zero signal triggers an action; without an owner and review cadence it is a dashboard for show.
• Reproducibility targets for critical artifacts — large engineering cost (deterministic builds across Python versions, sdist contents, timestamps) for marginal additional protection over the trusted-publisher attestation chain delivered by PR-F3. Revisit only if the threat model shifts (e.g., distrust of GitHub-hosted publish identity).
• Agent-credential exposure audit (originally PR-G 7c) — different threat surface from CI hardening (local agent runtimes ≠ untrusted-PR-in-CI). Without a concrete trigger (new agent integration, incident, compliance driver), the deliverable would be a write-for-writing's-sake doc nobody references. Revisit only when one of those triggers appears.

Pressure-Tested Rollout Order

1. PR-B first (fastest code-level blast-radius reduction).
2. PR-A in parallel (policy-level controls, may require admin coordination).
3. PR-D early (highest practical risk area: self-hosted/GPU path).
4. PR-E completed (scheduled supply-chain controls baseline landed).
5. PR-C deferred unless GPU/self-hosted strategy is reintroduced.
6. PR-F completed (PR-F1: Trusted publish gate for PyPI workflow #1150, PR-F1b: Align publish docs and legacy PyPI links #1157, PR-F2: Tighten trusted-publish OIDC context and policy docs #1163, PR-F3: Add SBOM and provenance baseline to publish pipeline #1184, PR-F4: Add release verification docs for attestations + SBOM evidence #1206 merged).
7. PR-G completed (2026-04-27): 7a merged (PR-G 7a: deprecate Tier 1 third-party actions + zizmor provider safelist #1221), 7b applied (repo Actions allowlist). 7c descoped — no concrete trigger.

Go/No-Go Gates

• Gate 1 (completed): explicit least-privilege permissions in ci.yml and ci-gpu.yml, no unsafe credential persistence.
• Gate 2 (deferred): only applicable if PR-C is reactivated; requires runner ephemerality/reset guarantees.
• Gate 3 (completed): scanner checks required and dependency-review gate active.

Milestone Checkpoints

1. 48-hour checkpoint: PR-B merged; temporary GPU lockdown in place.
2. 1-week checkpoint: PR-A and PR-D merged; no untrusted PR code on privileged/self-hosted paths.
3. Completed: PR-E merged with scanner/dependency gating baseline established.
4. Completed: PR-F phase-1 merged (#1150, #1157).
5. Completed: PR-F fully merged (#1163, #1184, #1206) — trusted release hardening done.
6. Completed: PR-G 7a merged (PR-G 7a: deprecate Tier 1 third-party actions + zizmor provider safelist #1221, 2026-04-27) — Tier 1 deprecated, zizmor unpinned-uses safelist live at medium gate, 5 PR-B credential gaps closed, silent fail-closed BLOCKER in changes job fixed.
7. Completed: PR-G 7b applied 2026-04-27 (repo Actions allowlist live; layered defense with 7a's in-PR zizmor gate).
8. Completed: PR-G 7c descoped 2026-04-27 — agent-credential audit doesn't pass the "what attack does this stop, with this team's resources" test without a concrete trigger.

All in-scope work has landed. PR-C (re-enable GPU lane) remains deferred pending a GPU strategy. The umbrella can close, or stay open as a maintenance bookmark for the deferred item.

Acceptance Criteria

• Untrusted PR code runs only in least-privilege contexts with no secrets.
• Any workflow with write/secrets access does not execute untrusted PR code directly.
• Privileged CI requires explicit maintainer promotion/approval.
• Self-hosted/GPU usage is gated and isolated.
• Action usage, token scopes, and exceptions are auditable.

Current Files In Scope

• .github/workflows/ci.yml
• .github/workflows/ci-gpu.yml
• .github/workflows/codeql-analysis.yml
• .github/workflows/publish-pypi.yml
• .github/workflows/workflow-security.yml
• .github/actionlint.yaml
• .github/zizmor.yml

[lmeyerov]

Correction to prior comment (shell quoting issue). PR-B local changes completed:

• Added explicit top-level least-privilege permissions to .github/workflows/ci.yml and .github/workflows/ci-gpu.yml (contents: read).
• Added persist-credentials: false to all actions/checkout steps in those workflows.
• Added targeted job elevation in .github/workflows/ci-gpu.yml for cancel step (actions: write).

Next step: open PR with these changes and validate CI behavior.

[lmeyerov]

Correction to prior comment (shell quoting issue).

PR-B opened: #1138

Scope:

• explicit workflow permissions
• persist-credentials: false on CI workflow checkouts
• scoped actions: write only for GPU cancel job

[lmeyerov]

Quality/security audit pass on umbrella issue completed after PR-B opened.

Actions taken:

• Removed two malformed comments caused by shell-quoting artifacts.
• Confirmed rollout order is pressure-tested and starts with PR-B.
• Confirmed gates/checkpoints align with threat model (least privilege, runner isolation, no privileged untrusted execution).
• Confirmed scope remains comprehensive across immediate/short/medium/long-term controls.

Current status:

• PR-B is active in PR PR-B: Explicit workflow permissions + disable checkout credential persistence #1138.
• Added changelog entry for PR-B CI credential/token hardening.

[lmeyerov]

PR-B update: #1138 now includes changelog entry, and all current checks are green (with expected docs-only skips for matrix-heavy lanes).

[lmeyerov]

PR-B is now merged: #1138 (merge commit ).\n\nNext per plan:\n1. PR-A (admin/settings lane) in parallel\n2. PR-D (GPU/self-hosted isolation + promotion gate) as next code PR\n3. PR-C trust-tier split after PR-D guardrails are in place

[lmeyerov]

Correction to prior comment (shell quoting issue).

PR-B is now merged: #1138 (merge commit 3aa8ff67).

Next per plan:

1. PR-A (admin/settings lane) in parallel
2. PR-D (GPU/self-hosted isolation + promotion gate) as next code PR
3. PR-C trust-tier split after PR-D guardrails are in place

[lmeyerov]

PR-D opened: #1141

Scope in this PR:

• Temporarily disable ci-gpu execution on gpu_public while runner availability and PR-D hardening are pending
• Add fail-fast guard job so GPU-triggered runs fail with clear re-enable guidance (issue Umbrella: Harden untrusted PR CI + Codex agent security model #1130)
• Add changelog note documenting temporary GPU lockdown

Re-enable target remains issue #1130 acceptance criteria (self-hosted/GPU isolation + explicit promotion gate).