feat(jira): improve mapping setup UX

[peteroden]

Goal

Create a deployment-time setup experience where the user edits one friendly YAML file, uses a helper command to validate/render it, and deploys without needing GitLab project IDs, clone URLs, or unsafe secret handling.

User experience we want

First deploy should feel obvious

• User copies one short YAML example.
• They change a Jira project key and GitLab repo path.
• They run validate and optionally show.
• They do not look up project IDs or clone URLs.
• If the repo can use the default PAT, they do nothing extra.
• Startup clearly reports the accepted mapping.

Adding another team should feel like repeating the same step

• User adds another YAML binding using the same pattern.
• If that team needs a different PAT, they add a friendly credential_ref.
• They run the same helper commands again.
• They do not change formats or embed secrets.

Troubleshooting should feel safe

• Helper validation errors point to a specific YAML binding.
• Auth errors mention the missing or unknown credential_ref, not a raw token.
• Startup summaries show which repo uses the default credential vs a named credential ref.

Concrete v1 decisions

• Use Jira project keys like PROJ.
• Use GitLab repo paths like group/repo.
• v1 runtime behavior stays one Jira project -> one repo -> one MR.
• No backward compatibility or migration layer will be built.
• No live editing of running services.
• No admin page in this phase.
• ACA/Terraform is the primary deployment path to optimize for; Helm should follow the same model.
• The YAML file plus helper command is the user-facing workflow.

Recommended user-facing model

YAML source of truth

defaults:
  target_branch: main
  credential_ref: default

bindings:
  - jira_project: PROJ
    repo: group/service-a

  - jira_project: OPS
    repo: group/platform-tools
    target_branch: develop
    credential_ref: platform_team

Helper command responsibilities

The helper command is the main deployment-time UX layer.

Recommended v1 commands:

• validate PATH: schema checks, duplicate Jira projects, required credential aliases, readable errors
• show PATH: human-readable summary of bindings and credential usage
• render-json PATH: emit the runtime JSON for JIRA_PROJECT_MAP

Optional later commands:

• render-terraform
• render-helm

Runtime input

The deployment layer uses helper output to populate the runtime JSON env var consumed by the app at startup.

Secret and credential model

• Default token remains GITLAB_TOKEN.
• A binding may optionally specify a friendly credential_ref.
• Raw PATs never appear inside the YAML mapping file.
• For ACA/Terraform, inject the same credential set into both controller and task-runner.
• Named repo credentials should use an alias-based env convention such as GITLAB_TOKEN__PLATFORM_TEAM.
• The helper command should report which credential aliases are required by the YAML file.
• The runtime resolves credential_ref to the matching injected secret at startup and in task-runner startup.

Current code reality to account for

• Settings() and TaskRunnerSettings currently load from environment variables only, once at process startup.
• JIRA_PROJECT_MAP is currently a raw JSON env-var string parsed in main.py.
• The app currently assumes a single global settings.gitlab_token across repo resolution, clone, push, MR creation, webhook/MR review, /copilot comment handling, and GitLab polling.
• ACA/Terraform and Helm currently inject one global GitLab token into controller and runner.

Architecture in service of the experience

Shared mapping core

Build one shared mapping core used by:

• helper command validation/rendering
• startup parsing and validation
• repo resolution
• tests

This core should define:

• YAML and JSON binding models
• defaults handling
• resolved runtime models
• validation rules for duplicate Jira projects, missing repos, missing credential refs, and invalid repo paths

Helper-first deployment flow

• User edits YAML.
• Helper validates and renders JSON.
• Deployment consumes helper output.
• App starts and re-validates the rendered config.

Credential registry

Add a startup-loaded credential registry that:

• reads the default GITLAB_TOKEN
• reads named GITLAB_TOKEN__* aliases
• resolves each binding to the correct token
• never logs raw secrets

Resolved project registry

At controller startup:

• parse rendered JSON from the helper-generated YAML model
• resolve repo path -> GitLab project ID using the correct token
• derive or supply clone URL
• build a registry that can answer:
  • Jira project -> resolved repo context
  • GitLab project ID -> resolved repo context

This registry becomes the source of truth for Jira flows and GitLab-side flows.

Credential-aware runtime flows

Replace single-token assumptions across the app so the selected mapping credential is used consistently for:

• Jira coding orchestration
• task-runner clone and push flow
• webhook/MR review orchestration
• /copilot comment handling
• GitLab polling

For GitLab polling, group projects by credential reference and use one client per credential group.

Implementation Plan (Updated)

Problem

The current JIRA_PROJECT_MAP workflow requires users to manually construct JSON with GitLab project IDs and clone URLs, and any mapping change requires a full redeployment. Issue #270 defines a friendly YAML-first workflow with helper commands, credential aliases, and a resolved project registry.

This plan finalizes the implementation breakdown from issue #270 with two additions:

1. Graceful shutdown — controller drains in-flight orchestration before restart.
2. Hot-reload — mapping changes applied without restart via a reload endpoint; _processed_issues cleared on reload (Jira status transitions prevent true duplicate processing).

Approach

Split into 5 PRs (issue #270 proposed 3; we add graceful shutdown + hot-reload as a separate PR and split docs/E2E into its own PR). Each PR is ≤200 diff lines, standalone, and fully tested.

Gate after PR 1: User reviews and confirms the YAML/JSON schema before any runtime changes begin.

Agent Assignments & Parallelization

Agent	Role
@product	Confirm acceptance criteria for each PR before implementation begins
@architect	Review schema design (PR 1 gate), credential registry design (PR 2), hot-reload design (PR 4)
@developer	Implement all PRs. PRs are sequential (each depends on prior), but within each PR some files can be parallelized
@designer	Not needed — no UI components
Code review agent	Cross-model review before every PR push (per hard rules)

Parallelization opportunities

• Within PR 1: mapping_models.py and mapping_cli.py can be developed in parallel once the schema is agreed. Tests for each can also be parallelized.
• Within PR 2: credential_registry.py and project_registry.py are independent modules — develop and test in parallel, then wire into main.py.
• Within PR 3: Runtime flow changes across coding_orchestrator.py, orchestrator.py, mr_comment_handler.py, gitlab_poller.py, and jira_poller.py are independent file edits — parallelize with a fleet of developer agents.
• Within PR 5: Docs updates (configuration-reference.md, deployment-guide.md, testing-guide.md, README.md) are independent files. E2E mock updates (mock_jira.py, mock_gitlab.py) and run.sh test additions are also independent. Full fleet parallelization.
• Across PRs: Strictly sequential due to dependencies.

PR Breakdown

PR 1: YAML mapping models + helper CLI

Goal: Make first deploy obvious — user edits one YAML file, runs helper commands, deploys.

⚠️ GATE: After this PR is merged, user confirms the YAML source format and rendered JSON output format before PR 2 begins. This prevents wasted work if the schema needs revision.

Changes:

• src/gitlab_copilot_agent/mapping_models.py (new) — Pydantic models for YAML bindings (BindingConfig, MappingFile) and rendered JSON (RenderedProjectMap). Defaults handling (target_branch, credential_ref). Validation: duplicate Jira keys, invalid repo paths, missing credential refs.
• src/gitlab_copilot_agent/mapping_cli.py (new) — CLI entry point with three commands:
  • validate PATH — schema + semantic checks, human-readable errors pointing to specific bindings
  • show PATH — summary table of bindings, credential usage, defaults applied
  • render-json PATH — emit runtime JSON for JIRA_PROJECT_MAP env var
• pyproject.toml — add [project.scripts] entry for CLI, add pyyaml dependency.
• tests/test_mapping_models.py (new) — unit tests for models, validation, defaults, edge cases.
• tests/test_mapping_cli.py (new) — CLI integration tests for validate/show/render-json.

Does NOT change: existing project_mapping.py, config.py, main.py, or runtime behavior. The helper CLI is a standalone deployment-time tool.

Agent workflow:

1. @architect reviews schema design (models + CLI interface)
2. @developer (fleet of 2): one implements mapping_models.py + tests, one implements mapping_cli.py + tests
3. Code review agent (different model) reviews PR before push

Acceptance criteria:

• validate catches duplicate Jira keys, missing repos, unknown credential_refs.
• show displays a readable mapping summary.
• render-json output is valid input for the current JIRA_PROJECT_MAP env var.
• Error messages identify the specific binding at fault.

---

🚦 Schema Confirmation Gate

After PR 1 merges, the user reviews:

1. The YAML source format (MappingFile model)
2. The rendered JSON format (RenderedProjectMap model)
3. The CLI output of show and render-json

No further PRs proceed until the user explicitly confirms the schema.

---

PR 2: Runtime schema migration + credential registry

Goal: Replace internal JIRA_PROJECT_MAP schema with the new rendered format. Add credential registry for per-repo tokens.

Changes:

• src/gitlab_copilot_agent/credential_registry.py (new) — Startup-loaded registry that reads GITLAB_TOKEN (default) and GITLAB_TOKEN__<ALIAS> env vars. Resolves credential_ref → token. Never logs raw secrets.
• src/gitlab_copilot_agent/project_registry.py (new) — Parsed from rendered JSON at startup. Resolves repo path → GitLab project ID + clone URL using correct token. Answers: Jira project → resolved context, GitLab project ID → resolved context.
• src/gitlab_copilot_agent/config.py — Update Settings to accept the new rendered JSON format for JIRA_PROJECT_MAP. Add credential_registry property. Keep backward compat parsing during this PR (accept both old and new format).
• src/gitlab_copilot_agent/project_mapping.py — Update or replace models to align with mapping_models.py rendered output.
• src/gitlab_copilot_agent/main.py — Wire credential registry and project registry at startup. Replace raw ProjectMap.model_validate_json() with new registry construction. Log startup summary (accepted mappings, credential usage, no secrets).
• tests/test_credential_registry.py (new) — Tests for default token, named aliases, missing refs, secret sanitization.
• tests/test_project_registry.py (new) — Tests for repo path resolution, Jira key lookup, project ID lookup.
• Update tests/conftest.py — Add fixtures/factories for credential registry and project registry.

Agent workflow:

1. @architect reviews credential registry design and project registry API
2. @developer (fleet of 2): one implements credential_registry.py + tests, one implements project_registry.py + tests
3. @developer (sequential after above): wire into config.py and main.py
4. Code review agent reviews PR before push

Acceptance criteria:

• Default GITLAB_TOKEN works without any credential_ref.
• Named GITLAB_TOKEN__PLATFORM_TEAM resolves via credential_ref: platform_team.
• Startup fails fast with clear error if a binding references an unknown credential_ref.
• Startup summary logs accepted mappings without leaking secrets.

---

PR 3: Credential-aware runtime flows

Goal: Thread per-repo credentials through all runtime paths.

Changes:

• src/gitlab_copilot_agent/coding_orchestrator.py — Accept resolved project context (includes token) instead of bare GitLabProjectMapping. Use per-project token for clone, push, MR creation.
• src/gitlab_copilot_agent/task_executor.py / task_runner.py — Pass credential ref in task payload. Task runner resolves it from its own env.
• src/gitlab_copilot_agent/orchestrator.py — Look up project in registry by GitLab project ID for webhook-triggered reviews; use correct token.
• src/gitlab_copilot_agent/mr_comment_handler.py — Same: resolve token from registry by project ID.
• src/gitlab_copilot_agent/gitlab_poller.py — Group polled projects by credential_ref; one GitLabClient per credential group.
• src/gitlab_copilot_agent/jira_poller.py — Pass resolved project context (with token) to handler.
• Update existing tests to use credential-aware fixtures.

Agent workflow:

1. @developer (fleet of 5 — one per file group):
   • coding_orchestrator.py + its test
   • task_executor.py + task_runner.py + their tests
   • orchestrator.py + its test
   • mr_comment_handler.py + its test
   • gitlab_poller.py + jira_poller.py + their tests
2. @developer (sequential after fleet): integration pass to verify wiring across modules
3. Code review agent reviews PR before push

Acceptance criteria:

• Jira coding flow uses the per-binding token end-to-end (clone → push → MR).
• Webhook review flow resolves the correct token for the MR's project.
• /copilot comment handling uses the correct token.
• GitLab polling groups projects by credential and uses appropriate tokens.
• Task runner receives and uses the correct credential_ref.

---

PR 4: Graceful shutdown + hot-reload

Goal: Config changes without downtime.

Graceful shutdown changes:

• src/gitlab_copilot_agent/main.py — Lifespan shutdown: signal pollers to stop accepting new work, await in-flight tasks (with timeout), then exit. Use asyncio.Event for coordinated drain.
• src/gitlab_copilot_agent/jira_poller.py — Add drain() method: finish current _poll_once(), don't start another. Respect cancellation.
• src/gitlab_copilot_agent/gitlab_poller.py — Same drain pattern.

Hot-reload changes:

• src/gitlab_copilot_agent/main.py — Store project_registry and jira_poller in app.state. Add POST /config/reload endpoint.
• src/gitlab_copilot_agent/jira_poller.py — Add asyncio.Lock around _poll_once(). Add reload_map(new_map) method that acquires lock, swaps map, clears _processed_issues, logs warning.
• src/gitlab_copilot_agent/project_registry.py — Make registry swappable (atomic reference replacement).
• tests/test_graceful_shutdown.py (new) — Tests: drain completes in-flight work, drain respects timeout, shutdown order.
• tests/test_hot_reload.py (new) — Tests: reload swaps map atomically, clears processed_issues, logs warning, returns diff, rejects invalid config.

Design decisions:

• On reload, _processed_issues is cleared. This is safe because Jira status transitions ("AI Ready" → "In Progress") prevent the JQL from returning already-dispatched issues. A structured log warning is emitted on every reload.
• Reload re-reads env vars (or a mounted config file path) and re-validates through the same pipeline as startup.
• Reload endpoint returns a summary of what changed (added/removed bindings).

Agent workflow:

1. @architect reviews hot-reload design (locking strategy, reload endpoint contract)
2. @developer (fleet of 2): one implements graceful shutdown (drain pattern in pollers + main.py), one implements hot-reload (reload endpoint + lock + registry swap)
3. @developer (sequential): integration test for full reload flow
4. Code review agent reviews PR before push

Acceptance criteria:

• Controller restart completes in-flight Jira orchestration before exiting.
• POST /config/reload swaps the project map without restart.
• Reload endpoint validates new config before applying (rejects invalid YAML/JSON).
• Reload logs a summary of changes and a warning about cleared dedup state.

---

PR 5: Documentation, deployment wiring, and E2E tests

Goal: Complete documentation for the YAML-first workflow. Update deployment configs. Add E2E coverage for new scenarios.

Documentation changes:

• docs/wiki/configuration-reference.md — Rewrite "Jira Integration" and "Jira Project Map Format" sections for YAML-first workflow. Document credential_ref, named GITLAB_TOKEN__* convention, helper CLI usage (validate/show/render-json). Add hot-reload endpoint reference.
• docs/wiki/deployment-guide.md — Add new sections: "First Deploy with YAML Mapping", "Adding a New Team/Project" (step-by-step walkthrough showing YAML edit → validate → render-json → deploy or reload), "Hot-Reload Mapping Changes" (when to use reload vs. restart), "Graceful Shutdown Behavior".
• docs/wiki/testing-guide.md — Add section on testing mapping changes: how to write tests for new bindings, how to use make_settings() with credential registry fixtures, E2E test for multi-project mapping.
• docs/wiki/security-model.md — Update credential handling section for named tokens and credential registry. Document that raw secrets never appear in YAML or logs.
• docs/wiki/data-models.md — Add BindingConfig, MappingFile, RenderedProjectMap, CredentialRegistry, ProjectRegistry model documentation.
• README.md — Update quick start and configuration sections to reference YAML workflow and helper CLI.
• demo.env — Update example config with YAML mapping example (commented).

Deployment wiring:

• helm/gitlab-copilot-agent/values.yaml — Add gitlab.namedTokens map for GITLAB_TOKEN__* secrets. Add support for mapping file ConfigMap.
• helm/gitlab-copilot-agent/templates/secret.yaml — Template loop for named token secrets.
• helm/gitlab-copilot-agent/templates/configmap.yaml — Support for rendered JSON from helper CLI output.
• infra/variables-apps.tf — Add variables for named GitLab tokens.
• infra/keyvault.tf — Update bootstrap secret seeding for named tokens.
• infra/container-apps.tf — Wire named token env vars into controller and task-runner.

E2E test updates:

• tests/e2e/mock_jira.py — Add support for multiple Jira projects (currently handles one). Add /projects endpoint returning project metadata.
• tests/e2e/mock_gitlab.py — Add support for multiple repos (currently one bare repo). Accept different tokens per repo for credential verification.
• tests/e2e/run.sh — Add two new E2E test flows:
  • TEST 5: Multi-project Jira polling — Configure two Jira project → GitLab repo bindings. Verify both repos get MRs created with correct branches.
  • TEST 6: Hot-reload mapping — Start with one binding, call POST /config/reload to add a second binding, verify the second project starts getting processed without restart.
• tests/e2e/.env.e2e — Update with multi-project mapping JSON and named token env vars.

Agent workflow (highly parallelizable):

1. @developer (fleet of 6 — all independent):
   • configuration-reference.md update
   • deployment-guide.md update
   • testing-guide.md + security-model.md + data-models.md updates
   • README.md + demo.env updates
   • Helm chart updates (values.yaml, secret.yaml, configmap.yaml)
   • Terraform updates (variables-apps.tf, keyvault.tf, container-apps.tf)
2. @developer (fleet of 3 — all independent):
   • mock_jira.py multi-project support
   • mock_gitlab.py multi-repo + per-token support
   • run.sh new test flows (TEST 5 + TEST 6)
3. @developer (sequential after all above): integration pass — run full make e2e-test
4. Code review agent reviews PR before push

Acceptance criteria:

• Documentation covers first-deploy, add-a-team, hot-reload, and troubleshooting workflows.
• A user reading only the deployment guide can configure a multi-project mapping from scratch.
• Helm and Terraform examples are updated for named credential secrets.
• E2E tests verify multi-project Jira flow and hot-reload scenario.
• make e2e-test passes with all 6 test flows.

Dependencies

PR 1 (models + CLI)
  └→ 🚦 Schema confirmation gate (user reviews and approves schema)
       └→ PR 2 (runtime schema + credential registry)
            └→ PR 3 (credential-aware flows)
                 └→ PR 4 (graceful shutdown + hot-reload)
                      └→ PR 5 (docs + deployment wiring + E2E tests)

Each PR is independently mergeable and leaves the app functional.

Out of Scope (per issue #270)

• One Jira issue fan-out to multiple repos
• Admin UI/page
• Persistent mapping storage outside deployment config
• Backward compatibility migration tooling
• Raw secrets embedded in mapping YAML
• Direct runtime YAML file loading (runtime always uses rendered JSON)

Risks / Notes

• PR 2 backward compat: During transition, accept both old and new JIRA_PROJECT_MAP formats. Remove old format support in a follow-up after deployment migration.
• PR 3 task payload size: Adding credential_ref to task payloads is a schema change. Task runner must handle both old (no ref → default token) and new payloads during rollout.
• PR 4 reload security: The /config/reload endpoint should not be exposed publicly. It reads from env/mounted config — no user-supplied payload. Consider adding auth or restricting to localhost.
• Hot-reload and credential registry: Reload re-reads GITLAB_TOKEN__* env vars. In containerized deployments, env vars are immutable after start — so adding a new credential alias requires a restart. Mapping changes (add/remove bindings using existing credentials) work via reload. Document this limitation.
• E2E test stability: Multi-project E2E tests increase mock complexity. Keep mock services deterministic — use fixed project IDs and predictable response ordering.