Add Copilot pre-flight diagnostic for GHES environments#20975
Merged
Conversation
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Collaborator
|
@claude[agent] fix the pr title and summary |
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Contributor
Author
|
Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details. Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
Collaborator
|
@claude[agent] fix the pr title and summary |
Contributor
Author
I'm unable to update the PR title and description due to GitHub API access restrictions (HTTP 403). The PR should have:
Could you please update these manually, or grant the necessary API permissions? The commit de9842f has the proper title in its message. Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
This PR adds a pre-flight diagnostic step for GitHub Copilot CLI on GitHub Enterprise Server (GHES) environments to provide clear error messages before execution fails. Changes
|
Claude
AI
changed the title
Update all Copilot engine tests to account for the new pre-flight diagnostic step (2 steps instead of 1). Extract a shared helper for finding the Copilot execution step, and regenerate WASM golden files to include the new step. Fixes: - TestFirewallArgsInCopilotEngine - TestFirewallBlockedDomainsInCopilotEngine - TestFirewallLogLevelInCopilotEngine - TestChrootModeInAWFContainer - TestChrootModeEnvFlags - TestMCPScriptsWithFirewallIncludesHostDockerInternal - TestEngineAWFEnableApiProxy - TestWasmGolden_CompileFixtures (golden files) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…0985) * Initial plan * fix: correct test conditions for preflight step count checks Fix two categories of incorrectly written tests introduced with the Copilot pre-flight diagnostic step: 1. copilot_engine_test.go: Two "skips preflight" sub-tests checked `len(steps) != 2` but preflight is intentionally skipped, returning only 1 step. Change condition to `!= 1`. 2. engine_agent_import_test.go: Claude and Codex tests expected 2 steps (preflight + execution) like Copilot, but those engines only return 1 execution step. Change to `!= 1` and use `steps[0]`. Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Copilot CLI pre-flight diagnostic step (primarily for GHES) to detect token/licensing/config issues early and provide clearer, actionable errors before/alongside Copilot execution.
Changes:
- Introduces
copilot_preflight_diagnostic.shto validate token exchange against the Copilot internal token endpoint and emit GHES-focused guidance. - Updates Copilot engine execution to inject the preflight diagnostic step ahead of the CLI execution step (with skip logic).
- Updates unit tests and multiple golden/locked workflow fixtures to reflect the additional step.
Reviewed changes
Copilot reviewed 91 out of 91 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| actions/setup/sh/copilot_preflight_diagnostic.sh | New shell diagnostic script to validate Copilot token exchange and GHES configuration hints. |
| pkg/workflow/copilot_engine_execution.go | Injects the preflight diagnostic GitHub Actions step before Copilot CLI execution. |
| pkg/workflow/copilot_engine_test.go | Adds/updates tests for preflight injection and skip behavior; updates step-count expectations. |
| pkg/workflow/enable_api_proxy_test.go | Adds a shared helper to assert preflight+execution steps; updates Copilot-related assertions. |
| pkg/workflow/tools_timeout_test.go | Updates to account for preflight step and validate env vars on the execution step. |
| pkg/workflow/sandbox_mounts_test.go | Updates step indexing to account for preflight step in Copilot execution. |
| pkg/workflow/mcp_scripts_firewall_test.go | Updates to use the shared helper and target the execution step after preflight. |
| pkg/workflow/gh_cli_mount_test.go | Updates to use the shared helper and target the execution step after preflight. |
| pkg/workflow/firewall_log_level_test.go | Updates to use the shared helper and target the execution step after preflight. |
| pkg/workflow/firewall_blocked_domains_test.go | Updates to use the shared helper and target the execution step after preflight. |
| pkg/workflow/firewall_args_test.go | Updates to use the shared helper and target the execution step after preflight. |
| pkg/workflow/engine_agent_import_test.go | Updates step-count and step indexing expectations due to preflight insertion. |
| pkg/workflow/testdata/wasm_golden/TestWasmGolden_CompileFixtures/basic-copilot.golden | Golden workflow updated to include the preflight diagnostic step. |
| pkg/workflow/testdata/wasm_golden/TestWasmGolden_CompileFixtures/smoke-copilot.golden | Golden workflow updated to include the preflight diagnostic step. |
| pkg/workflow/testdata/wasm_golden/TestWasmGolden_CompileFixtures/with-imports.golden | Golden workflow updated to include the preflight diagnostic step. |
| .github/workflows/ace-editor.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/agent-persona-explorer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/bot-detection.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/chroma-issue-indexer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/ci-doctor.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/code-simplifier.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/contribution-check.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/constraint-solving-potd.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/daily-cli-tools-tester.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/daily-firewall-report.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/daily-performance-summary.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/daily-regulatory.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/daily-semgrep-scan.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/daily-team-status.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/dependabot-burner.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/dependabot-go-checker.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/example-permissions-warning.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/firewall.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/functional-pragmatist.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/github-remote-mcp-auth-test.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/glossary-maintainer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/gpclean.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/grumpy-reviewer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/hourly-ci-cleaner.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/issue-monster.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/issue-triage-agent.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/jsweep.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/layout-spec-maintainer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/mcp-inspector.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/mergefest.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/metrics-collector.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/notion-issue-summary.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/org-health-report.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/pdf-summary.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/plan.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/poem-bot.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/portfolio-analyst.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/pr-nitpick-reviewer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/pr-triage-agent.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/python-data-charts.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/q.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/refiner.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/release.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/repo-audit-analyzer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/repo-tree-map.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/repository-quality-improver.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/research.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/security-compliance.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/security-review.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/slide-deck-maintainer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/smoke-copilot-arm.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/smoke-copilot.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/smoke-multi-pr.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/smoke-project.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/smoke-temporary-id.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/smoke-test-tools.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/smoke-workflow-call-with-inputs.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/smoke-workflow-call.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/stale-repo-identifier.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/sub-issue-closer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/super-linter.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/technical-doc-writer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/terminal-stylist.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/test-dispatcher.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/test-project-url-default.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/test-workflow.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/tidy.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/ubuntu-image-analyzer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/video-analyzer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/weekly-editors-health-check.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/weekly-issue-summary.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/weekly-safe-outputs-spec-review.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/workflow-generator.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/workflow-health-manager.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/workflow-normalizer.lock.yml | Locked workflow updated to include preflight diagnostic step. |
| .github/workflows/workflow-skill-extractor.lock.yml | Locked workflow updated to include preflight diagnostic step. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+490
to
+494
| // Use COPILOT_GITHUB_TOKEN for the diagnostic | ||
| // #nosec G101 -- This is a GitHub Actions expression template, not a hardcoded credential | ||
| step = append(step, " COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }}") | ||
| step = append(step, " GITHUB_SERVER_URL: ${{ github.server_url }}") | ||
| step = append(step, " GITHUB_API_URL: ${{ github.api_url }}") |
Comment on lines
+35
to
+38
| HTTP_STATUS=$(curl -s -o /tmp/copilot_token_exchange.json -w "%{http_code}" \ | ||
| -H "Authorization: Bearer ${COPILOT_GITHUB_TOKEN}" \ | ||
| -H "Accept: application/json" \ | ||
| "$TOKEN_EXCHANGE_URL" 2>&1 || echo "000") |
Comment on lines
304
to
309
| if len(steps) == 0 { | ||
| t.Fatal("Expected at least one execution step") | ||
| } | ||
|
|
||
| stepContent := strings.Join(steps[0], "\n") | ||
| stepContent := strings.Join(steps[1], "\n") | ||
|
|
Contributor
|
Copilot stopped work on behalf of
pelikhan due to an error
March 14, 2026 21:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a pre-flight diagnostic step for GitHub Copilot CLI on GitHub Enterprise Server (GHES) environments to provide clear error messages before execution fails.
Problem Solved
On GHES, Copilot CLI failures are opaque - users get generic errors like "Error loading models: 400 Bad Request" when the actual issue is licensing (403 from token exchange endpoint). This diagnostic provides clear, actionable error messages before Copilot CLI execution.
Changes Made
copilot_preflight_diagnostic.sh: Shell script that validatesCOPILOT_GITHUB_TOKENcan exchange for Copilot access by testing${GITHUB_API_URL}/copilot_internal/v2/tokencopilot_engine_execution.goto inject the diagnostic step before Copilot CLI executioncopilot_engine_test.goto validate the pre-flight diagnostic behaviorKey Features
Token Exchange Test: Validates that the GitHub token can be exchanged for Copilot access before running the CLI.
Error Detection:
GHES-Specific Checks: Validates
engine.api-targetconfiguration matchesGITHUB_API_URLSmart Skipping: Diagnostic is skipped when:
copilot-requestsfeature is enabled (uses GitHub Actions token)Files Changed
actions/setup/sh/copilot_preflight_diagnostic.sh- Pre-flight diagnostic scriptpkg/workflow/copilot_engine_execution.go- Inject diagnostic step before executionpkg/workflow/copilot_engine_test.go- Comprehensive unit tests