Skip to content

Commit 58c2078

Browse files
Add conformance test CI workflow
- Runs on pull requests to main - Compares PR branch against merge-base with origin/main - Outputs full conformance report to GitHub Actions Job Summary - Uploads detailed report as artifact for deeper investigation - Does not fail the build on differences (may be intentional)
1 parent 1281e4e commit 58c2078

File tree

1 file changed

+69
-0
lines changed

1 file changed

+69
-0
lines changed

.github/workflows/conformance.yml

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
name: Conformance Test
2+
3+
on:
4+
pull_request:
5+
6+
permissions:
7+
contents: read
8+
9+
jobs:
10+
conformance:
11+
runs-on: ubuntu-latest
12+
13+
steps:
14+
- name: Check out code
15+
uses: actions/checkout@v6
16+
with:
17+
# Fetch full history to access merge-base
18+
fetch-depth: 0
19+
20+
- name: Set up Go
21+
uses: actions/setup-go@v6
22+
with:
23+
go-version-file: "go.mod"
24+
25+
- name: Download dependencies
26+
run: go mod download
27+
28+
- name: Run conformance test
29+
id: conformance
30+
run: |
31+
# Run conformance test, capture stdout for summary
32+
script/conformance-test > conformance-summary.txt 2>&1 || true
33+
34+
# Output the summary
35+
cat conformance-summary.txt
36+
37+
# Check result
38+
if grep -q "RESULT: ALL TESTS PASSED" conformance-summary.txt; then
39+
echo "status=passed" >> $GITHUB_OUTPUT
40+
else
41+
echo "status=differences" >> $GITHUB_OUTPUT
42+
fi
43+
44+
- name: Generate Job Summary
45+
run: |
46+
# Add the full markdown report to the job summary
47+
echo "# MCP Server Conformance Report" >> $GITHUB_STEP_SUMMARY
48+
echo "" >> $GITHUB_STEP_SUMMARY
49+
echo "Comparing PR branch against merge-base with \`origin/main\`" >> $GITHUB_STEP_SUMMARY
50+
echo "" >> $GITHUB_STEP_SUMMARY
51+
52+
# Extract and append the report content (skip the header since we added our own)
53+
tail -n +5 conformance-report/CONFORMANCE_REPORT.md >> $GITHUB_STEP_SUMMARY
54+
55+
echo "" >> $GITHUB_STEP_SUMMARY
56+
echo "---" >> $GITHUB_STEP_SUMMARY
57+
echo "" >> $GITHUB_STEP_SUMMARY
58+
59+
# Add interpretation note
60+
if [ "${{ steps.conformance.outputs.status }}" = "passed" ]; then
61+
echo "✅ **All conformance tests passed** - No behavioral differences detected." >> $GITHUB_STEP_SUMMARY
62+
else
63+
echo "⚠️ **Differences detected** - Review the diffs above to ensure changes are intentional." >> $GITHUB_STEP_SUMMARY
64+
echo "" >> $GITHUB_STEP_SUMMARY
65+
echo "Common expected differences:" >> $GITHUB_STEP_SUMMARY
66+
echo "- New tools/toolsets added" >> $GITHUB_STEP_SUMMARY
67+
echo "- Tool descriptions updated" >> $GITHUB_STEP_SUMMARY
68+
echo "- Capability changes (intentional improvements)" >> $GITHUB_STEP_SUMMARY
69+
fi

0 commit comments

Comments
 (0)