diff --git a/cu-benchmark/README.md b/cu-benchmark/README.md new file mode 100644 index 0000000..9e50d86 --- /dev/null +++ b/cu-benchmark/README.md @@ -0,0 +1,60 @@ +# CU Benchmark Report + +Posts a PR comment with per-instruction **Avg CU deltas vs `main`** from your program's compute-unit report, and optionally keeps a committed baseline in sync. + +## Contract: what your tests must produce + +A **`cu_report.md`** file with a Markdown table that has (at least) `Instruction` and `Avg CUs` columns: + +```markdown +| Instruction | Avg CUs | +| ----------- | ------- | +| create_plan | 1250 | +| claim | 1980 | + +*Generated: 2026-06-08* +``` + +- Column order is free; extra columns (`Min CUs`, `Est Cost`, โ€ฆ) are passed through to the comment. +- `Avg CUs` must be a plain integer. The `ฮ”` is computed on it, joined by `Instruction`. +- An optional `*Generated:` line is carried into the comment footer. + +**Reference implementations:** + +- See how the subscriptions program does it: [`cu_tracker.rs`](https://github.com/solana-foundation/subscriptions/blob/main/tests/integration-tests/src/utils/cu_tracker.rs) +- See how the rewards program does it: [`cu_utils.rs`](https://github.com/solana-foundation/rewards/blob/main/tests/integration-tests/src/utils/cu_utils.rs) + +## How it works + +1. Auto-locate one `cu_report.md` (ignoring `target/`, `node_modules/`, `.git/`); baseline is its sibling `cu_baseline.md`. +2. Diff against the baseline committed on `main`, upsert a comment: `๐Ÿ”บ +N` ยท `๐Ÿ”ป -N` ยท `โ€“` unchanged ยท `๐Ÿ†•` new ยท `๐Ÿ—‘` removed. No baseline โ†’ all `๐Ÿ†•`. +3. If `commit-baseline` is on (and same-repo PR), commit the refreshed baseline to the **PR's head branch** โ€” it reaches `main` via the normal merge, never a direct push. Fork PRs skip. + +## Inputs + +| input | default | notes | +| ----------------- | -------------------- | ------------------------------------------------------------------------ | +| `report-path` | `""` (auto-discover) | Set only if a repo emits several `cu_report.md`. | +| `commit-baseline` | `false` | Opt-in baseline refresh. Needs `contents: write` + a same-repo check. | + +## Usage + +```yaml +on: pull_request +permissions: + contents: write # only if commit-baseline is enabled + pull-requests: write +jobs: + compute-units: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + with: { fetch-depth: 0 } + - uses: ./.github/actions/setup + - run: just test-and-benchmark # must write cu_report.md (see contract) + - uses: solana-developers/github-actions/cu-benchmark@vX + with: + commit-baseline: ${{ github.event.pull_request.head.repo.full_name == github.repository }} +``` + +Comment-only mode needs just `pull-requests: write`. diff --git a/cu-benchmark/action.yaml b/cu-benchmark/action.yaml new file mode 100644 index 0000000..1ec71b2 --- /dev/null +++ b/cu-benchmark/action.yaml @@ -0,0 +1,214 @@ +name: "CU Benchmark Report" +description: "Diffs a freshly generated compute-unit report against the committed baseline, posts an upsert PR comment with per-instruction deltas, and refreshes the baseline on same-repo PRs." + +inputs: + report-path: + description: "Override for the generated CU report path. Leave empty to auto-discover a single cu_report.md (set this only when a repo emits several). The baseline is its sibling cu_baseline.md." + required: false + default: "" + commit-baseline: + description: "Opt-in: auto-commit the refreshed baseline to the head branch. Requires `contents: write` and a same-repo check (forks skip). False = comment-only." + required: false + default: "false" + +runs: + using: "composite" + steps: + - name: Locate report and baseline + shell: bash + env: + REPORT_PATH_INPUT: ${{ inputs.report-path }} + run: | + set -euo pipefail + REPORT_PATH="$REPORT_PATH_INPUT" + if [ -z "$REPORT_PATH" ]; then + found="$(find . -name cu_report.md \ + -not -path '*/target/*' -not -path '*/node_modules/*' -not -path '*/.git/*')" + count="$(printf '%s\n' "$found" | grep -c . || true)" + if [ "$count" -eq 0 ]; then + echo "::error::No cu_report.md found; set report-path." >&2 + exit 1 + fi + if [ "$count" -gt 1 ]; then + echo "::error::Multiple cu_report.md found; set report-path. Found: $found" >&2 + exit 1 + fi + REPORT_PATH="${found#./}" + fi + if [ ! -f "$REPORT_PATH" ]; then + echo "::error::Report not found at $REPORT_PATH" >&2 + exit 1 + fi + BASELINE_PATH="$(dirname "$REPORT_PATH")/cu_baseline.md" + echo "REPORT_PATH=$REPORT_PATH" >> "$GITHUB_ENV" + echo "BASELINE_PATH=$BASELINE_PATH" >> "$GITHUB_ENV" + echo "Report: $REPORT_PATH" + echo "Baseline: $BASELINE_PATH" + + - name: Read baseline from base branch + shell: bash + env: + BASE_REF: main + run: | + set -euo pipefail + BASELINE_FROM_BASE="$(mktemp)" + if git fetch origin "$BASE_REF" --depth=1 2>/dev/null \ + && git show "FETCH_HEAD:$BASELINE_PATH" > "$BASELINE_FROM_BASE" 2>/dev/null; then + echo "Baseline found on $BASE_REF." + else + : > "$BASELINE_FROM_BASE" + echo "No baseline on $BASE_REF; treating all instructions as new." + fi + echo "BASELINE_FROM_BASE=$BASELINE_FROM_BASE" >> "$GITHUB_ENV" + + - name: Render comment with deltas + shell: bash + env: + MARKER: "" + BASE_REF: main + run: | + set -euo pipefail + COMMENT_BODY="$(mktemp)" + echo "COMMENT_BODY=$COMMENT_BODY" >> "$GITHUB_ENV" + python3 - "$REPORT_PATH" "$BASELINE_FROM_BASE" "$COMMENT_BODY" <<'PY' + import sys + + report_path, baseline_path, out_path = sys.argv[1], sys.argv[2], sys.argv[3] + import os + marker = os.environ["MARKER"] + base_ref = os.environ["BASE_REF"] + + def parse_table(text): + rows = [l.strip() for l in text.splitlines() if l.strip().startswith("|")] + if not rows: + return [], [] + def cells(line): + return [c.strip() for c in line.strip().strip("|").split("|")] + header = cells(rows[0]) + body = [] + for line in rows[1:]: + cs = cells(line) + if all(set(c) <= set("-: ") for c in cs): + continue + body.append(cs) + return header, body + + def find_footer(text): + for line in text.splitlines(): + if line.strip().startswith("*Generated:"): + return line.strip() + return "" + + with open(report_path) as f: + report_text = f.read() + with open(baseline_path) as f: + baseline_text = f.read() + + header, rows = parse_table(report_text) + b_header, b_rows = parse_table(baseline_text) + + def col_index(hdr, name): + return hdr.index(name) if name in hdr else None + + ix_i = col_index(header, "Instruction") + avg_i = col_index(header, "Avg CUs") + + baseline_avg = {} + if b_header and ix_i is not None: + b_ix = col_index(b_header, "Instruction") + b_avg = col_index(b_header, "Avg CUs") + if b_ix is not None and b_avg is not None: + for r in b_rows: + try: + baseline_avg[r[b_ix]] = int(r[b_avg]) + except (ValueError, IndexError): + pass + + def delta_cell(name, avg_str): + if name not in baseline_avg: + return "๐Ÿ†•" + try: + d = int(avg_str) - baseline_avg[name] + except ValueError: + return "" + if d > 0: + return f"๐Ÿ”บ +{d}" + if d < 0: + return f"๐Ÿ”ป {d}" + return "โ€“" + + out = [marker, "", "## Compute Unit Report", ""] + if not header: + out.append("_No CU measurements recorded._") + else: + new_header = header[:] + delta_label = f"ฮ” Avg vs `{base_ref}`" + insert_at = (avg_i + 1) if avg_i is not None else len(new_header) + new_header.insert(insert_at, delta_label) + out.append("| " + " | ".join(new_header) + " |") + out.append("|" + "|".join(["---"] * len(new_header)) + "|") + present = set() + for r in rows: + name = r[ix_i] if ix_i is not None else "" + present.add(name) + cell = delta_cell(name, r[avg_i]) if avg_i is not None else "" + row = r[:] + row.insert(insert_at, cell) + out.append("| " + " | ".join(row) + " |") + removed = [n for n in baseline_avg if n not in present] + if removed: + out.append("") + for n in sorted(removed): + out.append(f"๐Ÿ—‘ Removed since `{base_ref}`: `{n}` (was {baseline_avg[n]} avg CUs)") + out.append("") + out.append(f"๐Ÿ”บ increase ยท ๐Ÿ”ป decrease ยท โ€“ unchanged ยท ๐Ÿ†• new ยท ๐Ÿ—‘ removed (vs `{base_ref}`)") + + footer = find_footer(report_text) + if footer: + out += ["", footer] + + with open(out_path, "w") as f: + f.write("\n".join(out) + "\n") + PY + + - name: Upsert PR comment + shell: bash + env: + GH_TOKEN: ${{ github.token }} + REPO: ${{ github.repository }} + PR: ${{ github.event.pull_request.number }} + MARKER: "" + run: | + set -euo pipefail + COMMENT_ID="$(gh api "repos/$REPO/issues/$PR/comments" --paginate \ + --jq ".[] | select(.body | contains(\"$MARKER\")) | .id" | head -1)" + if [ -n "$COMMENT_ID" ]; then + gh api "repos/$REPO/issues/comments/$COMMENT_ID" -X PATCH -F body=@"$COMMENT_BODY" + else + gh api "repos/$REPO/issues/$PR/comments" -F body=@"$COMMENT_BODY" + fi + + - name: Refresh committed baseline + if: ${{ inputs.commit-baseline == 'true' }} + shell: bash + env: + HEAD_REF: ${{ github.head_ref }} + run: | + set -euo pipefail + if [ -z "$HEAD_REF" ]; then + echo "No head ref; skipping baseline refresh." + exit 0 + fi + git fetch origin "$HEAD_REF":"refs/heads/$HEAD_REF" --depth=1 + git checkout "$HEAD_REF" + if [ -f "$BASELINE_PATH" ] && cmp -s "$REPORT_PATH" "$BASELINE_PATH"; then + echo "Baseline already current; nothing to commit." + exit 0 + fi + mkdir -p "$(dirname "$BASELINE_PATH")" + cp "$REPORT_PATH" "$BASELINE_PATH" + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add "$BASELINE_PATH" + git commit -m "chore(cu): update CU baseline" + git push origin "HEAD:$HEAD_REF"