Skip to content

docs: update build performance benchmarks (3.4.1)#647

Merged
carlos-alm merged 7 commits intomainfrom
benchmark/build-v3.4.1-20260327-034137
Mar 27, 2026
Merged

docs: update build performance benchmarks (3.4.1)#647
carlos-alm merged 7 commits intomainfrom
benchmark/build-v3.4.1-20260327-034137

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

Automated build benchmark update for 3.4.1 from workflow run #484.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 27, 2026

Greptile Summary

This PR is an automated benchmark update for version 3.4.1, appending new measurement rows across all benchmark tables in BUILD-BENCHMARKS.md, updating the README.md perf summary, and refining the release skill with a roadmap ordering cross-check step.\n\nKey changes:\n- BUILD-BENCHMARKS.md: 3.4.1 rows added to the main table, incremental rebuild table, and query benchmark table; raw totals updated; 50k estimates refreshed; embedded JSON block updated. An engine parity note is added referencing tracking issue #649, directly addressing the concern raised in the previous review thread.\n- README.md: Perf metrics updated to match the new 3.4.1 figures (native 5.7 ms/file, WASM 11.9 ms/file, 1-file rebuild 316 ms, ~285s at 50k files). All values cross-check correctly against the raw JSON in BUILD-BENCHMARKS.md.\n- SKILL.md: A new step 5 formalises a roadmap ordering cross-check (README vs docs/roadmap/ROADMAP.md) that runs on every release, preventing silent drift between the two files.

Confidence Score: 5/5

Safe to merge — data is internally consistent, prior review concern is addressed, and the SKILL.md improvement is additive

All percentage deltas, per-file averages, raw totals, and 50k extrapolations were spot-checked against the embedded JSON and cross-file references in README.md — every figure is consistent. The engine parity divergence that was flagged in the previous review is now surfaced with a prominent callout note and a tracking issue rather than being silently overwritten. The SKILL.md change is purely additive documentation. No logic bugs, data inconsistencies, or unaddressed concerns remain.

No files require special attention

Important Files Changed

Filename Overview
generated/benchmarks/BUILD-BENCHMARKS.md Adds 3.4.1 benchmark rows across all tables (build, incremental, query, 50k estimates) and raw totals; adds engine parity note referencing tracking issue #649 — all math verified correct
README.md Updates perf table in README to reflect 3.4.1 metrics; all values are consistent with BUILD-BENCHMARKS.md data
.claude/skills/release/SKILL.md Inserts new step 5 (roadmap ordering cross-check against ROADMAP.md) and renumbers subsequent steps; also updates step 7 to ensure the check runs even when no user-facing changes are present

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    CI[CI workflow run #484] -->|produces| RAW[Raw benchmark JSON]
    RAW -->|appended to| BBM[BUILD-BENCHMARKS.md\n3.4.1 rows + raw totals]
    BBM -->|per-file averages| README[README.md\nperf table updated]
    BBM -->|parity gap detected| NOTE[Engine parity note\nNative 10,982 vs WASM 10,956 nodes]
    NOTE -->|references| ISS[GitHub Issue #649\ntracking divergence]
    SKILL[.claude/skills/release/SKILL.md] -->|new step 5| ROADMAP_CHECK[Roadmap ordering\ncross-check vs ROADMAP.md]
    ROADMAP_CHECK -->|runs every release| DRIFT[Prevents silent\nphase-order drift]
Loading

Reviews (2): Last reviewed commit: "fix(docs): restore 3.4.1 raw totals lost..." | Re-trigger Greptile

Comment on lines +8 to +9
| 3.4.1 | native | 2026-03-27 | 473 | 5.7 ↑8% | 11.7 ~ | 23.2 ~ | 44.1 ~ | 57725 ↑5% |
| 3.4.1 | wasm | 2026-03-27 | 473 | 11.9 ↓2% | 12.2 ↑4% | 23.2 ~ | 44.1 ~ | 42276 ↑5% |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Engine parity gap violates repo policy

CLAUDE.md is explicit: "Both engines must produce identical results. If they diverge, the less-accurate engine has a bug — fix it, don't document the gap."

The 3.4.1 entries reveal a divergence across every parity-sensitive metric:

Metric Native WASM Delta
Nodes (raw) 10,982 10,956 –26
Edges (raw) 20,869 20,870 +1
DB size 26.0 MB 19.1 MB +36 %

The 26-node shortfall means the WASM engine is silently missing ~0.24 % of symbols on this codebase. The DB size spread (57,725 vs 42,276 bytes/file) suggests the discrepancy extends beyond just node count into how the schema is populated.

This divergence was already present in 3.4.0 (23.2 vs 23.1 nodes/file, 44.2 vs 44.4 edges/file), so this PR doesn't introduce it — but committing another version's benchmark without a linked issue means the gap will continue to be silently overwritten each release. Per the repo's own policy, the correct path is to:

  1. File a tracking issue for the WASM extraction/resolution bug causing the node shortfall.
  2. Reference that issue in the benchmark notes or a code comment so future benchmark runs surface the outstanding bug rather than burying it.

Context Used: CLAUDE.md (source)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. Created tracking issue #649 for the native/WASM node count divergence (10,982 vs 10,956) and DB size gap (26.0 MB vs 19.1 MB).

Added an engine parity note in the "Raw totals (latest)" section of BUILD-BENCHMARKS.md that references #649, so future benchmark runs surface the outstanding bug rather than burying it.

…034137' into benchmark/build-v3.4.1-20260327-034137
…note (#647)

The merge with main reverted raw totals, phase breakdowns, incremental
rebuild rows, query rows, and the JSON blob back to 3.4.0 values.
Restores the correct 3.4.1 data and adds a note referencing #649 for
the native/WASM node count divergence (10,982 vs 10,956).
@carlos-alm
Copy link
Copy Markdown
Contributor

@greptileai

carlos-alm added a commit that referenced this pull request Mar 27, 2026
…note (#647)

The merge with main reverted raw totals, phase breakdowns, incremental
rebuild rows, query rows, and the JSON blob back to 3.4.0 values.
Restores the correct 3.4.1 data and adds a note referencing #649 for
the native/WASM node count divergence (10,982 vs 10,956).
@carlos-alm carlos-alm merged commit aa6499a into main Mar 27, 2026
12 checks passed
@carlos-alm carlos-alm deleted the benchmark/build-v3.4.1-20260327-034137 branch March 27, 2026 04:42
@github-actions github-actions bot locked and limited conversation to collaborators Mar 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant