Skip to content

docs: update query benchmarks (3.4.1)#646

Merged
carlos-alm merged 9 commits intomainfrom
benchmark/query-v3.4.1-20260327-034126
Mar 27, 2026
Merged

docs: update query benchmarks (3.4.1)#646
carlos-alm merged 9 commits intomainfrom
benchmark/query-v3.4.1-20260327-034126

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

Automated query benchmark update for 3.4.1 from workflow run #484.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 27, 2026

Greptile Summary

This is an automated benchmark documentation update for version 3.4.1, generated from workflow run #484. It appends new rows to the summary tables, updates the "Latest results" and "Build Phase Breakdown" sections, and inserts a full JSON benchmark block for 3.4.1 in both BUILD-BENCHMARKS.md and QUERY-BENCHMARKS.md. No source code is changed.\n\nKey observations from the review:\n\n- Data consistency verified: All per-file figures, raw totals, extrapolated 50k-file estimates, incremental rebuild timings, and delta percentages are internally consistent across the summary tables, detail sections, and embedded JSON blocks.\n- Previously requested note added: The Note (3.4.1) block in QUERY-BENCHMARKS.md correctly explains the ↑45% diffImpact jump as an inflated delta caused by 3.4.0's unusually low baseline, notes the mid-target change from rule to noTests, and places the absolute values in historical context.\n- Engine parity note in BUILD-BENCHMARKS.md: A new parity callout documents the 26-node native/WASM discrepancy and links to issue #649. While the issue link is the correct response, the wording could be sharpened to emphasise that this is an open bug (consistent with CLAUDE.md's guidance against normalising engine divergences).\n- No new regressions: Build time is +8%/−2% per file (within noise), fnDeps grew only 5–7% (consistent with codebase expansion), and fnImpact is flat-to-down. The 27% improvement in native 1-file rebuild time is a notable positive."

Confidence Score: 5/5

Safe to merge — documentation-only update with verified internal consistency and all previously requested changes in place.

All data values have been cross-checked and are consistent between summary tables, detail sections, and embedded JSON. The Note (3.4.1) requested in the prior review thread is present. The engine parity note is the only new concern, and it is non-blocking: it explicitly links to issue #649 rather than framing the discrepancy as acceptable, so it doesn't fully violate CLAUDE.md's guidance. No source code is touched.

No files require special attention; both files are auto-generated documentation.

Important Files Changed

Filename Overview
generated/benchmarks/BUILD-BENCHMARKS.md Adds 3.4.1 build/incremental/query rows and JSON block; all per-file numbers, totals, and delta percentages are internally consistent. Includes a new engine parity note for the 26-node native/WASM discrepancy, linked to issue #649.
generated/benchmarks/QUERY-BENCHMARKS.md Adds 3.4.1 query latency rows, updates the 'Latest results' section (mid-target changed from rule to noTests), and inserts the requested Note (3.4.1) explaining the inflated ↑45% diffImpact delta. Summary table, detail tables, and JSON block are all consistent.

Sequence Diagram

sequenceDiagram
    participant WF as GitHub Actions Workflow #484
    participant CG as codegraph CLI (native + WASM)
    participant BUILD as BUILD-BENCHMARKS.md
    participant QUERY as QUERY-BENCHMARKS.md
    participant PR as Pull Request #646

    WF->>CG: Run build benchmark (473 files)
    CG-->>WF: Build metrics (time, nodes, edges, DB size)
    WF->>CG: Run query benchmark (fnDeps, fnImpact, diffImpact)
    CG-->>WF: Query latencies (hub=buildGraph, mid=noTests, leaf=docs)
    WF->>BUILD: Prepend 3.4.1 rows + JSON block + parity note
    WF->>QUERY: Prepend 3.4.1 rows + JSON block + Note(3.4.1)
    BUILD-->>PR: Commit updated BUILD-BENCHMARKS.md
    QUERY-->>PR: Commit updated QUERY-BENCHMARKS.md
Loading

Reviews (3): Last reviewed commit: "fix(docs): restore 3.4.1 query benchmark..." | Re-trigger Greptile

Comment on lines +8 to +9
| 3.4.1 | native | 8.9 ↑5% | 8.9 ↑5% | 8.8 ↑5% | 3.2 ~ | 3.1 ↓3% | 3.1 ↓3% | 8.1ms ↑45% |
| 3.4.1 | wasm | 9.1 ↑7% | 9.2 ↑7% | 9.1 ↑7% | 3.3 ~ | 3.2 ↓3% | 3.2 ↓3% | 7.1ms ↑45% |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unexplained 45% diffImpact regression may benefit from a note

Both engines show a significant diffImpact latency jump from 3.4.0 → 3.4.1 (5.6ms→8.1ms native, 4.9ms→7.1ms wasm, both +45%). Precedent in this file — the <!-- NOTES_START --> block already contains explanatory notes for the large deltas in 3.3.1 and 3.4.0.

In absolute terms the new values (8.1ms / 7.1ms) fall within the historical range (e.g. 3.1.3 native: 8.3ms, 3.3.0 native: 8.8ms), so the +45% delta is inflated by 3.4.0 being an unusually low baseline. Still, without a note, a future reader comparing only consecutive rows may flag this as a regression worth investigating.

If there is a known explanation (e.g. 3.4.0's diffImpact was unusually fast due to a small staged diff, or the mid-query target changed from rule to noTests affecting scope), adding a Note (3.4.1): entry in the NOTES_START block — consistent with the notes for 3.3.1 and 3.4.0 — would keep the file self-explanatory.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a Note (3.4.1) entry in the NOTES_START block explaining the +45% diffImpact delta: the 3.4.0 baseline was unusually low (5.6ms/4.9ms), the new absolute values (8.1ms/7.1ms) fall within historical norms, and the mid-query target changed from rule to noTests. Consistent with the existing note format for 3.3.1 and 3.4.0.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2de77ca. The previous fix commit (9dbca9f) accidentally removed all 3.4.1 data instead of adding the note. This commit restores the full 3.4.1 dataset (summary table rows, Latest results section, and JSON block) alongside the Note (3.4.1) in the NOTES_START block that was already present.

…note (#647)

The merge with main reverted raw totals, phase breakdowns, incremental
rebuild rows, query rows, and the JSON blob back to 3.4.0 values.
Restores the correct 3.4.1 data and adds a note referencing #649 for
the native/WASM node count divergence (10,982 vs 10,956).
@carlos-alm
Copy link
Copy Markdown
Contributor

@greptileai

…034126' into benchmark/query-v3.4.1-20260327-034126
…ts (#646)

The previous fix commit accidentally removed all 3.4.1 data instead of
adding the explanatory note. This restores the summary table rows,
"Latest results" section, and JSON block for 3.4.1.
@carlos-alm
Copy link
Copy Markdown
Contributor

All three P1 issues addressed in 2de77ca:

  1. 3.4.1 summary table rows — restored (native + wasm rows at top of table)
  2. Latest results section — updated to 3.4.1 / 2026-03-27 with mid=noTests targets and correct values
  3. Note (3.4.1) in NOTES_START — was already present in the file (added by the merge from main). The JSON block for 3.4.1 is also restored.

The previous fix commit (9dbca9f) accidentally removed all 3.4.1 data during merge conflict resolution. The net diff is no longer empty — the PR now correctly adds the full 3.4.1 benchmark record plus the explanatory note.

@carlos-alm
Copy link
Copy Markdown
Contributor

@greptileai

@carlos-alm carlos-alm merged commit 398f9f8 into main Mar 27, 2026
13 checks passed
@carlos-alm carlos-alm deleted the benchmark/query-v3.4.1-20260327-034126 branch March 27, 2026 05:12
@github-actions github-actions bot locked and limited conversation to collaborators Mar 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant