Skip to content

docs: update build performance benchmarks (3.4.1) #647

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
carlos-alm merged 7 commits into from
Mar 27, 2026
Merged

docs: update build performance benchmarks (3.4.1) #647

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
bde928d
docs: update build performance benchmarks (3.4.1)
github-actions[bot] Mar 27, 2026
55992af
Merge branch 'main' into benchmark/build-v3.4.1-20260327-034137
carlos-alm Mar 27, 2026
bd2c505
fix(skills): add roadmap ordering cross-check to /release skill
carlos-alm Mar 27, 2026
101a838
Merge branch 'main' into benchmark/build-v3.4.1-20260327-034137
carlos-alm Mar 27, 2026
738ac4e
Merge remote-tracking branch 'origin/benchmark/build-v3.4.1-20260327-…
carlos-alm Mar 27, 2026
28f9e75
fix(docs): restore 3.4.1 raw totals lost in merge, add engine parity …
carlos-alm Mar 27, 2026
22d084d
Merge branch 'main' into benchmark/build-v3.4.1-20260327-034137
carlos-alm Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -553,14 +553,14 @@ Self-measured on every release via CI ([build benchmarks](generated/benchmarks/B

| Metric | Latest |
|---|---|
| Build speed (native) | **5.3 ms/file** |
| Build speed (WASM) | **12.2 ms/file** |
| Build speed (native) | **5.7 ms/file** |
| Build speed (WASM) | **11.9 ms/file** |
| Query time | **12ms** |
| No-op rebuild (native) | **12ms** |
| 1-file rebuild (native) | **432ms** |
| Query: fn-deps | **1.7ms** |
| Query: path | **1.6ms** |
| ~50,000 files (est.) | **~265.0s build** |
| No-op rebuild (native) | **14ms** |
| 1-file rebuild (native) | **316ms** |
| Query: fn-deps | **1.9ms** |
| Query: path | **1.9ms** |
| ~50,000 files (est.) | **~285.0s build** |

Metrics are normalized per file for cross-version comparability. Times above are for a full initial build — incremental rebuilds only re-parse changed files.

Expand Down
157 changes: 135 additions & 22 deletions generated/benchmarks/BUILD-BENCHMARKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ Metrics are normalized per file for cross-version comparability.

| Version | Engine | Date | Files | Build (ms/file) | Query (ms) | Nodes/file | Edges/file | DB (bytes/file) |
|---------|--------|------|------:|----------------:|-----------:|-----------:|-----------:|----------------:|
| 3.4.1 | native | 2026-03-27 | 473 | 5.7 ↑8% | 11.7 ~ | 23.2 ~ | 44.1 ~ | 57725 ↑5% |
| 3.4.1 | wasm | 2026-03-27 | 473 | 11.9 ↓2% | 12.2 ↑4% | 23.2 ~ | 44.1 ~ | 42276 ↑5% |
Comment on lines +8 to +9
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps bot Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Engine parity gap violates repo policy

CLAUDE.md is explicit: "Both engines must produce identical results. If they diverge, the less-accurate engine has a bug — fix it, don't document the gap."

The 3.4.1 entries reveal a divergence across every parity-sensitive metric:

Metric Native WASM Delta
Nodes (raw) 10,982 10,956 –26
Edges (raw) 20,869 20,870 +1
DB size 26.0 MB 19.1 MB +36 %

The 26-node shortfall means the WASM engine is silently missing ~0.24 % of symbols on this codebase. The DB size spread (57,725 vs 42,276 bytes/file) suggests the discrepancy extends beyond just node count into how the schema is populated.

This divergence was already present in 3.4.0 (23.2 vs 23.1 nodes/file, 44.2 vs 44.4 edges/file), so this PR doesn't introduce it — but committing another version's benchmark without a linked issue means the gap will continue to be silently overwritten each release. Per the repo's own policy, the correct path is to:

  1. File a tracking issue for the WASM extraction/resolution bug causing the node shortfall.
  2. Reference that issue in the benchmark notes or a code comment so future benchmark runs surface the outstanding bug rather than burying it.

Context Used: CLAUDE.md (source)

Copy link
Copy Markdown
Contributor

@carlos-alm carlos-alm Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. Created tracking issue #649 for the native/WASM node count divergence (10,982 vs 10,956) and DB size gap (26.0 MB vs 19.1 MB).

Added an engine parity note in the "Raw totals (latest)" section of BUILD-BENCHMARKS.md that references #649, so future benchmark runs surface the outstanding bug rather than burying it.

| 3.4.0 | native | 2026-03-26 | 473 | 5.3 ↓13% | 11.6 ↑63% | 23.2 ↑32% | 44.2 ↑21% | 55041 ↑13% |
| 3.4.0 | wasm | 2026-03-26 | 473 | 12.2 ↑9% | 11.7 ↑62% | 23.1 ↑32% | 44.4 ↑22% | 40198 ↓15% |
| 3.3.1 | native | 2026-03-20 | 442 | 6.1 ↑74% | 7.1 ↑122% | 17.6 ↑31% | 36.5 ↑27% | 48707 ↑10% |
Expand Down Expand Up @@ -43,58 +45,64 @@ Metrics are normalized per file for cross-version comparability.

### Raw totals (latest)

> **Engine parity note:** Native reports 10,982 nodes vs WASM 10,956 (−26 nodes, ~0.24%).
> DB size also diverges: 26.0 MB native vs 19.1 MB WASM (+36%).
> Tracked in [#649](https://github.com/optave/codegraph/issues/649).

#### Native (Rust)

| Metric | Value |
|--------|-------|
| Build time | 2.5s |
| Build time | 2.7s |
| Query time | 12ms |
| Nodes | 10,963 |
| Edges | 20,921 |
| DB size | 24.8 MB |
| Nodes | 10,982 |
| Edges | 20,869 |
| DB size | 26.0 MB |
| Files | 473 |

#### WASM

| Metric | Value |
|--------|-------|
| Build time | 5.8s |
| Build time | 5.6s |
| Query time | 12ms |
| Nodes | 10,937 |
| Edges | 21,022 |
| DB size | 18.1 MB |
| Nodes | 10,956 |
| Edges | 20,870 |
| DB size | 19.1 MB |
| Files | 473 |

### Build Phase Breakdown (latest)

| Phase | Native (build) | WASM (build) | Native (1-file) | WASM (1-file) |
|-------|---------------:|-------------:|----------------:|--------------:|
| Parse | 1348.3 ms | 2438.2 ms | 78.3 ms | 239.9 ms |
| Insert nodes | 192 ms | 209.8 ms | 15.6 ms | 14.1 ms |
| Resolve imports | 13.3 ms | 15.1 ms | 1.7 ms | 1.6 ms |
| Build edges | 144.8 ms | 149.1 ms | 32.3 ms | 20.2 ms |
| Structure | 24.9 ms | 26.1 ms | 29.9 ms | 30.4 ms |
| Roles | 65 ms | 77.9 ms | 55.9 ms | 61 ms |
| AST nodes | 337.8 ms | 217.2 ms | 0.7 ms | 2.4 ms |
| Complexity | 25.8 ms | 348.1 ms | 0.7 ms | 7.9 ms |
| CFG | 160.3 ms | 370.5 ms | 0.3 ms | 0.3 ms |
| Dataflow | 117 ms | 119.5 ms | 0.5 ms | 0.5 ms |
| Parse | 1391.1 ms | 2374.7 ms | 80.8 ms | 224.1 ms |
| Insert nodes | 233.2 ms | 242.3 ms | 16.7 ms | 19 ms |
| Resolve imports | 12 ms | 14.7 ms | 1.5 ms | 1.6 ms |
| Build edges | 151.2 ms | 147.6 ms | 31.7 ms | 20.7 ms |
| Structure | 28.3 ms | 28.7 ms | 33.6 ms | 33.1 ms |
| Roles | 64.4 ms | 72 ms | 40.4 ms | 42.1 ms |
| AST nodes | 401.5 ms | 249.5 ms | 0.6 ms | 0.6 ms |
| Complexity | 28 ms | 330.8 ms | 0.6 ms | 0.7 ms |
| CFG | 182.1 ms | 368 ms | 0.3 ms | 0.4 ms |
| Dataflow | 131.2 ms | 133.6 ms | 0.4 ms | 0.5 ms |

### Estimated performance at 50,000 files

Extrapolated linearly from per-file metrics above.

| Metric | Native (Rust) | WASM |
|--------|---:|---:|
| Build time | 265.0s | 610.0s |
| DB size | 2624.6 MB | 1916.8 MB |
| Nodes | 1,160,000 | 1,155,000 |
| Edges | 2,210,000 | 2,220,000 |
| Build time | 285.0s | 595.0s |
| DB size | 2752.5 MB | 2015.9 MB |
| Nodes | 1,160,000 | 1,160,000 |
| Edges | 2,205,000 | 2,205,000 |

### Incremental Rebuilds

| Version | Engine | No-op (ms) | 1-file (ms) |
|---------|--------|----------:|-----------:|
| 3.4.1 | native | 14 ↑17% | 316 ↓27% |
| 3.4.1 | wasm | 16 ↑60% | 487 ↓22% |
| 3.4.0 | native | 12 ~ | 432 ↑22% |
| 3.4.0 | wasm | 10 ↓17% | 621 ↑23% |
| 3.3.1 | native | 12 ↑33% | 353 ↑33% |
Expand Down Expand Up @@ -129,6 +137,8 @@ Extrapolated linearly from per-file metrics above.

| Version | Engine | fn-deps (ms) | fn-impact (ms) | path (ms) | roles (ms) |
|---------|--------|------------:|--------------:|----------:|----------:|
| 3.4.1 | native | 1.9 ↑12% | 2 ↑18% | 1.9 ↑19% | 21.6 ↑9% |
| 3.4.1 | wasm | 1.9 ↑6% | 2.1 ↑24% | 1.9 ↑19% | 20.7 ↓4% |
| 3.4.0 | native | 1.7 ↑21% | 1.7 ↑21% | 1.6 ↑23% | 19.9 ↑32% |
| 3.4.0 | wasm | 1.8 ↑38% | 1.7 ↑21% | 1.6 ↑23% | 21.5 ↑45% |
| 3.3.1 | native | 1.4 ↑56% | 1.4 ↑56% | 1.3 ↑44% | 15.1 ↑50% |
Expand Down Expand Up @@ -200,6 +210,109 @@ pre-parse that previously added ~388ms on native builds.

<!-- BENCHMARK_DATA
[
{
"version": "3.4.1",
"date": "2026-03-27",
"files": 473,
"wasm": {
"buildTimeMs": 5627,
"queryTimeMs": 12.2,
"nodes": 10956,
"edges": 20870,
"dbSizeBytes": 19996672,
"perFile": {
"buildTimeMs": 11.9,
"nodes": 23.2,
"edges": 44.1,
"dbSizeBytes": 42276
},
"noopRebuildMs": 16,
"oneFileRebuildMs": 487,
"oneFilePhases": {
"setupMs": 2,
"parseMs": 224.1,
"insertMs": 19,
"resolveMs": 1.6,
"edgesMs": 20.7,
"structureMs": 33.1,
"rolesMs": 42.1,
"astMs": 0.6,
"complexityMs": 0.7,
"cfgMs": 0.4,
"dataflowMs": 0.5,
"finalizeMs": 4.7
},
"queries": {
"fnDepsMs": 1.9,
"fnImpactMs": 2.1,
"pathMs": 1.9,
"rolesMs": 20.7
},
"phases": {
"setupMs": 23.1,
"parseMs": 2374.7,
"insertMs": 242.3,
"resolveMs": 14.7,
"edgesMs": 147.6,
"structureMs": 28.7,
"rolesMs": 72,
"astMs": 249.5,
"complexityMs": 330.8,
"cfgMs": 368,
"dataflowMs": 133.6,
"finalizeMs": 39.1
}
},
"native": {
"buildTimeMs": 2704,
"queryTimeMs": 11.7,
"nodes": 10982,
"edges": 20869,
"dbSizeBytes": 27303936,
"perFile": {
"buildTimeMs": 5.7,
"nodes": 23.2,
"edges": 44.1,
"dbSizeBytes": 57725
},
"noopRebuildMs": 14,
"oneFileRebuildMs": 316,
"oneFilePhases": {
"setupMs": 2.5,
"parseMs": 80.8,
"insertMs": 16.7,
"resolveMs": 1.5,
"edgesMs": 31.7,
"structureMs": 33.6,
"rolesMs": 40.4,
"astMs": 0.6,
"complexityMs": 0.6,
"cfgMs": 0.3,
"dataflowMs": 0.4,
"finalizeMs": 0.4
},
"queries": {
"fnDepsMs": 1.9,
"fnImpactMs": 2,
"pathMs": 1.9,
"rolesMs": 21.6
},
"phases": {
"setupMs": 20.9,
"parseMs": 1391.1,
"insertMs": 233.2,
"resolveMs": 12,
"edgesMs": 151.2,
"structureMs": 28.3,
"rolesMs": 64.4,
"astMs": 401.5,
"complexityMs": 28,
"cfgMs": 182.1,
"dataflowMs": 131.2,
"finalizeMs": 3.4
}
}
},
{
"version": "3.4.0",
"date": "2026-03-26",
Expand Down
Loading