You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
### 2) Gains persist across size bins, with strongest lift in 1M-5M proxy bucket
111
111
112
-
Curated GT deltas (`MCP - baseline`):
113
-
-`<1M`: F1@10 +0.1007, Total +0.1318
114
-
-`1M-5M`: F1@10 +0.2680, Total +0.2392
115
-
-`5M-20M`: F1@10 +0.0648, Total +0.0565
116
-
-`>20M`: F1@10 +0.1247, Total +0.1075
112
+
Curated GT deltas (`MCP - baseline`) by revised LOC size bands:
113
+
-`<400K` (n=15): F1@10 +0.2503, Total +0.2780
114
+
-`400K-2M` (n=31): F1@10 +0.2618, Total +0.2424
115
+
-`2M-8M` (n=143): F1@10 +0.1796, Total +0.1622
116
+
-`8M-40M` (n=74): F1@10 +0.0719, Total +0.0590
117
+
-`>40M` (n=3): F1@10 +0.0242, Total +0.0667
118
+
-`unknown` (n=63): F1@10 +0.0992, Total +0.1601
117
119
118
120
Interpretation: retrieval lift is not uniform, but MCP shows clear upside where task context is more distributed and retrieval-heavy.
119
121
120
-
Method note: I corrected an Org path-normalization bug in an earlier draft where some baseline paths were mismatched due to path shape differences (for example `repo/repo/path` vs `repo/path`).
122
+
Method note: I corrected an Org path-normalization bug in an earlier draft where some baseline paths were mismatched due to path shape differences (for example `repo/repo/path` vs `repo/path`). I also replaced SDLC size proxies with non-proxy repository size mapping for the size-bin slice in this version.
These slices indicate MCP retrieval gains are larger on multi-repo tasks than single-repo tasks in this snapshot.
1005
1007
1006
-
Methodology note: size bins here are metadata-driven proxies (`repo_set` fixture LOC totals for Org;`context_length` proxy forSDLC with fallback), so they should be interpreted as directional rather than exact physical repository size measurements.
1008
+
Methodology note: size bins here are no longer `context_length` proxies. Org tasks use fixture `loc_estimate` totals;SDLC tasks use repository size from GitHub metadata mapped into the same LOC bands (`docs/analysis/repo_size_bins_revised_20260303.json` conventions). `unknown` indicates tasks without resolved size metadata in this pass.
0 commit comments