Skip to content

Commit 09e6cf9

Browse files
author
LoCoBench Bot
committed
Merge remote-tracking branch 'origin/ralph/gapfill-nlqa'
# Conflicts: # configs/selected_benchmark_tasks.json # prd.json
2 parents 9d74730 + 6c65049 commit 09e6cf9

File tree

46 files changed

+4218
-627
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+4218
-627
lines changed

‎benchmarks/ccb_nlqa/nlqa-arch-001/environment/Dockerfile‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
FROM ubuntu:22.04
2+
3+
WORKDIR /workspace
4+
5+
# Install dependencies
6+
RUN apt-get update && apt-get install -y \
7+
git \
8+
curl \
9+
python3 \
10+
npm \
11+
&& rm -rf /var/lib/apt/lists/*
12+
13+
# Install Claude Code CLI
14+
RUN npm install -g @anthropic-ai/claude-code
15+
16+
# Clone Envoy at v1.34.0 release (pinned SHA)
17+
RUN git clone --filter=blob:none --no-checkout https://github.com/envoyproxy/envoy.git . && \
18+
git checkout d7809ba2b07fd869d49bfb122b27f6a7977b4d94 && \
19+
git config user.email "agent@example.com" && \
20+
git config user.name "Agent"
21+
22+
# Create output directories
23+
RUN mkdir -p /logs/agent /logs/verifier

‎benchmarks/ccb_nlqa/nlqa-arch-001/instruction.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Architecture Q&A: Envoy HTTP Filter Chain
2+
3+
**Repository:** envoyproxy/envoy
4+
**Task Type:** Architecture Q&A (investigation only — no code changes)
5+
6+
## Background
7+
8+
Envoy is a high-performance service proxy that processes HTTP requests through a layered filter architecture. Understanding how a request flows from initial TCP accept through to the upstream server is fundamental to working with Envoy's codebase.
9+
10+
## Questions
11+
12+
Answer ALL of the following questions about Envoy's HTTP request processing pipeline:
13+
14+
### Q1: Listener to Connection Manager
15+
16+
When a downstream client opens a TCP connection to Envoy, how does the listener hand off the connection to the HTTP connection manager? Specifically:
17+
- What mechanism selects which network filter chain to use for an incoming connection?
18+
- How is the HTTP connection manager (`ConnectionManagerImpl`) installed as a network filter?
19+
- What happens in `onData()` when the first bytes arrive?
20+
21+
### Q2: HTTP Filter Chain Creation and Iteration
22+
23+
Once HTTP request headers are parsed, how does Envoy build and iterate through the HTTP filter chain?
24+
- At what point in request processing is the HTTP filter chain created?
25+
- In what order are decoder filters invoked vs. encoder filters?
26+
- What return values can a filter use to control iteration (stop, continue, buffer)?
27+
28+
### Q3: Router and Upstream
29+
30+
How does the router filter (the terminal HTTP filter) forward requests to upstream servers?
31+
- How does the router obtain the target cluster and select a specific upstream host?
32+
- What is the role of `UpstreamRequest` and the upstream filter chain?
33+
- How does the response flow back through the filter chain to the downstream client?
34+
35+
### Q4: Architectural Boundaries
36+
37+
Explain the distinction between these two "filter chain" concepts in Envoy:
38+
- The network-level filter chain (managed by `FilterChainManager`)
39+
- The HTTP-level filter chain (managed by `FilterManager`)
40+
41+
Why are they separate, and how do they relate to each other?
42+
43+
## Output Requirements
44+
45+
Write your answer to `/logs/agent/investigation.md` with the following structure:
46+
47+
```
48+
# Envoy HTTP Filter Chain Architecture
49+
50+
## Q1: Listener to Connection Manager
51+
<answer with specific file paths, class names, and function references>
52+
53+
## Q2: HTTP Filter Chain Creation and Iteration
54+
<answer with specific file paths, class names, and function references>
55+
56+
## Q3: Router and Upstream
57+
<answer with specific file paths, class names, and function references>
58+
59+
## Q4: Architectural Boundaries
60+
<answer with specific file paths, class names, and function references>
61+
62+
## Evidence
63+
<consolidated list of key file paths and line references>
64+
```
65+
66+
## Constraints
67+
68+
- Do NOT modify any source files
69+
- Cite specific files, classes, and functions — avoid vague or speculative answers
70+
- Focus on the `source/common/http/`, `source/common/router/`, and `source/common/listener_manager/` directories

‎benchmarks/ccb_nlqa/nlqa-arch-001/task.toml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
version = "1.0"
2+
3+
[metadata]
4+
name = "nlqa-arch-001"
5+
description = "Explain Envoy's HTTP filter chain architecture: listener to upstream"
6+
license = "Apache-2.0"
7+
8+
[task]
9+
id = "nlqa-arch-001"
10+
repo = "envoyproxy/envoy"
11+
category = "architecture_qa"
12+
language = "cpp"
13+
difficulty = "hard"
14+
time_limit_sec = 1200
15+
16+
[verification]
17+
type = "test"
18+
command = "bash /tests/test.sh"
19+
20+
reward_type = "checklist"
21+
description = "Weighted checklist scoring Q&A report against ground-truth findings"
22+
23+
[environment]
24+
build_timeout_sec = 1800.0
25+
26+
[environment.setup_scripts]
27+
mcp_config = """#!/bin/bash
28+
# Setup Sourcegraph MCP if credentials provided
29+
if [ -n "$SOURCEGRAPH_ACCESS_TOKEN" ] && [ -n "$SOURCEGRAPH_URL" ]; then
30+
echo "Setting up Sourcegraph MCP configuration..."
31+
mkdir -p /root/.config/claude
32+
33+
cat > /root/.config/claude/mcp.json << 'MCPEOF'
34+
{
35+
"mcpServers": {
36+
"sourcegraph": {
37+
"command": "npx",
38+
"args": ["-y", "@sourcegraph/mcp-server"],
39+
"env": {
40+
"SRC_ACCESS_TOKEN": "$SOURCEGRAPH_ACCESS_TOKEN",
41+
"SOURCEGRAPH_URL": "$SOURCEGRAPH_URL"
42+
}
43+
}
44+
}
45+
}
46+
MCPEOF
47+
48+
echo "MCP configuration created"
49+
else
50+
echo "No Sourcegraph credentials provided, MCP disabled"
51+
fi
52+
exit 0
53+
"""

‎benchmarks/ccb_nlqa/nlqa-arch-001/tests/ground_truth.json‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
{
2+
"task_id": "nlqa-arch-001",
3+
"description": "Envoy HTTP filter chain architecture: listener to upstream",
4+
"weights": {
5+
"required_findings": 0.40,
6+
"file_references": 0.30,
7+
"causal_chain": 0.20,
8+
"negative_checks": 0.10
9+
},
10+
"required_findings": [
11+
{
12+
"id": "f1",
13+
"description": "Identifies ConnectionManagerImpl as a Network::ReadFilter",
14+
"patterns": ["ConnectionManagerImpl", "conn_manager_impl"],
15+
"weight": 0.15
16+
},
17+
{
18+
"id": "f2",
19+
"description": "Explains FilterChainManager selects network filter chain for incoming connections",
20+
"patterns": ["FilterChainManager|filter_chain_manager", "findFilterChain|find_filter_chain"],
21+
"weight": 0.15
22+
},
23+
{
24+
"id": "f3",
25+
"description": "Identifies FilterManager as the HTTP-level filter chain manager",
26+
"patterns": ["FilterManager|filter_manager"],
27+
"weight": 0.10
28+
},
29+
{
30+
"id": "f4",
31+
"description": "Explains decoder filters iterate forward (A→B→C) and encoder filters iterate in reverse (C→B→A)",
32+
"patterns": ["reverse|backward|opposite.*order|C.*B.*A|encoder.*reverse|reverse.*encoder"],
33+
"weight": 0.15
34+
},
35+
{
36+
"id": "f5",
37+
"description": "Identifies Router::Filter as the terminal HTTP decoder filter",
38+
"patterns": ["[Rr]outer.*[Ff]ilter|router\\.cc|[Tt]erminal.*filter|isTerminalFilter"],
39+
"weight": 0.15
40+
},
41+
{
42+
"id": "f6",
43+
"description": "Explains route resolution via refreshCachedRoute or route() callback",
44+
"patterns": ["refreshCachedRoute|cached.*route|callbacks_->route|routeEntry"],
45+
"weight": 0.10
46+
},
47+
{
48+
"id": "f7",
49+
"description": "Mentions UpstreamRequest and upstream filter chain",
50+
"patterns": ["UpstreamRequest|upstream_request", "upstream.*filter"],
51+
"weight": 0.10
52+
},
53+
{
54+
"id": "f8",
55+
"description": "Describes filter return status values (StopIteration, Continue, StopAllIterationAndBuffer)",
56+
"patterns": ["StopIteration|Stop.*Iteration|FilterHeadersStatus|StopAll.*Buffer|ContinueAndDontEndStream"],
57+
"weight": 0.10
58+
}
59+
],
60+
"file_references": [
61+
{
62+
"id": "r1",
63+
"description": "References conn_manager_impl.h or conn_manager_impl.cc",
64+
"patterns": ["source/common/http/conn_manager_impl\\.(h|cc)", "conn_manager_impl\\.cc", "conn_manager_impl\\.h"],
65+
"weight": 0.25
66+
},
67+
{
68+
"id": "r2",
69+
"description": "References filter_manager.h or filter_manager.cc",
70+
"patterns": ["source/common/http/filter_manager\\.(h|cc)", "filter_manager\\.h", "filter_manager\\.cc"],
71+
"weight": 0.25
72+
},
73+
{
74+
"id": "r3",
75+
"description": "References router.h or router.cc",
76+
"patterns": ["source/common/router/router\\.(h|cc)", "common/router/router\\.cc", "common/router/router\\.h"],
77+
"weight": 0.25
78+
},
79+
{
80+
"id": "r4",
81+
"description": "References filter_chain_manager_impl.h or listener_impl.h",
82+
"patterns": ["filter_chain_manager_impl\\.h|listener_impl\\.h|listener_manager/"],
83+
"weight": 0.15
84+
},
85+
{
86+
"id": "r5",
87+
"description": "References envoy/http/filter.h or envoy/network/filter.h interface files",
88+
"patterns": ["envoy/http/filter\\.h|envoy/network/filter\\.h|http/filter\\.h|network/filter\\.h"],
89+
"weight": 0.10
90+
}
91+
],
92+
"causal_chain": [
93+
{
94+
"id": "c1",
95+
"description": "Full request flow: listener accepts → FilterChainManager selects network filters → HCM receives bytes → codec parses HTTP → ActiveStream created → HTTP filter chain iterates → Router forwards to upstream",
96+
"patterns": [
97+
"listener|accept|TCP",
98+
"FilterChainManager|network.*filter|filter.*chain.*select",
99+
"ConnectionManagerImpl|HCM|conn.*manager|onData",
100+
"codec|dispatch|HTTP.*pars",
101+
"ActiveStream|newStream|decodeHeaders",
102+
"Router|upstream|cluster"
103+
],
104+
"ordered": true,
105+
"weight": 0.60
106+
},
107+
{
108+
"id": "c2",
109+
"description": "Response flow: upstream responds → Router encodeHeaders → encoder filters iterate in reverse → downstream client receives response",
110+
"patterns": [
111+
"upstream.*respon|response.*upstream",
112+
"encodeHeaders|encode.*header",
113+
"reverse|backward|encoder.*filter"
114+
],
115+
"ordered": true,
116+
"weight": 0.40
117+
}
118+
],
119+
"negative_checks": [
120+
{
121+
"id": "n1",
122+
"description": "Does NOT confuse network-level FilterChainManager with HTTP-level FilterManager as the same component",
123+
"patterns": ["FilterChainManager.*same.*FilterManager|FilterManager.*same.*FilterChainManager|network.*filter.*chain.*identical.*http.*filter"],
124+
"must_be_absent": true,
125+
"weight": 0.50
126+
},
127+
{
128+
"id": "n2",
129+
"description": "Does NOT claim the router filter directly writes to the upstream TCP connection (it uses UpstreamRequest/GenericConnPool)",
130+
"patterns": ["router.*direct.*write.*upstream.*socket|router.*direct.*TCP|router.*bypass.*connection.*pool"],
131+
"must_be_absent": true,
132+
"weight": 0.50
133+
}
134+
]
135+
}

0 commit comments

Comments
 (0)