Skip to content

Commit e1ad33b

Browse files
LoCoBench Botclaude
LoCoBench Bot
and
claude
committed
feat: US-006 - Create nlqa-debug-001 and nlqa-debug-002 debugging Q&A tasks
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent 11e2eef commit e1ad33b

File tree

12 files changed

+931
-1
lines changed

12 files changed

+931
-1
lines changed

‎benchmarks/ccb_nlqa/nlqa-debug-001/environment/Dockerfile‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
FROM ubuntu:22.04
2+
3+
ENV DEBIAN_FRONTEND=noninteractive
4+
5+
# Install git and dependencies
6+
RUN apt-get update && \
7+
apt-get install -y git ca-certificates && \
8+
rm -rf /var/lib/apt/lists/*
9+
10+
# Clone VS Code at pinned commit (v1.99.3)
11+
RUN git clone --filter=blob:none --no-checkout https://github.com/microsoft/vscode.git /workspace/vscode && \
12+
cd /workspace/vscode && \
13+
git checkout 17baf841131aa23349f217ca7c570c76ee87b957
14+
15+
WORKDIR /workspace/vscode

‎benchmarks/ccb_nlqa/nlqa-debug-001/instruction.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Debug Q&A: VS Code Extension Host Isolation
2+
3+
**Repository:** microsoft/vscode
4+
**Task Type:** Debug Q&A (investigation only — no code changes)
5+
6+
## Background
7+
8+
VS Code uses a multi-process architecture where extensions run in a separate process called the "extension host." Users have observed that when an extension crashes or encounters a fatal error, the main VS Code window continues to function normally — they can still edit files, open new tabs, and use the UI.
9+
10+
## Behavior Observation
11+
12+
**What happens:** When a VS Code extension crashes (e.g., throws an uncaught exception, segfaults in a native module, or hits a fatal error), the main VS Code window remains responsive and functional. The user may see a notification about the extension host restarting, but the editor UI doesn't freeze or crash.
13+
14+
**Why is this notable?** Many applications that load plugins or extensions crash entirely when a plugin fails. VS Code's architecture prevents extension failures from bringing down the whole application.
15+
16+
## Questions
17+
18+
Answer ALL of the following questions to explain this behavior:
19+
20+
### Q1: Process Isolation Architecture
21+
22+
How is the extension host process isolated from the main VS Code window process?
23+
- What mechanism is used to spawn the extension host as a separate process?
24+
- What is the relationship between the main process and the extension host process at the OS level (parent/child, detached, etc.)?
25+
- How does this isolation prevent a crash in one process from affecting the other?
26+
27+
### Q2: Communication Between Processes
28+
29+
Since the extension host and main window are separate processes, how do they communicate?
30+
- What IPC (inter-process communication) mechanism is used?
31+
- What happens to this communication channel when the extension host crashes?
32+
- How does the main window detect that the extension host has crashed vs. just being slow to respond?
33+
34+
### Q3: Crash Detection and Recovery
35+
36+
What happens when the extension host crashes?
37+
- What component in the main process detects the crash?
38+
- How does VS Code decide whether to automatically restart the extension host vs. showing a user prompt?
39+
- What prevents infinite restart loops if an extension crashes repeatedly?
40+
41+
### Q4: Isolation Mechanisms
42+
43+
What specific architectural features ensure the main window's lifecycle is independent of the extension host?
44+
- Are there any OS-specific isolation techniques (e.g., Windows vs. Linux/macOS)?
45+
- What prevents exceptions or errors in the extension host from propagating to the main process?
46+
- How does the architecture ensure the main window continues functioning even when extensions are unavailable?
47+
48+
## Output Requirements
49+
50+
Write your answer to `/logs/agent/investigation.md` with the following structure:
51+
52+
```
53+
# VS Code Extension Host Isolation
54+
55+
## Q1: Process Isolation Architecture
56+
<answer with specific file paths, class names, and function references>
57+
58+
## Q2: Communication Between Processes
59+
<answer with specific file paths, class names, and function references>
60+
61+
## Q3: Crash Detection and Recovery
62+
<answer with specific file paths, class names, and function references>
63+
64+
## Q4: Isolation Mechanisms
65+
<answer with specific file paths, class names, and function references>
66+
67+
## Evidence
68+
<consolidated list of key file paths and line references>
69+
```
70+
71+
## Constraints
72+
73+
- Do NOT modify any source files
74+
- Cite specific files, classes, and functions — avoid vague or speculative answers
75+
- Focus on the `src/vs/workbench/services/extensions/`, `src/vs/platform/utilityProcess/`, and `src/vs/workbench/api/node/` directories

‎benchmarks/ccb_nlqa/nlqa-debug-001/task.toml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
version = "1.0"
2+
3+
[metadata]
4+
name = "nlqa-debug-001"
5+
description = "Explain why VS Code extension host crashes don't kill the main window"
6+
license = "MIT"
7+
8+
[task]
9+
id = "nlqa-debug-001"
10+
repo = "microsoft/vscode"
11+
category = "debug_qa"
12+
language = "typescript"
13+
difficulty = "hard"
14+
time_limit_sec = 1200
15+
16+
[verification]
17+
type = "test"
18+
command = "bash /tests/test.sh"
19+
20+
reward_type = "checklist"
21+
description = "Weighted checklist scoring Q&A report against ground-truth findings"
22+
23+
[environment]
24+
build_timeout_sec = 1800.0
25+
26+
[environment.setup_scripts]
27+
mcp_config = """#!/bin/bash
28+
# Setup Sourcegraph MCP if credentials provided
29+
if [ -n "$SOURCEGRAPH_ACCESS_TOKEN" ] && [ -n "$SOURCEGRAPH_URL" ]; then
30+
echo "Setting up Sourcegraph MCP configuration..."
31+
mkdir -p /root/.config/claude
32+
33+
cat > /root/.config/claude/mcp.json << 'MCPEOF'
34+
{
35+
"mcpServers": {
36+
"sourcegraph": {
37+
"command": "npx",
38+
"args": ["-y", "@sourcegraph/mcp-server"],
39+
"env": {
40+
"SRC_ACCESS_TOKEN": "$SOURCEGRAPH_ACCESS_TOKEN",
41+
"SOURCEGRAPH_URL": "$SOURCEGRAPH_URL"
42+
}
43+
}
44+
}
45+
}
46+
MCPEOF
47+
48+
echo "MCP configuration created"
49+
else
50+
echo "No Sourcegraph credentials provided, MCP disabled"
51+
fi
52+
exit 0
53+
"""

‎benchmarks/ccb_nlqa/nlqa-debug-001/tests/ground_truth.json‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
{
2+
"task_id": "nlqa-debug-001",
3+
"description": "VS Code extension host crash isolation: why doesn't it kill the main window?",
4+
"weights": {
5+
"required_findings": 0.40,
6+
"file_references": 0.30,
7+
"causal_chain": 0.20,
8+
"negative_checks": 0.10
9+
},
10+
"required_findings": [
11+
{
12+
"id": "f1",
13+
"description": "Identifies that extension host runs as an Electron utility process (separate OS process)",
14+
"patterns": ["utility.*process|utilityProcess|UtilityProcess|WindowUtilityProcess"],
15+
"weight": 0.15
16+
},
17+
{
18+
"id": "f2",
19+
"description": "Explains extension host has separate PID from main process",
20+
"patterns": ["separate.*PID|separate.*process.*ID|isolated.*process|OS.*process|child.*process"],
21+
"weight": 0.10
22+
},
23+
{
24+
"id": "f3",
25+
"description": "Identifies ExtensionHostManager or NativeExtensionService as the main-side lifecycle manager",
26+
"patterns": ["ExtensionHostManager|NativeExtensionService|AbstractExtensionService"],
27+
"weight": 0.15
28+
},
29+
{
30+
"id": "f4",
31+
"description": "Identifies ExtensionHostCrashTracker as the crash tracking component",
32+
"patterns": ["ExtensionHostCrashTracker|crash.*track|_localCrashTracker"],
33+
"weight": 0.15
34+
},
35+
{
36+
"id": "f5",
37+
"description": "Explains IPC via MessagePort, MessageChannelMain, or message passing protocol",
38+
"patterns": ["MessagePort|MessageChannel|message.*passing|IMessagePassingProtocol|RPCProtocol"],
39+
"weight": 0.15
40+
},
41+
{
42+
"id": "f6",
43+
"description": "Describes crash detection via onExit event listener",
44+
"patterns": ["onExit|on.*exit|exit.*event|process.*exit|terminated.*unexpectedly"],
45+
"weight": 0.10
46+
},
47+
{
48+
"id": "f7",
49+
"description": "Explains 5-minute / 3-crash auto-restart policy",
50+
"patterns": ["5.*minute|3.*crash|TIME_LIMIT|CRASH_LIMIT|shouldAutomaticallyRestart"],
51+
"weight": 0.10
52+
},
53+
{
54+
"id": "f8",
55+
"description": "Mentions detached process on Windows (or OS-specific isolation)",
56+
"patterns": ["detached|windows.*detach|orphan|process.*group"],
57+
"weight": 0.10
58+
}
59+
],
60+
"file_references": [
61+
{
62+
"id": "r1",
63+
"description": "References nativeExtensionService.ts or abstractExtensionService.ts",
64+
"patterns": ["nativeExtensionService\\.ts|abstractExtensionService\\.ts|extensions/electron.*nativeExtensionService"],
65+
"weight": 0.25
66+
},
67+
{
68+
"id": "r2",
69+
"description": "References extensionHostManager.ts",
70+
"patterns": ["extensionHostManager\\.ts|extensions/common/extensionHostManager"],
71+
"weight": 0.20
72+
},
73+
{
74+
"id": "r3",
75+
"description": "References utilityProcess.ts or extensionHostStarter.ts",
76+
"patterns": ["utilityProcess\\.ts|extensionHostStarter\\.ts|platform/utilityProcess|platform/extensions.*extensionHostStarter"],
77+
"weight": 0.25
78+
},
79+
{
80+
"id": "r4",
81+
"description": "References localProcessExtensionHost.ts or extensionHostProcess.ts",
82+
"patterns": ["localProcessExtensionHost\\.ts|extensionHostProcess\\.ts|api/node/extensionHostProcess"],
83+
"weight": 0.20
84+
},
85+
{
86+
"id": "r5",
87+
"description": "References rpcProtocol.ts or ipc.js/ipc.net.js",
88+
"patterns": ["rpcProtocol\\.ts|ipc\\.js|ipc\\.net\\.js|base/parts/ipc"],
89+
"weight": 0.10
90+
}
91+
],
92+
"causal_chain": [
93+
{
94+
"id": "c1",
95+
"description": "Full crash handling flow: extension host crashes → OS delivers exit event → ExtensionHostManager detects via onExit → CrashTracker records crash → decides auto-restart or user prompt → startExtensionHosts() if auto-restart",
96+
"patterns": [
97+
"crash|fatal.*error|exception",
98+
"onExit|exit.*event|process.*terminat",
99+
"ExtensionHostManager|NativeExtensionService|detect",
100+
"CrashTracker|registerCrash|shouldAutomaticallyRestart",
101+
"startExtensionHosts|restart|relaunch"
102+
],
103+
"ordered": true,
104+
"weight": 0.70
105+
},
106+
{
107+
"id": "c2",
108+
"description": "Process spawning: ExtensionHostStarter.createExtensionHost() → WindowUtilityProcess.start() → Electron utility process spawned with separate PID",
109+
"patterns": [
110+
"createExtensionHost|ExtensionHostStarter",
111+
"WindowUtilityProcess|utilityProcess|start",
112+
"spawn|separate.*PID|isolated.*process"
113+
],
114+
"ordered": true,
115+
"weight": 0.30
116+
}
117+
],
118+
"negative_checks": [
119+
{
120+
"id": "n1",
121+
"description": "Does NOT claim the extension host runs in the same process as the main window (e.g., as a thread or web worker in the renderer)",
122+
"patterns": ["same.*process.*main|main.*window.*process.*extension.*host|renderer.*thread|single.*process"],
123+
"must_be_absent": true,
124+
"weight": 0.50
125+
},
126+
{
127+
"id": "n2",
128+
"description": "Does NOT claim crashes are caught via try/catch (they're detected via process exit events, not exception handlers)",
129+
"patterns": ["try.*catch.*crash|exception.*handler.*crash|catch.*fatal.*error"],
130+
"must_be_absent": true,
131+
"weight": 0.50
132+
}
133+
]
134+
}

0 commit comments

Comments
 (0)