Skip to content

Commit 37fd73e

Browse files
committed
Add quality-scan skill and quality-loop command
**New quality tooling:** - .claude/skills/quality-scan/ - Comprehensive quality scanning skill with: - Documentation quality and junior dev friendliness checks - Critical bug detection (logic, caching, workflow) - CI optimization opportunities - Security vulnerability scanning - 7 specialized scan types - .claude/commands/quality-loop.md - Iterative quality improvement command: - Run quality scan - Fix issues in batches - Re-scan to verify fixes - Continue until all issues resolved **Purpose:** - Enable comprehensive codebase quality analysis - Identify bugs, documentation gaps, and optimization opportunities - Support iterative quality improvement workflow - Maintain consistent quality standards across Socket repositories **Usage:** ```bash # Run full quality scan /quality-loop # Use quality-scan skill directly Use Task tool with subagent_type='quality-scan' ``` Copied from socket-btm repository where these tools successfully identified and fixed 20+ issues including critical bugs, documentation barriers, and caching problems.
1 parent 057555e commit 37fd73e

File tree

2 files changed

+1183
-491
lines changed

2 files changed

+1183
-491
lines changed

.claude/skills/quality-scan/SKILL.md

Lines changed: 113 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: quality-scan
3-
description: Cleans up junk files (SCREAMING_TEXT.md, temp files) and performs comprehensive quality scans across codebase to identify critical bugs, logic errors, caching issues, and workflow problems. Spawns specialized agents for targeted analysis and generates prioritized improvement tasks. Use when improving code quality, before releases, or investigating issues.
3+
description: Validates structural consistency, cleans up junk files (SCREAMING_TEXT.md, temp files), and performs comprehensive quality scans across codebase to identify critical bugs, logic errors, caching issues, and workflow problems. Spawns specialized agents for targeted analysis and generates prioritized improvement tasks. Use when improving code quality, before releases, or investigating issues.
44
---
55

66
# quality-scan
@@ -26,8 +26,9 @@ This is Socket Security's binary tooling manager (BTM) that:
2626
2. **logic** - Algorithm errors, edge cases, type guards, off-by-one errors
2727
3. **cache** - Cache staleness, race conditions, invalidation bugs
2828
4. **workflow** - Build scripts, CI issues, cross-platform compatibility
29-
5. **security** - GitHub Actions workflow security (zizmor scanner)
30-
6. **documentation** - README accuracy, outdated docs, missing documentation
29+
5. **workflow-optimization** - CI optimization checks (build-required conditions on cached builds)
30+
6. **security** - GitHub Actions workflow security (zizmor scanner)
31+
7. **documentation** - README accuracy, outdated docs, missing documentation, junior developer friendliness
3132

3233
**Why Quality Scanning Matters:**
3334
- Catches bugs before they reach production
@@ -107,16 +108,15 @@ pnpm run update
107108

108109
<validation>
109110
**Expected Results:**
110-
- Dependencies updated in socket-cli
111+
- Dependencies updated in socket-btm
111112
- Report number of packages updated
112113
- Continue with scan even if update fails
113114

114115
**Track for reporting:**
115116
- Packages updated: N
116117
- Update status: Success/Failed (with warning)
117118

118-
**Important:** Only update dependencies in the current repository (socket-cli). Do NOT attempt to update sibling repositories (socket-btm, socket-sbom-generator, ultrathink) as this is out of scope and could have unintended side effects.
119-
</validation>
119+
**Important:** Only update dependencies in the current repository (socket-btm). Do NOT attempt to update sibling repositories (socket-sbom-generator, socket-cli, ultrathink) as this is out of scope and could have unintended side effects.</validation>
120120

121121
---
122122

@@ -195,7 +195,56 @@ find . -type f -name '*.log' \
195195

196196
---
197197

198-
### Phase 4: Determine Scan Scope
198+
### Phase 4: Structural Validation
199+
200+
<action>
201+
Run automated consistency checker to validate architectural patterns:
202+
</action>
203+
204+
**Validation Tasks:**
205+
206+
Run the consistency checker to validate monorepo structure:
207+
208+
```bash
209+
node scripts/check-consistency.mjs
210+
```
211+
212+
**The consistency checker validates:**
213+
1. **Required files** - README.md, package.json existence
214+
2. **Vitest configurations** - Proper mergeConfig usage
215+
3. **Test scripts** - Correct test patterns per package type
216+
4. **Coverage scripts** - Coverage setup where appropriate
217+
5. **External tools** - external-tools.json format validation
218+
6. **Build output structure** - Standard build/{mode}/out/Final/ layout
219+
7. **Package.json structure** - Standard fields and structure
220+
8. **Workspace dependencies** - Proper workspace:* and catalog: usage
221+
222+
<validation>
223+
**Expected Results:**
224+
- Errors: 0 (any errors should be reported as Critical findings)
225+
- Warnings: 2 or fewer (expected deviations documented in checker)
226+
- Info: Multiple info messages are normal (observations only)
227+
228+
**If errors found:**
229+
1. Report as Critical findings in the final report
230+
2. Include file:line references from checker output
231+
3. Suggest fixes based on checker recommendations
232+
4. Continue with remaining scans
233+
234+
**If warnings found:**
235+
- Report as Low findings (these are expected deviations)
236+
- Document in final report under "Structural Validation"
237+
238+
**Track for reporting:**
239+
- Number of packages validated
240+
- Number of errors/warnings/info messages
241+
- Any architectural pattern violations
242+
243+
</validation>
244+
245+
---
246+
247+
### Phase 5: Determine Scan Scope
199248

200249
<action>
201250
Ask user which scans to run:
@@ -206,8 +255,9 @@ Ask user which scans to run:
206255
2. **logic** - Logic errors (algorithms, edge cases, type guards)
207256
3. **cache** - Caching issues (staleness, races, invalidation)
208257
4. **workflow** - Workflow problems (scripts, CI, git hooks)
209-
5. **security** - GitHub Actions security (template injection, cache poisoning, etc.)
210-
6. **documentation** - Documentation accuracy (README errors, outdated docs)
258+
5. **workflow-optimization** - CI optimization (build-required checks for cached builds)
259+
6. **security** - GitHub Actions security (template injection, cache poisoning, etc.)
260+
7. **documentation** - Documentation accuracy (README errors, outdated docs)
211261

212262
**User Interaction:**
213263
Use AskUserQuestion tool:
@@ -236,7 +286,7 @@ If user requests non-existent scan type, report error and suggest valid types.
236286

237287
---
238288

239-
### Phase 5: Execute Scans
289+
### Phase 6: Execute Scans
240290

241291
<action>
242292
For each enabled scan type, spawn a specialized agent using Task tool:
@@ -283,59 +333,20 @@ Scan systematically and report all findings. If no issues found, state that expl
283333
- Cache scan: reference.md starting at line ~200
284334
- Workflow scan: reference.md starting at line ~300
285335
- Security scan: reference.md starting at line ~400
286-
- Documentation scan: reference.md starting at line ~810
336+
- Workflow-optimization scan: reference.md starting at line ~860
337+
- Documentation scan: reference.md starting at line ~1040
287338

288339
<validation>
289-
**Structured Output Validation:**
290-
291-
After each agent returns, validate output structure before parsing:
292-
293-
```bash
294-
# 1. Verify agent completed successfully
295-
if [ -z "$AGENT_OUTPUT" ]; then
296-
echo "ERROR: Agent returned no output"
297-
exit 1
298-
fi
299-
300-
# 2. Check for findings or clean report
301-
if ! echo "$AGENT_OUTPUT" | grep -qE '(File:.*Issue:|No .* issues found|✓ Clean)'; then
302-
echo "WARNING: Agent output missing expected format"
303-
echo "Agent may have encountered an error or found no issues"
304-
fi
305-
306-
# 3. Verify severity levels if findings exist
307-
if echo "$AGENT_OUTPUT" | grep -q "File:"; then
308-
if ! echo "$AGENT_OUTPUT" | grep -qE 'Severity: (Critical|High|Medium|Low)'; then
309-
echo "WARNING: Findings missing severity classification"
310-
fi
311-
fi
312-
313-
# 4. Verify fix suggestions if findings exist
314-
if echo "$AGENT_OUTPUT" | grep -q "File:"; then
315-
if ! echo "$AGENT_OUTPUT" | grep -q "Fix:"; then
316-
echo "WARNING: Findings missing suggested fixes"
317-
fi
318-
fi
319-
```
320-
321-
**Manual Verification Checklist:**
322-
- [ ] Agent output includes findings OR explicit "No issues found" statement
323-
- [ ] All findings include file:line references
324-
- [ ] All findings include severity level (Critical/High/Medium/Low)
325-
- [ ] All findings include suggested fixes
326-
- [ ] Agent output is parseable and structured
327-
328-
**For each scan completion:**
340+
For each scan completion:
329341
- Verify agent completed without errors
330-
- Extract findings from agent output (or confirm "No issues found")
342+
- Extract findings from agent output
331343
- Parse into structured format (file, issue, severity, fix)
332344
- Track scan coverage (files analyzed)
333-
- Log any validation warnings for debugging
334345
</validation>
335346

336347
---
337348

338-
### Phase 6: Aggregate Findings
349+
### Phase 7: Aggregate Findings
339350

340351
<action>
341352
Collect all findings from agents and aggregate:
@@ -374,7 +385,7 @@ interface Finding {
374385

375386
---
376387

377-
### Phase 7: Generate Report
388+
### Phase 8: Generate Report
378389

379390
<action>
380391
Create structured quality report with all findings:
@@ -389,6 +400,34 @@ Create structured quality report with all findings:
389400
**Files Scanned:** N
390401
**Findings:** N critical, N high, N medium, N low
391402

403+
## Dependency Updates
404+
405+
**Repositories Updated:** N/4
406+
- **socket-btm**: N packages updated
407+
- **socket-sbom-generator**: N packages updated (or "skipped - not found")
408+
- **socket-cli**: N packages updated (or "skipped - not found")
409+
- **ultrathink**: N packages updated (or "skipped - not found")
410+
411+
**Total Packages Updated:** N across all repositories
412+
413+
## Structural Validation
414+
415+
**Consistency Checker Results:**
416+
- Packages validated: 12
417+
- Errors: N (reported as Critical below)
418+
- Warnings: N (reported as Low below)
419+
- Info: N observations
420+
421+
**Validation Categories:**
422+
✓ Required files
423+
✓ Vitest configurations
424+
✓ Test scripts
425+
✓ Coverage scripts
426+
✓ External tools
427+
✓ Build output structure
428+
✓ Package.json structure
429+
✓ Workspace dependencies
430+
392431
## Critical Issues (Priority 1) - N found
393432

394433
### packages/node-smol-builder/src/patcher.mts:89
@@ -413,6 +452,8 @@ Create structured quality report with all findings:
413452

414453
## Scan Coverage
415454

455+
- **Dependency updates**: N/4 repositories updated (socket-btm, socket-sbom-generator, socket-cli, ultrathink)
456+
- **Structural validation**: 12 packages validated, 8 architectural patterns checked
416457
- **Critical scan**: N files analyzed in packages/node-smol-builder/, scripts/
417458
- **Logic scan**: N files analyzed (patch logic, build scripts)
418459
- **Cache scan**: N files analyzed (if applicable)
@@ -447,7 +488,7 @@ Create structured quality report with all findings:
447488

448489
---
449490

450-
### Phase 8: Complete
491+
### Phase 9: Complete
451492

452493
<completion_signal>
453494
```xml
@@ -460,13 +501,27 @@ Report these final metrics to the user:
460501

461502
**Quality Scan Complete**
462503
========================
504+
✓ Dependency updates: N/4 repositories updated (N packages)
505+
✓ Structural validation: N packages validated (N errors, N warnings)
463506
✓ Repository cleanup: N junk files removed
464507
✓ Scans completed: [list of scan types]
465508
✓ Total findings: N (N critical, N high, N medium, N low)
466509
✓ Files scanned: N
467510
✓ Report generated: Yes
468511
✓ Scan duration: [calculated from start to end]
469512

513+
**Dependency Update Summary:**
514+
- socket-btm: N packages updated
515+
- socket-sbom-generator: N packages updated (or "skipped - not found")
516+
- socket-cli: N packages updated (or "skipped - not found")
517+
- ultrathink: N packages updated (or "skipped - not found")
518+
519+
**Structural Validation Summary:**
520+
- Packages validated: 12
521+
- Consistency errors: N (included in critical findings)
522+
- Consistency warnings: N (included in low findings)
523+
- Architectural patterns checked: 8
524+
470525
**Repository Cleanup Summary:**
471526
- SCREAMING_TEXT.md files removed: N
472527
- Temporary test files removed: N
@@ -506,8 +561,9 @@ See `reference.md` for detailed agent prompts with structured tags:
506561
- **logic-scan** - Off-by-one errors, type guards, edge cases, algorithm correctness
507562
- **cache-scan** - Invalidation, key generation, memory management, concurrency
508563
- **workflow-scan** - Scripts, package.json, git hooks, CI configuration
564+
- **workflow-optimization-scan** - CI optimization checks (build-required on installation steps with checkpoint caching)
509565
- **security-scan** - GitHub Actions workflow security (runs zizmor scanner)
510-
- **documentation-scan** - README accuracy, outdated examples, incorrect package names, missing documentation
566+
- **documentation-scan** - README accuracy, outdated examples, incorrect package names, missing documentation, junior developer friendliness (beginner-friendly explanations, troubleshooting, getting started guides)
511567

512568
All agent prompts follow Claude best practices with <context>, <instructions>, <pattern>, <output_format>, and <quality_guidelines> tags.
513569

0 commit comments

Comments
 (0)