Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new “PVR triage” taskflow to triage Private Vulnerability Reports (draft repository GHSAs): fetch the advisory, verify claims against code at the affected version (not HEAD), and emit a structured markdown triage report.
Changes:
- Introduces a new
pvr_ghsaMCP server (gh CLI-based) to fetch draft advisories, resolve version tags to SHAs, fetch file contents at a ref, and save a triage report. - Adds a new 5-step
pvr_triagetaskflow plus a dedicated analyst personality and model configuration. - Wires the new MCP server via a toolbox YAML.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/seclab_taskflows/toolboxes/pvr_ghsa.yaml | Adds toolbox wiring for the new PVR GHSA MCP server (stdio). |
| src/seclab_taskflows/taskflows/pvr_triage/pvr_triage.yaml | Defines the end-to-end PVR triage pipeline (fetch → verify → report → save). |
| src/seclab_taskflows/personalities/pvr_analyst.yaml | Adds a dedicated “PVR analyst” personality for evidence-based triage. |
| src/seclab_taskflows/mcp_servers/pvr_ghsa.py | Implements the gh CLI-backed MCP tools for advisory fetch/version resolution/file fetch/report save. |
| src/seclab_taskflows/configs/model_config_pvr_triage.yaml | Adds a taskflow-specific model mapping and temperatures. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…cking - pvr_ghsa.py: reject/withdraw/comment write-back tools, similarity search, read_triage_report; _gh_api now accepts a request body for PATCH/POST - reporter_reputation.py: new MCP server tracking per-reporter triage history in SQLite with record/history/score tools - pvr_triage.yaml: extended to 8 tasks (quality gate, patch status at HEAD, CVSS assessment, response draft, reputation update) - pvr_respond.yaml: write-back taskflow (action=reject|comment|withdraw) - pvr_triage_batch.yaml: inbox scoring taskflow with ranked markdown output - reporter_reputation.yaml, pvr_ghsa.yaml: new toolbox + confirm gates - tests/test_pvr_mcp.py: 23 unit tests, all passing - README.md: usage docs for all three taskflows
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated 8 comments.
Comments suppressed due to low confidence (3)
src/seclab_taskflows/mcp_servers/pvr_ghsa.py:454
- Docstring says matching is done on “header lines” and for both
vuln_typeandaffected_component, but the implementation currently scans the entire file and matches if either substring is present. Please update the docstring to reflect the actual behavior (or adjust the code to match the documented behavior) so taskflow expectations stay aligned.
"""
Search existing triage reports for similar vulnerability types and affected components.
Scans REPORT_DIR for *_triage.md files and performs case-insensitive substring
matching on the header lines for vuln_type and affected_component.
Returns a JSON list of matching reports with ghsa_id, verdict, quality, and path.
"""
if not REPORT_DIR.exists():
return json.dumps([])
matches = []
vuln_lower = vuln_type.lower()
component_lower = affected_component.lower()
for report_path in sorted(REPORT_DIR.glob("*_triage.md")):
# Skip batch queue reports and response drafts — only match individual GHSA triage reports
stem = report_path.stem # e.g. "GHSA-xxxx-xxxx-xxxx_triage"
if stem.startswith("batch_queue_") or stem.endswith("_response_triage"):
continue
try:
content = report_path.read_text(encoding="utf-8")
except OSError:
continue
content_lower = content.lower()
if vuln_lower not in content_lower and component_lower not in content_lower:
continue
tests/test_pvr_mcp.py:316
- This import is unused:
from seclab_taskflows.mcp_servers.reporter_reputation import get_reporter_history. The test callsself.backend.get_reporter_history(...)directly, so the extra import can be removed (or update the test to exercise the MCP tool wrapper if that was the intent).
from seclab_taskflows.mcp_servers.reporter_reputation import get_reporter_history
# Use the MCP tool wrapper to test the string return
src/seclab_taskflows/taskflows/pvr_triage/README.md:33
- The environment variable table lists
REPORTER_DB_DIRas required forpvr_respond, butpvr_respond.yamldoesn’t use the reporter_reputation toolbox or write to the DB. Please update the table to avoid implying an unnecessary requirement.
| `REPORT_DIR` | all | Directory where triage reports are written. Defaults to `./reports` |
| `LOG_DIR` | all | Directory for MCP server logs. Auto-detected via `platformdirs` if not set |
| `REPORTER_DB_DIR` | `pvr_triage`, `pvr_respond` | Directory for the reporter reputation SQLite database. Auto-detected if not set |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…lose, and reputation thresholds
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated 11 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Add from __future__ import annotations for Python 3.9 compat (dict|None, Path|str) - Fix REPORT_DIR empty-string handling: treat empty env var as unset - Add pagination to list_pvr_advisories; return JSON list consistently (empty list instead of string on no results) - Guard find_similar_triage_reports against empty/whitespace inputs; update docstring to reflect full-file scan - ReporterReputationBackend: use explicit "sqlite://" sentinel for in-memory; mkdir for filesystem paths instead of silent fallback - get_reporter_history MCP tool: return JSON list consistently (empty list instead of "No history" string) - pvr_ghsa.yaml: add default value for REPORT_DIR env var - pvr_triage_batch.yaml: remove dead already_triaged_penalty from scoring formula (entries are filtered out before scoring; aligns with SCORING.md) - Tests: remove unused sys/get_reporter_history imports; switch tempfile to TemporaryDirectory with tearDown cleanup; fix setUp to use sqlite:// sentinel; rename _patch_mcp_data_dir_pvr_ghsa -> _patch_report_dir - Docs: align AI_API_ENDPOINT default to https://api.githubcopilot.com across README, pvr_triage.yaml, model_config; remove pvr_respond from REPORTER_DB_DIR required-by list
pvr_ghsa.py:
- Remove backwards fallback quality regex in find_similar_triage_reports
(r"\b(High|Medium|Low)\b.*quality" matched wrong direction; primary regex sufficient)
- save_triage_report: return error on empty safe_name after sanitization
- fetch_file_at_ref: cap length at 500 lines; return error if start_line exceeds file length
- list_pvr_advisories: add max_pages=50 guard on pagination loop
reporter_reputation.py:
- Add UniqueConstraint("login","ghsa_id") to ReporterRecord table
- Add VALID_VERDICTS / VALID_QUALITIES constants; validate inputs in record_triage_result
- MCP record_triage_result tool: surface ValueError as error string to agent
pvr_triage.yaml:
- Task 5: also store triage_outcome {verdict, quality} in memcache after report generation
- Tasks 7+8: use triage_outcome.verdict/quality directly instead of re-parsing report text
pvr_respond.yaml, pvr_triage_batch.yaml:
- Fix AI_API_ENDPOINT comment to show actual default URL
tests:
- Add test_save_triage_report_empty_after_sanitization
- Add test_record_invalid_verdict_raises, test_record_invalid_quality_raises
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| GH_TOKEN: "{{ env('GH_TOKEN') }}" | ||
| LOG_DIR: "{{ env('LOG_DIR') }}" | ||
| REPORTER_DB_DIR: "{{ env('REPORTER_DB_DIR', '') }}" |
There was a problem hiding this comment.
The toolbox passes GH_TOKEN into the reporter_reputation MCP server even though the server code never uses it. This unnecessarily exposes a privileged token to a process that only needs local SQLite access; consider removing GH_TOKEN from this toolbox env block (keep only LOG_DIR / REPORTER_DB_DIR).
| GH_TOKEN: "{{ env('GH_TOKEN') }}" | |
| LOG_DIR: "{{ env('LOG_DIR') }}" | |
| REPORTER_DB_DIR: "{{ env('REPORTER_DB_DIR', '') }}" | |
| LOG_DIR: "{{ env('LOG_DIR') }}" | |
| REPORTER_DB_DIR: "{{ env('REPORTER_DB_DIR', '') }}" |
…integration - Add pvr_respond_batch.yaml for bulk response actions - pvr_triage.yaml: 3-path fast-close (high trust / skepticism / normal) - pvr_triage_batch.yaml: created_at in scored entries, Age column + tie-break sort - pvr_respond.yaml: call mark_response_sent on success - pvr_ghsa.py: add list_pending_responses, mark_response_sent tools - README.md: update for taskflow 4, batch/respond/output-file sections - run_pvr_triage.sh: add respond_batch subcommand - tests: expand to 32 passing tests covering new tools and taskflows
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Reads REPORT_DIR/{ghsa_id}_triage.md and returns its content. | ||
| Returns an error string if the file does not exist. | ||
| """ | ||
| safe_name = "".join(c for c in ghsa_id if c.isalnum() or c in "-_") |
There was a problem hiding this comment.
read_triage_report sanitizes ghsa_id, but if it sanitizes to an empty string it will attempt to read REPORT_DIR/_triage.md. For consistency with save_triage_report / mark_response_sent (which error on empty-after-sanitization) and to avoid surprising reads, add an explicit empty check and return a clear error when safe_name is empty.
| safe_name = "".join(c for c in ghsa_id if c.isalnum() or c in "-_") | |
| safe_name = "".join(c for c in ghsa_id if c.isalnum() or c in "-_") | |
| if not safe_name: | |
| return "Error: ghsa_id produced an empty filename after sanitization" |
PT009/PT027: test_pvr_mcp.py uses unittest.TestCase style assertions throughout. Converting 71 assertions is not warranted; suppress via per-file-ignores for tests/*. PLC0415: imports inside setUp/test methods are deliberate — needed for the patch.object pattern that avoids early import side-effects. SIM105: pvr_ghsa.py:315 try/except assigns to a variable on success; contextlib.suppress cannot capture that assignment. False positive; suppress globally. Also carry forward PLW0603 (global statement for module-level state), S101 (assert in tests), and SLF001 (private member access in tests) suppressions from the shell-toolbox branch.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
|
|
||
| import json | ||
| import logging | ||
| import os |
There was a problem hiding this comment.
os is imported but not used in this module. Please remove the unused import (or use it if needed) to keep the MCP server code lint-clean.
| import os |
GitHub API PATCH only accepts published/closed/draft. rejected and withdrawn are not valid PATCH states. Remove withdraw_pvr_advisory (withdrawn is read-only, reporter-initiated) and fix reject_pvr_advisory to use closed.
Accepting a PVR (confirmed vulnerability) moves the advisory from triage to draft state. New tool accept_pvr_advisory PATCHes state=draft and posts a comment. Added as action=accept throughout respond/respond_batch taskflows, runner script, and docs.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| Retrieve "pvr_queue" from memcache. | ||
|
|
||
| Extract owner and repo from "{{ globals.repo }}" (format: owner/repo). | ||
|
|
||
| For each advisory in pvr_queue: |
There was a problem hiding this comment.
list_pvr_advisories returns a JSON string, but this task later treats pvr_queue as an iterable list. To avoid the agent iterating over a raw string, please add an explicit step to parse the JSON returned by list_pvr_advisories (and/or the memcache value) into a list before looping.
| existing_desc = adv_data.get("description", "") or "" | ||
| updated_desc = existing_desc + f"\n\n## Maintainer Response\n\n{body}" | ||
| _, patch_err = _gh_api(adv_path, method="PATCH", body={"description": updated_desc}) | ||
| if patch_err: | ||
| return f"Error updating advisory description: {patch_err}" | ||
| return "Comment appended to advisory description (comments API unavailable)." |
There was a problem hiding this comment.
Fallback behavior appends a new "## Maintainer Response" section to the advisory description every time the comments API is unavailable. This can easily lead to duplicated responses on retries or batch runs. Consider making the fallback idempotent by replacing an existing "Maintainer Response" section (if present) or adding a unique marker and updating in place instead of always appending.
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Helpers: patch mcp_data_dir so imports don't fail in CI (no platformdirs dir) |
There was a problem hiding this comment.
This comment says the helper patches mcp_data_dir, but _patch_report_dir() actually patches REPORT_DIR in pvr_ghsa. Please update the comment to match what’s being patched (or rename the helper) to avoid confusion when debugging CI import failures.
| # Helpers: patch mcp_data_dir so imports don't fail in CI (no platformdirs dir) | |
| # Helpers: patch pvr_ghsa REPORT_DIR so imports don't fail in CI (no platformdirs dir) |
| Call get_reporter_score with that login and store the result as reporter_score. | ||
|
|
||
| Call find_similar_triage_reports with: | ||
| - vuln_type: pvr_parsed.vuln_type | ||
| - affected_component: pvr_parsed.affected_component |
There was a problem hiding this comment.
get_reporter_score and find_similar_triage_reports both return JSON strings from MCP tools (see their implementations). This prompt uses reporter_score.recommendation and treats similar_reports as a list, which will be unreliable unless the agent explicitly parses the returned JSON first. Please add an explicit step to JSON-parse both tool outputs before accessing fields / storing them in quality_gate.
The /repos/.../security-advisories/{ghsa_id}/comments endpoint does not
exist in the public API (404). Remove _post_advisory_comment and
add_pvr_advisory_comment entirely. Strip the comment parameter from
accept_pvr_advisory and reject_pvr_advisory — both now only apply the
state transition (triage→draft and triage→closed respectively).
Remove the comment action from pvr_respond and pvr_respond_batch; valid
actions are now accept and reject only.
Add MANUAL_RESPONSE.md with steps for posting the generated response
draft manually via the advisory URL after the state transition is applied.
Taskflow for triaging Private Vulnerability Reports arriving as GHSAs in triage state.
Fetches the advisory, verifies the vulnerability claim against source code at the affected version, and writes a structured markdown triage report with response draft.
Files
New MCP servers:
mcp_servers/pvr_ghsa.py— advisory fetch, version-to-SHA resolution, file fetch at ref, write-back (accept/reject/comment), similarity search, report read/save, pending-response listing, sent-marker creationmcp_servers/reporter_reputation.py— per-reporter triage history in SQLite; record/history/score toolsNew toolboxes:
toolboxes/pvr_ghsa.yaml— confirm-gated for write-back toolstoolboxes/reporter_reputation.yamlNew/updated taskflows:
taskflows/pvr_triage/pvr_triage.yaml— 8 tasks: quality gate with reputation-gated fast-close decision tree, patch status at HEAD, CVSS assessment, response draft, reputation updatetaskflows/pvr_triage/pvr_respond.yaml— write-back taskflow (action=accept|reject|comment); callsmark_response_senton successtaskflows/pvr_triage/pvr_triage_batch.yaml— inbox scoring with Age (days) column and tie-break sort by created_attaskflows/pvr_triage/pvr_respond_batch.yaml— bulk respond: scans REPORT_DIR for unsent drafts and posts them in one sessiontaskflows/pvr_triage/README.md— full usage docs including all four taskflowstaskflows/pvr_triage/SCORING.md— authoritative scoring reference with three-path fast-close decision table and reputation x fast-close matrixTests:
tests/test_pvr_mcp.py— 32 unit tests (MCP tools + YAML structure)Scripts:
scripts/run_pvr_triage.sh— local test/demo scriptUsage
Requires
GH_TOKEN(repo + security_events scope),AI_API_TOKEN,AI_API_ENDPOINT.Advisory state transitions
accept— triage → draft (vulnerability confirmed, publishing intent)comment— no state change (request more info or acknowledge)reject— triage → closed (invalid or low-quality report)Maintainer quality-of-life features
Age column in batch report —
pvr_triage_batchnow showsAge (days)for each advisory (today minus created_at). Ties in priority score are broken by age ascending (oldest first).Reporter trust gates fast-close — the quality gate in
pvr_triageuses a three-path decision tree:Bulk respond —
pvr_respond_batchscansREPORT_DIRfor drafts without a*_response_sent.mdmarker and posts them all in one session. Bothpvr_respondandpvr_respond_batchwrite the sent marker on success so re-runs are idempotent.