feat: github-project-export (org Project v2 → TSV)

.cursor/rules/specify-rules.mdc

@@ -0,0 +1,42 @@
+---
+description: Project Development Guidelines
+globs: ["**/*"]
+alwaysApply: true
+---
+
+# tpm-utils Development Guidelines
+
+Auto-generated from all feature plans. Last updated: 2026-04-06
+
+## Active Technologies
+- Python 3.10+ (match `foc-pr-report`) + `requests` (uv); stdlib `json`, `csv` (tab delimiter), `argparse`, `pathlib`, `urllib.parse` (001-export-board-issues)
+- N/A (read-only export) (001-export-board-issues)
+
+- Python 3.10+ (match `foc-pr-report`) + `requests` (via uv); stdlib `csv`, `argparse`, `urllib.parse` (001-export-board-issues)
+
+## Project Structure
+
+```text
+foc-pr-report/
+github-project-export/
+```
+
+## Commands
+
+```bash
+# Run tests (from tool directory, e.g. github-project-export/)
+uv sync --group dev
+GITHUB_TOKEN=$(gh auth token) uv run pytest
+```
+
+## Code Style
+
+Python 3.10+ (match `foc-pr-report`): Follow standard conventions
+
+## Recent Changes
+- 001-export-board-issues: Added Python 3.10+ (match `foc-pr-report`) + `requests` (uv); stdlib `json`, `csv` (tab delimiter), `argparse`, `pathlib`, `urllib.parse`
+
+- 001-export-board-issues: Added Python 3.10+ (match `foc-pr-report`) + `requests` (via uv); stdlib `csv`, `argparse`, `urllib.parse`
+
+<!-- MANUAL ADDITIONS START -->
+<!-- MANUAL ADDITIONS END -->

README.md

@@ -78,6 +78,17 @@ cd foc-pr-report && uv sync && GITHUB_TOKEN=your_token uv run foc-pr-report -o r
 
 See [foc-pr-report/README.md](foc-pr-report/README.md).
 
+### 📤 GitHub project TSV export
+**Directory:** [github-project-export/](github-project-export/)
+
+Export board items from an organization Project (v2) to **TSV** using a **JSON** file (filter, columns, and stdout vs file path all live in the config—no duplicate CLI flags). Uses the REST list-items API with server-side `q`, shared with the FOC PR report client.
+
+```bash
+cd github-project-export && uv sync && GITHUB_TOKEN=$(gh auth token) uv run github-project-export examples/export.example1.json
+```
+
+See [github-project-export/README.md](github-project-export/README.md).
+
 ### 🎯 GitHub Milestone Manager
 **Directory:** [github-milestone-creator/](github-milestone-creator/)
 

github-project-export/.gitignore

@@ -0,0 +1,3 @@
+.venv/
+__pycache__/
+.pytest_cache/

github-project-export/README.md

@@ -0,0 +1,99 @@
+# GitHub project export (TSV)
+
+Export items from a **GitHub Organization Project (Projects v2)** to **TSV** using a **JSON configuration file**. Filtering uses the same **project search** syntax as the board, applied **server-side** via the REST API (see the [list project items](https://docs.github.com/en/rest/projects/items#list-items-for-an-organization-owned-project) endpoint).
+
+## Requirements
+
+- Python 3.10+
+- `uv`
+- GitHub token with **`read:project`** (and access to the org project), e.g.:
+
+  ```bash
+  export GITHUB_TOKEN=$(gh auth token)
+  gh auth refresh -s read:project
+  ```
+
+## Usage
+
+```bash
+cd github-project-export
+uv sync
+GITHUB_TOKEN=$(gh auth token) uv run github-project-export path/to/config.json
+```
+
+Optional flags (they do **not** duplicate JSON settings):
+
+- `--token` — PAT (default: `GITHUB_TOKEN`)
+- `--quiet` / `-q` — less progress on stderr
+- `--help`
+
+**Output**
+
+- If `outputFile` is **omitted** or **`null`**, TSV is written to **stdout** (errors and progress go to **stderr** unless `--quiet`).
+- If `outputFile` is a **non-empty string**, the file is **overwritten** with UTF-8 TSV.
+
+Exit codes: `0` success (including zero matching items → header-only TSV), `1` config/user error, `2` GitHub API error.
+
+## Testing
+
+- **Unit / default:** `uv sync --group dev && uv run pytest` — integration tests **skip** without a token.
+- **Live API (no mocks):** `GITHUB_TOKEN=$(gh auth token) uv run pytest` — runs [tests/fixtures/fixture_1_input.json](tests/fixtures/fixture_1_input.json) and compares stdout TSV to [tests/fixtures/fixture_1_output.tsv](tests/fixtures/fixture_1_output.tsv). Rows are compared after sorting by the `url` column so row order from GitHub does not matter.
+
+## JSON configuration
+
+| Key | Required | Description |
+|-----|----------|-------------|
+| `projectUrl` | Yes | Org project URL: `https://github.com/orgs/ORG/projects/N` |
+| `query` | No* | Single project filter string (`q`). If both `query` and `queryParts` exist, **non-empty `query` wins**. |
+| `queryParts` | No* | Array of strings; joined with spaces to form `q` when `query` is not used (or is empty). **Every element must be a string.** |
+| `fields` | Yes | Non-empty array of column headers. Order = TSV column order. |
+| `outputFile` | No | `null` or omit → stdout; non-empty string → file path. **`""` is invalid.** |
+
+\* You must end up with a non-empty filter: supply a non-empty `query` or a `queryParts` array that joins to a non-empty string.
+
+### Project field names
+
+Each `fields` entry is resolved **in order**:
+
+1. **Board field** — case-insensitive match to a field **display name** on that project (from GitHub’s fields API).
+2. **Synthetic column** — if not a board field, must match one of the documented synthetic keys (case-insensitive; see below).
+
+Duplicate headers that match **case-insensitively** are rejected.
+
+### Synthetic columns
+
+Values come from the linked **issue or pull request** (`content`), not from custom project fields. Recognized header aliases (internal keys):
+
+| User header examples | Meaning |
+|----------------------|---------|
+| `Repository`, `repo` | `repository.full_name` or parsed from `html_url` |
+| `url`, `link`, `html_url` | `html_url` |
+| `Kind`, `Type` | Issue vs pull request (`issue` / `pull_request`); use **`Kind`** when the board already has a *Type* column (e.g. Epic/Task) |
+| `Id`, `number` | Issue/PR number |
+| `title`¹ | Linked issue/PR **`title`** (REST issue / pull object on the item’s `content`) |
+
+¹ **Name collision:** Matching board fields is **case-insensitive**. If the project defines a board column *Title*, headers like `Title` or `title` use that column (GitHub encodes it as `fields[].value.raw`). Otherwise `title` is synthetic and reads `content.title` from the embedded issue/PR.
+
+### REST `content` (linked issue / pull request)
+
+[List items for a project](https://docs.github.com/en/rest/projects/items#list-items-for-an-organization-owned-project) embeds a full **Issue** or **Pull Request** object in `content`. Typical canonical properties:
+
+| REST property | Role | Synthetic TSV headers (aliases) |
+|---------------|------|--------------------------------|
+| `title` | Headline text | `title` (if no board field steals the name) |
+| `html_url` | Browser URL (`…/pull/411`, `…/issues/42`) | `html_url`, `url`, `link` |
+| `number` | Issue/PR number in the repo | `number`, `Id` |
+| `url` | API URL (`api.github.com/repos/…`) | *not mapped* (defaults keep `url` = `html_url` for spreadsheets) |
+
+### Zero matching items
+
+You still get a **TSV header row** and **no data rows**.
+
+### Examples
+
+- [examples/export.example1.json](examples/export.example1.json) — narrow filter, matches the live integration fixture.
+- [examples/export.example2.json](examples/export.example2.json) — broader columns (milestone, assignees, reviewers, etc.).
+
+## Implementation notes
+
+- Reuses **`foc_project14_client`** (`list_project_v2_field_ids_by_name`, `fetch_project_v2_items_rest`) from `foc-pr-report/foc_pr_report/`.

github-project-export/examples/export.example1.json

@@ -0,0 +1,15 @@
+{
+  "projectUrl": "https://github.com/orgs/FilOzone/projects/14",
+  "queryParts": [
+    "status:\"🎉 Done\"",
+    "411"
+  ],
+  "fields": [
+    "Repository",
+    "Id",
+    "Title",
+    "Status",
+    "url"
+  ],
+  "outputFile": null
+}

github-project-export/examples/export.example2.json

@@ -0,0 +1,22 @@
+{
+  "projectUrl": "https://github.com/orgs/FilOzone/projects/14",
+  "queryParts": [
+    "-status:\"🎉 Done\"", 
+    "milestone:\"M4.1: mainnet ready\",\"M4.2: mainnet GA\"",
+    "450"
+  ],
+  "fields": [
+    "Repository",
+    "Id",
+    "url",
+    "Title",
+    "Status",
+    "Kind",
+    "Milestone",
+    "Assignees",
+    "Reviewers",
+    "Cycle Theme",
+    "Dev Days Estimate"
+  ],
+  "outputFile": null
+}

github-project-export/github_project_export/__init__.py

@@ -0,0 +1,3 @@
+"""Export GitHub Org Projects v2 items to TSV from a JSON configuration file."""
+
+__version__ = "0.1.0"

github-project-export/github_project_export/board_url.py

@@ -0,0 +1,32 @@
+"""Parse GitHub organization project URLs."""
+
+from __future__ import annotations
+
+import re
+from typing import Tuple
+
+
+_ORG_PROJECT_RE = re.compile(
+    r"^https://github\.com/orgs/(?P<org>[^/]+)/projects/(?P<num>[0-9]+)/?(?:\?.*)?$",
+    re.IGNORECASE,
+)
+
+
+def parse_org_project_url(project_url: str) -> Tuple[str, int]:
+    """
+    Parse ``https://github.com/orgs/{org}/projects/{n}`` into (org_login, project_number).
+
+    Raises:
+        ValueError: if the URL does not match the expected organization-project pattern.
+    """
+    raw = (project_url or "").strip()
+    m = _ORG_PROJECT_RE.match(raw.split("#", 1)[0].strip())
+    if not m:
+        raise ValueError(
+            "projectUrl must look like "
+            "https://github.com/orgs/ORG_LOGIN/projects/N "
+            f"(got: {project_url!r})",
+        )
+    org = m.group("org")
+    num = int(m.group("num"))
+    return org, num

github-project-export/github_project_export/cli.py

@@ -0,0 +1,120 @@
+#!/usr/bin/env python3
+"""CLI: JSON config → GitHub Project v2 items → TSV."""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+import requests
+
+# foc_project14_client lives in foc-pr-report/foc_pr_report/ (moved there in PR #24)
+_FOC_PR_REPORT = Path(__file__).resolve().parents[2] / "foc-pr-report"
+if str(_FOC_PR_REPORT) not in sys.path:
+    sys.path.insert(0, str(_FOC_PR_REPORT))
+
+from github_project_export.config_schema import ConfigError, load_export_config
+from github_project_export.rest_export import FieldResolutionError, export_rows
+from github_project_export.tsv_write import write_tsv
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Export GitHub Organization Project v2 items to TSV using a JSON configuration file. "
+            "Board URL, filter, fields, and output path are read only from the config (not CLI flags)."
+        ),
+    )
+    parser.add_argument(
+        "config",
+        type=Path,
+        help="Path to JSON configuration file",
+    )
+    parser.add_argument(
+        "--token",
+        help="GitHub token (default: GITHUB_TOKEN environment variable)",
+    )
+    parser.add_argument(
+        "-q",
+        "--quiet",
+        action="store_true",
+        help="Less progress output on stderr",
+    )
+    args = parser.parse_args()
+
+    token = args.token or os.environ.get("GITHUB_TOKEN")
+    if not token:
+        print("Error: set GITHUB_TOKEN or pass --token", file=sys.stderr)
+        sys.exit(1)
+
+    try:
+        cfg = load_export_config(args.config)
+    except ConfigError as e:
+        print(f"Configuration error: {e}", file=sys.stderr)
+        sys.exit(1)
+
+    session = requests.Session()
+    session.headers.update(
+        {
+            "Authorization": f"Bearer {token}",
+            "Content-Type": "application/json",
+        }
+    )
+
+    try:
+        rows = export_rows(
+            session,
+            cfg.org,
+            cfg.project_number,
+            cfg.query,
+            cfg.fields,
+            verbose=not args.quiet,
+        )
+    except FieldResolutionError as e:
+        print(f"Configuration error: {e}", file=sys.stderr)
+        sys.exit(1)
+    except requests.HTTPError as e:
+        resp = e.response
+        detail = ""
+        if resp is not None:
+            try:
+                body = resp.json()
+                if isinstance(body, dict) and "message" in body:
+                    detail = f": {body['message']}"
+            except Exception:  # noqa: S110
+                detail = f": {resp.text[:500]}"
+        print(f"GitHub API error ({resp.status_code if resp else 'n/a'}){detail}", file=sys.stderr)
+        sys.exit(2)
+    except Exception as e:  # noqa: BLE001 — surface client errors as API-class
+        # foc_project14_client raises Exception for GraphQL/scope errors; normalize to stderr + exit 2
+        msg = str(e)
+        print(f"GitHub error: {msg}", file=sys.stderr)
+        if "INSUFFICIENT_SCOPES" in msg or "read:project" in msg:
+            sys.exit(2)
+        sys.exit(2)
+
+    out_stream = sys.stdout
+    out_path: Path | None = None
+    if cfg.output_path:
+        out_path = Path(cfg.output_path)
+
+    try:
+        if out_path is not None:
+            # Write file as UTF-8; do not leak file content to stdout on success
+            out_path.parent.mkdir(parents=True, exist_ok=True)
+            with out_path.open("w", encoding="utf-8", newline="") as fh:
+                write_tsv(list(cfg.fields), rows, fh)
+            if not args.quiet:
+                print(f"Wrote {cfg.output_path}", file=sys.stderr)
+        else:
+            # stdout: TSV only
+            write_tsv(list(cfg.fields), rows, out_stream)
+    except OSError as e:
+        print(f"Output error: {e}", file=sys.stderr)
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()