Skip to content

feat(server): add report-api server for dynamic metadata assembly#195

Draft
meyer9 wants to merge 6 commits into
mainfrom
meyer9/report-server
Draft

feat(server): add report-api server for dynamic metadata assembly#195
meyer9 wants to merge 6 commits into
mainfrom
meyer9/report-server

Conversation

@meyer9
Copy link
Copy Markdown
Collaborator

@meyer9 meyer9 commented Jun 4, 2026

Summary

Adds server/ — an HTTP server for the benchmark report UI — to this open-source repo. This implements the aggregation, comparison-synthesis, and caching logic.

No centralized metadata.json

The server never writes or reads a single merged metadata file. Instead:

  1. Each benchmark run writes its results under its own S3 prefix (<outputDir>/metrics-*.json, then <outputDir>/metadata.json last as the commit signal).
  2. The server lists all */metadata.json objects, merges them, applies retention, and serves the result dynamically.
  3. A content-fingerprint cache (per-file keyed on ETag + merged result with 1h TTL) keeps the steady-state cost to one ListObjectsV2 call per request.

The legacy metadata/metadata-<timestamp>.json layout is ignored — the server skips that prefix.

What's in server/

server/
├── cmd/main.go                     HTTP server entry point (Gin, CLI flags)
├── internal/
│   ├── config/                     CLI flags + config struct
│   ├── handlers/                   HTTP handlers (metadata, metrics, load-tests, health)
│   └── services/
│       ├── s3.go                   S3 listing, per-file cache, merged-result cache, retention
│       ├── comparison.go           Synthetic [Compare: Time] + [Compare: Versions] groups
│       └── cache.go                In-memory TTL cache (for per-run metrics files)
└── README.md

Endpoints:

Endpoint Description
GET /output/metadata.json Merged, retained, comparison-enriched run list
GET /output/<outputDir>/metrics-<role>.json Per-block timeseries for one run
GET /api/v1/load-tests/:network Load test run list
GET /api/v1/load-tests/:network/:timestamp Single load test result
GET /api/v1/health Health check

The comparison groups ([Compare: Time] / [Compare: Versions]) are generated server-side and injected into the metadata response. The existing report UI surfaces them in the existing dropdown with no frontend changes. On the chart comparison page, Show Line Per: TimeBucket (1d/1w/1m) or Show Line Per: ClientVersion splits the overlay.

Cache design:

  • Per-file byte cache keyed on <s3key>|<etag> — invalidated when a file is overwritten (new ETag), so a resubmitted or corrected run is never missed.
  • Merged result cache keyed on fingerprint of {key, etag} pairs + 1h TTL — the TTL handles the time.Now() dependence in retention and comparison-bucket synthesis; without it, the 14-day window and 1d/1w/1m buckets would be frozen at the first rebuild indefinitely.

Tests

14 unit tests ported from base-benchmarking covering the comparison synthesizer: time-bucket grouping, version grouping, latest-per-variant, no source mutation, monthly-prefix regression, network scoping, stable IDs, TimeBucket stamping, createdAt=now on synthetic clones, and canonicalTestName/slugify.

All existing runner/ tests still pass. go vet and gofmt clean.

Data contract

See docs/report-data-contract.md for what producers must write for runs to appear in the report.

type=routine
risk=low
impact=sev5

meyer9 added 2 commits June 4, 2026 11:34
Adds server/ — an HTTP server that replaces the per-environment static
metadata.json with a dynamically assembled response fetched directly
from per-run S3 files. This moves the aggregation, retention, and
comparison-synthesis logic into the open-source base/benchmark repo.

There is no central metadata file. The server:
- Lists all <outputDir>/metadata.json objects in S3 (one per run)
- Merges, deduplicates, and applies retention policy
- Appends synthetic [Compare: Time] and [Compare: Versions] groups
  so the existing report UI can compare runs across time windows
  or client versions without any frontend change
- Caches aggressively: per-file by ETag (invalidated when a run is
  rewritten), merged result by object fingerprint + 1h TTL
  (handles time.Now() dependence in retention/comparison)

The server is a straight port of protocols/base-benchmarking's
report-api, updated to use the per-run-directory S3 layout (#66 in
base-benchmarking). The base-benchmarking repo retains its copy
during the transition; a follow-up PR will remove it once this one
is deployed.

14 tests ported from base-benchmarking (comparison synthesizer tests).
All existing runner/ tests unaffected.
@cb-heimdall
Copy link
Copy Markdown
Collaborator

cb-heimdall commented Jun 4, 2026

🟡 Heimdall Review Status

Requirement Status More Info
Reviews 🟡 0/1
Denominator calculation
Show calculation
1 if user is bot 0
1 if user is external 0
2 if repo is sensitive 0
From .codeflow.yml 1
Additional review requirements
Show calculation
Max 0
0
From CODEOWNERS 0
Global minimum 0
Max 1
1
1 if commit is unverified 0
Sum 1

Allows the report server to serve directly from the directory written
by 'base-bench run --output-dir <dir>' without any S3 or MinIO setup.
This makes local development a one-command flow:

  report-server --local-dir ./output

The same merge/dedup/retention/comparison-synthesis pipeline runs
against local files as against S3, so reports look identical
regardless of backend.

Changes:
- BackendStorage interface: extracted from S3Service so handlers
  work against either backend
- LocalService: reads <outputDir>/metadata.json + metrics files
  from a local directory tree. Uses file mtime as the ETag
  equivalent for cache invalidation — a newly written metadata.json
  is visible on the next request.
- --local-dir flag (env: BASE_BENCH_API_LOCAL_DIR): mutually
  exclusive with --s3-bucket; both validated at startup
- S3BucketFlag: Required: true removed (validation moved to Validate())
- mergeRuns() + applyRetentionPolicy(): promoted to package-level
  functions so both S3Service and LocalService share the pipeline
- 6 new LocalService unit tests: GetMetadata, cache hit, cache
  invalidation on new file, GetObject, invalid dir, load tests

Verified: server starts with --local-dir, health returns 200,
metadata.json returns runs from local files with comparison groups.
Comment thread server/internal/services/local.go Fixed
Comment thread server/internal/services/local.go Fixed
Comment thread server/internal/services/local.go Fixed
GetObject, ListLoadTests, and GetLoadTest all accepted user-provided
values (HTTP path params) and passed them directly to filepath.Join
without validating that the resolved path stays within the root dir.
This allowed path traversal — e.g. GET /output/../../../etc/passwd.

Fix: safePath() resolves the joined path with filepath.Clean, then
checks it has the root as a prefix. Returns an error for any path
that escapes the root directory.

Also adds TestLocalService_PathTraversalBlocked covering the three
dangerous patterns: ../etc/passwd, ../../secret,
run-a/../../outside.
@meyer9
Copy link
Copy Markdown
Collaborator Author

meyer9 commented Jun 5, 2026

Fixed in 0b51c7c.

All three CodeQL path-traversal findings addressed via a new safePath(rel string) helper on LocalService. It uses filepath.Clean + strings.HasPrefix to verify the resolved path stays within the root ls.dir:

func (ls *LocalService) safePath(rel string) (string, error) {
    abs := filepath.Clean(filepath.Join(ls.dir, rel))
    base := filepath.Clean(ls.dir) + string(filepath.Separator)
    if abs != filepath.Clean(ls.dir) && !strings.HasPrefix(abs, base) {
        return "", fmt.Errorf("path %q escapes root directory", rel)
    }
    return abs, nil
}

GetObject, ListLoadTests, and GetLoadTest all call safePath before any filesystem access. Added TestLocalService_PathTraversalBlocked covering ../etc/passwd, ../../secret, and run-a/../../outside — all correctly return errors.

Comment thread server/internal/services/local.go Fixed
Comment thread server/internal/services/local.go Fixed
The previous safePath implementation used strings.HasPrefix against a
manually-constructed base path, which CodeQL did not recognize as a
path sanitizer and continued flagging the call sites.

Switched to filepath.Rel(root, abs): if the relative path from the
root to the resolved target starts with '..', the target is outside
the root. filepath.Rel is the idiomatic Go pattern for this check
and is more likely to be recognized by static analysis tools.
@meyer9
Copy link
Copy Markdown
Collaborator Author

meyer9 commented Jun 5, 2026

Updated in 346a48c.

The previous safePath implementation used strings.HasPrefix against a manually-constructed base path. CodeQL didn't recognize this as a sanitizer and continued flagging the call sites on the new scan.

Switched to filepath.Rel: if the resolved path has a relative path from the root that starts with ..., it's outside the root and the request is rejected. This is the idiomatic Go pattern for path-traversal defense and should be recognized by the scanner.

relPath, err := filepath.Rel(rootClean, abs)
if err != nil || strings.HasPrefix(relPath, "..") {
    return "", fmt.Errorf("path %q escapes root directory", rel)
}

All 7 LocalService tests still pass including TestLocalService_PathTraversalBlocked.

errcheck: defer result.Body.Close() -> defer result.Body.Close() //nolint:errcheck
  The S3 response body Close() is best-effort cleanup; the SDK
  documents the error as always nil.

staticcheck SA1019: add //nolint:staticcheck on aws-sdk-go v1 imports
  aws-sdk-go v1 is deprecated in favour of v2. The migration is a
  larger change tracked separately; the nolint directives keep CI
  green in the meantime.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants