feat: add video frame format render option

[jrusso1020]

Takeover of #1142 — original work by @xuelongmu, rebased onto current main with merge conflicts resolved. The feature commit's authorship is preserved (Author: Xuelong Mu); a follow-up commit by me addresses PR-review nits. #1142 was approved and merge-greenlit, blocked only on a rebase; this carries it forward. Re-validated after the rebase (see Test plan).

What

Adds a first-class render option for source-video frame extraction:

hyperframes render --video-frame-format auto|jpg|png

and the matching programmatic config field RenderConfig.videoFrameFormat?: "auto" | "jpg" | "png". Default remains auto, preserving existing behavior: alpha or alpha-capable sources extract as PNG, opaque sources extract as JPG.

Why

When compositing screen recordings (e.g. an iPhone UI recording), the timeline/browser preview looks close to the source while the rendered MP4 shifts saturated UI colors. The shift is already present in the captured browser PNG frames — the root cause is that opaque source videos were extracted as JPEG before browser capture, and JPEG's RGB→YUV→RGB roundtrip shifts saturated colors (this is intrinsic to JPEG, not chroma subsampling — JPEG-444 shifts as much as 420). PNG stays in RGB end-to-end, so it's the only thing that preserves the pixel. Changing final H.264 color tags can't help because the shift happens earlier in the pipeline.

Representative sample on a saturated red UI indicator: source RGB(221,56,46) → JPEG extraction ~RGB(186,51,58) (maxΔ≈16) vs PNG extraction ~RGB(220,55,45) (maxΔ≈1).

auto stays the default deliberately: PNG-for-everything is ~8× slower extraction and ~5× larger intermediate frames on photographic content (where the JPEG shift is perceptually invisible), and on Lambda the frame bloat risks /tmp exhaustion. Opt-in is the correct shape.

How

• Adds --video-frame-format auto|jpg|png to hyperframes render and RenderConfig.videoFrameFormat.
• Threads the option through local render, Docker render, producer server render, and distributed planning.
• resolveFrameFormat: explicit png/jpg honored for opaque videos; auto/undefined preserves existing behavior; alpha / alpha-capable sources are forced to PNG, and that override now runs before the explicit-format check.
• Extraction cache key incorporates the effective frame format, so JPG and PNG frame caches can't collide.
• Documents the option in CLI and rendering docs.

⚠️ Behavior change for SDK consumers of extractAllVideoFrames

The alpha override in resolveFrameFormat is newly load-bearing against an explicit "jpg" request. Previously, calling extractAllVideoFrames(..., { format: "jpg" }) on an alpha-capable source returned JPEG and silently dropped the alpha channel; now alpha-capable sources are forced to PNG regardless of an explicit "jpg". This is the safer contract (tested at videoFrameExtractor.test.ts: hasAlpha=true + explicit "jpg" → "png"). No in-tree caller passes explicit "jpg" (the production path flows videoFrameFormat ?? "auto"), so this is observable only to external SDK consumers — flagging for changelogs.

Rebase note: since #1142 was opened, main moved the distributed-config validation out of packages/aws-lambda/src/sdk/validateConfig.ts into packages/producer/src/services/distributed/renderConfigValidation.ts, and removed the producer videoFrameExtractor re-export barrel. The videoFrameFormat allow-list validation was relocated accordingly; consumers import VideoFrameFormat directly from @hyperframes/engine. Also re-threaded the option into server.ts (which refactored to buildRenderJobConfig) and merged the docs/test hunks with the GIF feature that landed in between.

Review follow-up (commit 2): consolidated the ["auto","jpg","png"] allow-list — previously declared in three places (CLI flag parser, producer server, distributed-config validator) — into a single VIDEO_FRAME_FORMATS constant + isVideoFrameFormat type guard exported from @hyperframes/engine, imported by all three call sites. Behavior unchanged; also removes two as RenderConfig[...] casts in favor of the guard.

Test plan

Re-ran after both the rebase and the consolidation commit (all green):

• bun run --filter @hyperframes/engine test -- src/services/videoFrameExtractor.test.ts — 38 pass (incl. the PNG color-fidelity regression, run against ffmpeg)

• bun run --filter @hyperframes/cli test -- src/commands/render.test.ts src/utils/dockerRunArgs.test.ts — 53 pass

• bun run --filter @hyperframes/aws-lambda test -- src/sdk/validateConfig.test.ts — 35 pass (incl. relocated videoFrameFormat allow-list)

• typecheck clean for engine, producer, cli, aws-lambda

• bunx oxfmt --check + bunx oxlint clean on all changed files

• Unit tests added/updated

• Manual testing performed

• Documentation updated (if applicable)

Closes #1142

[jrusso1020]

Carrying forward the review context from #1142 (which I approved on 2026-06-03 and Miguel merge-greenlit pending rebase). This PR is that exact change, rebased onto current main with conflicts resolved and re-validated — Xuelong Mu's commit authorship is preserved (signed/Verified).

Diff vs. the original #1142, all forced by main moving underneath it:

• Distributed-config validation moved from aws-lambda/.../validateConfig.ts → producer/.../distributed/renderConfigValidation.ts; the videoFrameFormat allow-list rule was relocated there. validateConfig.test.ts (35 pass) exercises it through the re-export.
• Producer videoFrameExtractor re-export barrel was deleted on main; consumers now import VideoFrameFormat straight from @hyperframes/engine.
• Re-threaded the option into server.ts (now buildRenderJobConfig) and merged the docs/test hunks with the GIF feature that landed in between.

Re-ran post-rebase: engine 38, cli 53, aws-lambda 35 tests green; typecheck clean on engine/producer/cli/aws-lambda; oxfmt + oxlint clean.

Since I opened this PR I can't self-approve it — @miguel-heygen, this is ready for your stamp (or a quick re-review). The root-cause analysis and fix are unchanged from #1142.

[vanceingalls]

James — review on the takeover.

Summary of intent

Add a first-class --video-frame-format auto|jpg|png CLI flag plus the matching RenderConfig.videoFrameFormat field, threaded through local / Docker / producer-server / distributed-plan paths, so colour-sensitive UI recordings can opt into PNG source-frame extraction and dodge the JPEG RGB→YUV→RGB chroma drift. Default auto preserves historical behaviour; the extraction cache key now incorporates the effective format so JPG and PNG caches cannot collide. A new ffmpeg-backed regression test synthesises a saturated-red UI fixture and proves the PNG path keeps maxΔ ≤ 5 from source while the JPG path does not.

Takeover seam

Predecessor #1142 by xuelongmu was approved 2026-06-03 with a thorough JPEG-420/444/PNG colour table; merge-greenlit pending rebase; it then sat 260 commits behind main. The OAuth workflow-scope wall blocked a direct rebase-push to the original branch, so #1481 carries it forward on a fresh branch with Author: Xuelong Mu preserved on the single commit. The takeover's job, then, is precisely re-validation after rebase.

A single commit, 1462ee26, attributed to Xuelong Mu — structurally "one rebased commit" rather than "carried + new". The substantive engineering surface in this PR is the rebase resolution: relocating the distributed-config allow-list from packages/aws-lambda/src/sdk/validateConfig.ts into the newer packages/producer/src/services/distributed/renderConfigValidation.ts, re-threading the option through the refactored server.ts buildRenderJobConfig boundary, importing VideoFrameFormat directly from @hyperframes/engine now that the producer's videoFrameExtractor re-export barrel is gone, and merging the docs/test hunks with the intervening GIF feature. The prior reviewer ask on #1142 (a per-<video> data-frame-format attribute as a future refinement) was explicitly non-blocking and outside this PR's scope.

Stale-base posture

Base 78cce00c, head 1462ee26, merge-base 2d48369c. Only 8 commits behind current main. Of those, exactly one — 646ffff9 (#1478, the OpenRouter capture-captioning feature) — touches a file this PR also touches (docs/packages/cli.mdx). Per the diff-artefact-vs-real-deletion rule: I read the cli.mdx OpenRouter line at the merge-base (absent) and at PR head (still absent) — so it is a diff artefact, not a real deletion. Squash-merge will not revert anything. No stale-base squash hazard. A pre-merge rebase would still be courteous given the 8-commit gap, but it is not load-bearing.

Per-file findings

• packages/engine/src/services/videoFrameExtractor.ts:438-444 — resolveFrameFormat precedence reorder. Alpha-override now runs before the explicit-requested check. Pre-PR an explicit "jpg" on an alpha-capable source would have returned "jpg" and silently lost the alpha channel; post-PR it is forced to "png". This is the safer contract and is tested at videoFrameExtractor.test.ts:107-110 (hasAlpha=true + explicit "jpg" → "png"). The PR body's "alpha / alpha-capable codecs still force PNG regardless" slightly understates this — the word "still" implies prior parity, when in fact the alpha override is newly load-bearing against explicit-jpg callers. No in-tree caller currently passes explicit "jpg" (the production path at extractVideosStage.ts:197 flows videoFrameFormat ?? "auto"), so the behaviour change is observable only to external SDK consumers of extractAllVideoFrames. Worth a one-line note in the PR body for SDK changelogs; not a code blocker.

• Allow-list duplication across boundaries. The string set ["auto","jpg","png"] is independently declared three times — packages/cli/src/commands/render.ts:104 (VIDEO_FRAME_FORMATS), packages/producer/src/services/distributed/renderConfigValidation.ts:55 (ALLOWED_VIDEO_FRAME_FORMATS), and inline at packages/producer/src/server.ts:131 (["auto", "jpg", "png"].includes(...)). This is the duplicate-validation-at-boundaries shape from the band-aid bar — three runtime allow-lists for the same set, three places to forget when (say) a "webp" extraction format lands later. The fix shape is a single VIDEO_FRAME_FORMATS runtime constant exported alongside the VideoFrameFormat type from @hyperframes/engine, imported by all three call sites. Soft nit on this PR — the surface is small, the drift is currently zero, and you inherited the shape from the original — but a worthwhile follow-up before a fourth boundary appears.

• packages/cli/src/utils/dockerRunArgs.ts:125-127 — "auto" is correctly omitted from the forwarded args, "jpg" and "png" forwarded explicitly; matches the inner CLI default at render.ts:194 (default: "auto") so round-tripping auto doesn't surface as an undefined. Symmetry holds. Worth keeping the !== "auto" guard rather than if (format) — it is exactly the safety predicate the band-aid bar wants on this shape.

• packages/producer/src/services/render/stages/extractVideosStage.ts:197 — format: job.config.videoFrameFormat ?? "auto". This is the only in-tree caller of extractAllVideoFrames that supplies a format, and it always supplies one. Good.

• Cache key collision. Regression test at videoFrameExtractor.test.ts:531-571 exercises the three-way ladder: jpg-default cache-miss → png-explicit cache-miss → png-explicit cache-hit, asserting framePath extensions and cacheHits / cacheMisses counters per phase. Closes the lone load-bearing risk of "explicit png extracted, but cache served you stale jpg pixels". Good.

• Test plan vs CI. The "re-ran after the rebase (all green)" claim matches CI: Test, Typecheck, SDK: unit + contract + smoke, Lint, Format, CLI smoke (required), Build, Studio: load smoke, Test: runtime contract, Validate docs, Fallow audit, preview-regression, player-perf — all pass.

CI

All required green; only the slow Windows render verification shards and regression-shards 1–8 are still pending. None of those are required. CodeQL Java/TS analysis passes.

Verdict

Clear with nits. A faithful rebase of #1142, with a single behaviour change (resolveFrameFormat alpha-override precedence) that is strictly safer than the prior contract and is explicitly tested. CI is green on all required checks. No stale-base squash hazard. The two nits — (a) PR body wording around the strengthened alpha override, (b) the three-place allow-list duplication — are both soft; neither is load-bearing for this PR, but the allow-list one is worth a small follow-up before a fourth extraction format lands.

Cleared at 1462ee26; if HEAD moves materially before stamp, re-ping for re-review. Handing to Vai for stamp.

Review by Via

[jrusso1020]

Thanks for the thorough review @via — both nits addressed:

Nit (b) — allow-list duplication (fixed in code, commit 47284a1e). Hoisted the ["auto","jpg","png"] set into a single VIDEO_FRAME_FORMATS constant + an isVideoFrameFormat type guard exported from @hyperframes/engine, and routed all three boundaries through them:

• render.ts — dropped the local VIDEO_FRAME_FORMATS/isVideoFrameFormat, imports from engine
• renderConfigValidation.ts — dropped ALLOWED_VIDEO_FRAME_FORMATS, uses the guard + constant
• server.ts — replaced the inline ["auto","jpg","png"].includes(...) with the guard (also drops the as RenderConfig[...] cast)

A fourth boundary now has one place to update. Behavior unchanged — engine 38 / cli 53 / aws-lambda 35 tests still green; fallow audit clean on the changed files.

Nit (a) — strengthened alpha override. Good catch on the "still" wording understating it. Added an explicit "Behavior change for SDK consumers of extractAllVideoFrames" callout to the PR body: an explicit "jpg" on an alpha-capable source is now forced to PNG (previously returned JPEG and silently dropped alpha). No in-tree caller passes explicit "jpg", so it's SDK-consumer-only — flagged for changelogs as you suggested.

HEAD moved (1462ee26 → 47284a1e), so this needs a re-verify before stamp, per your note.

[miguel-heygen]

Re-review — new commit 47284a1e pushed after Via's COMMENTED review.

Via's review at 1462ee26 was thorough (alpha-override precedence, allow-list duplication, dockerRunArgs symmetry, cache-key correctness, CI summary). I read it in full. My job is additive: verify the fix commit addressed the allow-list nit and surface anything Via didn't flag.

✅ Fix commit (47284a1e) — allow-list duplication addressed cleanly

Via's nit: the ["auto","jpg","png"] set was independently declared three times — render.ts, server.ts, and renderConfigValidation.ts — three places to forget when a fourth format lands.

Fix: VIDEO_FRAME_FORMATS const + isVideoFrameFormat type guard hoisted into @hyperframes/engine (where VideoFrameFormat lives), re-exported from the barrel, and imported at all three call sites. Checked each site:

• packages/cli/src/commands/render.ts — local VIDEO_FRAME_FORMATS, VideoFrameFormat type, and isVideoFrameFormat function removed; now imports isVideoFrameFormat, type VideoFrameFormat from @hyperframes/engine. ✅
• packages/producer/src/server.ts:131-133 — inline ["auto","jpg","png"].includes(...) + as RenderConfig["videoFrameFormat"] cast replaced by isVideoFrameFormat(body.videoFrameFormat) ? body.videoFrameFormat : undefined (narrowing over assertion). ✅
• packages/producer/src/services/distributed/renderConfigValidation.ts:55 — ALLOWED_VIDEO_FRAME_FORMATS constant removed; isVideoFrameFormat + VIDEO_FRAME_FORMATS imported from engine; error-message string unchanged in shape. ✅

One subtle improvement in the fix: the original isVideoFrameFormat(value: string) signature in render.ts required the caller to pre-narrow to string before calling. The hoisted version uses isVideoFrameFormat(value: unknown) with an internal typeof === "string" check — strictly broader and correct. The server.ts call site now passes body.videoFrameFormat (typed unknown from the Record<string, unknown> body) directly, dropping the cast entirely. This is the right shape.

Nit (not covered by Via)

packages/producer/src/server.test.ts — zero test coverage for videoFrameFormat in parseRenderOptions. Via verified the implementation and CI; I checked the test file at 1462ee26 and confirmed there are no tests for: valid format passed → included in result; invalid string passed → dropped (undefined); key absent → undefined. The valid/invalid cases for format (mp4/webm/mov) and outputResolution are tested directly. Adding three short test cases for videoFrameFormat would close this gap and pin the allow-list contract at the parser boundary. Soft nit — the engine tests and AWS Lambda validator tests cover correctness end-to-end; this is unit coverage on the HTTP server parser.

CI

All fast checks green on 47284a1e. CLI smoke (required) and regression shards pending at time of review.

Verdict: APPROVE
Reasoning: Clean takeover rebase of #1142; resolveFrameFormat alpha-override is strictly safer and tested; cache-key collision prevented; fix commit addresses Via's allow-list duplication nit with exactly the right shape. One nit on parser test coverage is not load-bearing. Ship it.

— Vai

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
hyperframes	🟢 Ready	View Preview	Jun 16, 2026, 4:43 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.