Skip to content

feat(rustledger): in-process Component-Model engine behind a flag#161

Merged
robcohen merged 2 commits into
mainfrom
feat/component-engine
Jun 17, 2026
Merged

feat(rustledger): in-process Component-Model engine behind a flag#161
robcohen merged 2 commits into
mainfrom
feat/component-engine

Conversation

@robcohen

Copy link
Copy Markdown
Member

Adds an experimental, flag-gated rustledger engine that drives the typed
WASI Preview 2 / Component-Model component (rustledger-ffi-component,
rustledger #1384)
in-process via wasmtime-py — the successor to the current JSON-RPC engine,
which spawns the wasmtime CLI once per call and exchanges hand-mirrored JSON
DTOs over stdin/stdout.

What's here

  • RustledgerComponentEngine (src/rustfava/rustledger/component_engine.py)
    loads the component once, instantiates it with WASI p2, and calls the typed
    ledger / util / format exports directly (no JSON envelope, no
    subprocess). It mirrors RustledgerEngine's surface: version, load,
    query, validate, format_entries, get_account_type, is_encrypted,
    load_full.
  • A generic, type-driven marshaller (_marshal) converts the component's
    typed Record/Variant/list values into plain Python dicts/lists by
    walking the component's own type metadata (RecordType.fields,
    VariantType.cases). Variants render as discriminated {"type": <case>, ...}
    (mirroring the JSON-RPC tagged unions), so there are no hand-maintained
    per-type field lists
    .
  • get_engine() (rustfava.rustledger) selects the backend via
    RUSTFAVA_RUSTLEDGER_BACKEND=component; the default stays the JSON-RPC engine.
    The component module — and therefore wasmtime — is imported lazily, so
    the default path gains no new dependency. wasmtime is a new optional
    component extra.

Why now (and why behind a flag)

This is the consumer-migration proof for the rustledger Component-Model work.
The component is still publish = false upstream (rustledger ADR-0006 Phase 3),
so rustfava can't ship against a released artifact yet — but the in-process
typed-host path is now proven end-to-end, ready to flip on when it releases.
The in-process model also removes the per-call subprocess cost and the JSON DTO
drift.

Testing

tests/test_rustledger_component_engine.py covers version / load (typed
directive marshalling) / query / account-type / load_full (include resolution
over a WASI pre-open). They skip unless wasmtime is installed and the
wasip2 artifact is locatable (RUSTLEDGER_COMPONENT_WASM or bundled). Build it
with:

cargo build -p rustledger-ffi-component --target wasm32-wasip2 --release
  • Component tests: 5 passed (against a local build).
  • Default path unaffected: existing engine tests pass, and importing
    rustfava.rustledger pulls in no wasmtime.
  • ruff check + ruff format clean.

Follow-ups (not in scope here)

  • Bundle the component .wasm and flip the default once it's a release artifact.
  • load_full currently pre-opens only the entry file's directory tree, so
    cross-tree (../) includes are unreachable; a wider pre-open root would lift
    that (documented in the method).

🤖 Generated with Claude Code

robcohen added a commit that referenced this pull request Jun 16, 2026
Addresses the review of #161 — the generic marshaller diverged from the
JSON-RPC engine's shapes, so downstream loader/types/options mis-parsed
nearly every multi-word field:

- Map WIT kebab-case identifiers to snake_case keys / variant tags (the whole
  downstream reads snake_case; only single-word fields survived before).
- Render WIT string-keyed maps (`list<tuple<string, V>>` — display-precision,
  meta.user, …) as dicts so `options_from_json`'s `.items()` doesn't crash.
- Flatten `meta.user` into the meta object with scalar values (mirroring the
  JSON-RPC `Meta`'s `#[serde(flatten)]` + meta-value flattening).
- Fix the `result<ok, err>` branch: it lifts to a Variant and does NOT raise —
  surface `err` as an exception, unwrap `ok` (was returning a raw Variant).
- Serialize the shared, non-thread-safe wasmtime `Store` behind a lock (Fava
  is multi-threaded; `load_full` uses a fresh per-call instance).

Also:
- Regenerate `uv.lock` to include `wasmtime` (the `component` extra) — fixes
  the `check-lockfile` CI failure.
- mypy: treat optional `wasmtime` as untyped and allow Any returns from the
  FFI-boundary engine (two scoped overrides).
- Strengthen the tests to round-trip marshalled output through the real
  `options_from_json` / `directives_from_json`, which is what would have caught
  these bugs. Verified end-to-end against a locally-built component wasm.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@robcohen robcohen force-pushed the feat/component-engine branch from 66c1e52 to b2c35c4 Compare June 16, 2026 21:48
robcohen and others added 2 commits June 17, 2026 09:18
Adds `RustledgerComponentEngine`, an experimental successor to the JSON-RPC
`RustledgerEngine` that drives rustledger's typed WASI Preview 2 / Component
Model component (`rustledger-ffi-component`, rustledger beancount#1384) directly via
`wasmtime-py` — no `wasmtime` CLI subprocess per call, no hand-mirrored JSON
DTOs.

- `component_engine.py`: loads the component once, instantiates with WASI p2,
  and calls the typed `ledger`/`util`/`format` exports. A generic, type-driven
  marshaller (`_marshal`) walks the component's own `RecordType.fields` /
  `VariantType.cases` to turn typed `Record`/`Variant`/`list` values into plain
  dicts/lists (variants as discriminated `{"type": ...}`), so there are no
  hand-maintained per-type field lists. `load_full` pre-opens the entry file's
  directory into the WASI sandbox for include resolution.
- `get_engine()` dispatcher selects the backend via
  `RUSTFAVA_RUSTLEDGER_BACKEND=component`; default stays JSON-RPC. The component
  module (and `wasmtime`) is imported lazily, so the default path gains no
  dependency. `wasmtime` is a new optional `component` extra.
- Smoke tests (`test_rustledger_component_engine.py`) cover version/load/query/
  account-type/load-file; skipped unless `wasmtime` + the wasip2 artifact are
  present.

Gated behind the flag because the component is `publish = false` upstream
(rustledger ADR-0006 Phase 3); this proves the in-process typed-host path works
end-to-end ahead of that release.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the review of #161 — the generic marshaller diverged from the
JSON-RPC engine's shapes, so downstream loader/types/options mis-parsed
nearly every multi-word field:

- Map WIT kebab-case identifiers to snake_case keys / variant tags (the whole
  downstream reads snake_case; only single-word fields survived before).
- Render WIT string-keyed maps (`list<tuple<string, V>>` — display-precision,
  meta.user, …) as dicts so `options_from_json`'s `.items()` doesn't crash.
- Flatten `meta.user` into the meta object with scalar values (mirroring the
  JSON-RPC `Meta`'s `#[serde(flatten)]` + meta-value flattening).
- Fix the `result<ok, err>` branch: it lifts to a Variant and does NOT raise —
  surface `err` as an exception, unwrap `ok` (was returning a raw Variant).
- Serialize the shared, non-thread-safe wasmtime `Store` behind a lock (Fava
  is multi-threaded; `load_full` uses a fresh per-call instance).

Also:
- Regenerate `uv.lock` to include `wasmtime` (the `component` extra) — fixes
  the `check-lockfile` CI failure.
- mypy: treat optional `wasmtime` as untyped and allow Any returns from the
  FFI-boundary engine (two scoped overrides).
- Strengthen the tests to round-trip marshalled output through the real
  `options_from_json` / `directives_from_json`, which is what would have caught
  these bugs. Verified end-to-end against a locally-built component wasm.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@robcohen robcohen force-pushed the feat/component-engine branch from b2c35c4 to 42cb534 Compare June 17, 2026 09:18
@robcohen robcohen merged commit 084001b into main Jun 17, 2026
29 checks passed
@robcohen robcohen deleted the feat/component-engine branch June 17, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant