feat: support Lua and JavaScript extensions#1196
Conversation
✨ Highlights
🧾 Changes by Scope
🔝 Top Files
|
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
from
ef7ea6b to
2156b1c
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1196 +/- ##
========================================
Coverage 82.12% 82.12%
========================================
Files 33 33
Lines 3149 3149
Branches 734 734
========================================
Hits 2586 2586
Misses 387 387
Partials 176 176
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
An automated preview of the documentation is available at https://1196.mrdocs.prtest2.cppalliance.org/index.html If more commits are pushed to the pull request, the docs will rebuild at the same URL. 2026-05-22 18:08:58 UTC |
gennaroprota
changed the title
gennaroprota
changed the title
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
7 times, most recently
from
e58fb63 to
075f7bc
Compare
There was a problem hiding this comment.
The PR is great. It's really nice to see we already have examples that transform the corpus via the scripts.
I don't see much value in putting Lua scripts as Handlebars helpers (a cheap improvement win in an extension type that already exists) with Corpus-mutation extensions in Lua and JavaScript (a completely new feature that's huge).
a new
Describe.hpp
Doesn't it already exist? And wasn't the distinction between Describe.hpp and other headers related to reflection that Describe.hpp would only be about porting Boost.Describe?
MRDOCS_DESCRIBE_HIERARCHY
I believe this needs better in-source documentation. We even support that now (or are about to).
I also don't understand these *Hierarchy.hpp files that much. The PR describes that they exist, but not why they exist. All I know is they're using this pattern of including everything, which defeats the purpose of public headers. I also don't see why the user ever needs that information to be public.
They also seem to defeat the purpose of things in a sense. Because we're trying to avoid manually listing things with reflection, but we are still doing it there. The typical pattern we use for that is a single source of truth as inc files.
I also don't understand how a hierarchy can be described as a list unless there's some special convention about this list.
Tests
There's enough new functionality (even public functionality) here that seems to deserve unit tests.
Golden tests
Do they really cover everything that's possible? What are the typical use cases for this? Do we get errors on invalid transformations? How do we merge symbols in the corpus? How do you add symbols to the corpus?
Third-party:
third-party/patches/lua/CMakeLists.txtupdated for the vendored Lua build.
What do you mean by vendored Lua build here?
- No CI workflow changes needed — the existing golden-test job runs all of
test-files/golden-tests/on every build, so the new trees stay covered automatically going forward.
Documentation
I think the documentation should make the same distinction you made here. It could maybe be described as corpus-mutation extensions or something. Because we have this concept of extensions. Then, things people understand as extensions are these customizations of the template system and the corpus-mutation extensions. They are very different layers and share a common logic of knowing where to put addons, etc. They also have to know about supplemental addons, which they're probably going to use the most. The documentation concerns the user journey.
Also, any reference should be generated automatically. I don’t understand the API very well. Is mrdocs.set the only function we have?
I also don’t see the pattern we had discussed about taking inspiration from Lua/Darktable. It seems the new extension system uses the same pattern as the helpers extension system, but that pattern is not appropriate there. What if an extension wants to provide different functionalities? At a minimum, it's important to include a comparison and an explanation of why the PR chose a different strategy.
Is that the best Python and Lua API we could have on the Lua/JS side? What are the alternatives? They look very different from what things look like on the C++ side. Are they extendable, or will we need to break the Lua/JS API every time there’s a new feature?
The "Corpus argument" section describes a subset of a symbol's API (I don't really understand why, since the full reference is elsewhere), but the API of the corpus object itself isn't described.
|
Thanks for the thorough review. Great stuff in there :-). Here's a point by point reply:
Open decisions for you:
|
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
2 times, most recently
from
22b6462 to
f68a074
Compare
Up to you. I'll let you come up with the best design you can and we can iterate on it. |
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
from
f68a074 to
19fa6fa
Compare
I moved the new macro out of Describe.hpp (which now only contains the Boost.Describe port) and renamed it to |
|
Thanks for the iteration. Mostly looking good. In the branch: the Still open from our back-and-forth:
A few things from the original review that didn't get picked up in the back-and-forth:
One thing I'd like us to revisit before we close: "intentionally not supported, structural changes break invariants" on merging and adding symbols is technically true but too compact to act on. I understand there's a trade-off here between not breaking invariants and being able to perform anything useful. But I don't know what exactly is the trade-offs and what operations can help us escape the trade-off by ensuring invariants aren't broken. Because a transform extension that only receives constants as arguments is not useful for anything. To pick the right boundary we'd need to formalize in docs:
It's OK to leave things for the future. It's not OK to not be clear about what we have right now and where we're going with this. And without that written down in the docs, any group discussion in the channel will derail once more people join, since nobody's reasoning from the same data. Maybe some of that could become its own section in #1210, or a sibling design issue. Thanks for what we got here so far 😁 I know this is a much more complex feature than anything else we've been doing so far. |
|
Here's another review by claude:https://gist.github.com/alandefreitas/837bfc0f10e70fce33feffff5527b35a |
This mirrors the existing JS helpers for Lua. *.lua files placed in an
addon's generator/{common|<ext>}/helpers/ directory are auto-registered
as Handlebars helpers; files whose name starts with '_' run first as
utility scripts. Two golden fixtures (lua-helper/, lua-helper-layering/)
mirror their JS counterparts and cover the `addons-supplemental`
override.
Incidental fixes to issues uncovered by this patch:
- Added a qualification to `MRDOCS_TRY` / `MRDOCS_CHECK_*` /
`MRDOCS_CHECK_OR_*` / `MRDOCS_CHECK_OR_CONTINUE` to make them work
with nested namespaces named `detail`.
- Dropped onelua.c and ltests.c from the Lua build patch, because the
former defines `main`, which conflicted with our `main`, and the
latter is test scaffolding which shouldn't ship in a library build.
- Added `extern "C"` around the Lua includes.
The `__index` metamethod in `domObject_push_metatable()` retrieved the value correctly via `Object::get(key)`, then called `lua_replace(L, 1)` to move the result into the userdata's slot. `lua_replace` also pops the top, so, on return, the key string was at the top of the stack and Lua picked it up as the metamethod's single return value, making every field access on a `dom::Object` userdata silently return the key it was asked for. This was latent until now because no Lua script in the test suite previously read fields off a `dom::Object` userdata. Surfaced while wiring corpus extensions: a script doing `corpus.symbols[i]` saw `"symbols"` (the key) instead of the array.
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
from
19fa6fa to
db3824f
Compare
This adds a hook that runs user-provided Lua scripts after corpus extraction and finalization, before any generator runs. Extensions live in <addon>/extensions/*.lua for each addon root in the configuration. A script may define `transform_corpus(corpus)`, which is invoked once with a flat DOM view of the corpus. The script may mutate the corpus by calling pre-registered globals on the `mrdocs` table; currently: - `mrdocs.set_brief(symbol_id, text)`: replace a symbol's brief with a single-paragraph plain-text block. Each setter validates its arguments and raises a Lua error on misuse; any uncaught error in a script aborts the build. Multiple extensions run in alphabetical order by file path. The mutation surface is intentionally narrow; additional setters will land as concrete use cases surface. A golden fixture (test-files/golden-tests/extensions/lua-set-brief/) rewrites a function's brief from Lua and verifies the change reaches the xml output. Finally, this touches Lua wrapper for two new affordances: `Scope::pushDom` for the corpus argument, and a `Context::nativeState()` escape hatch for binding native C functions as Lua globals (the wrapper doesn't abstract that yet).
This mirrors the Lua corpus-mutation hook for JavaScript. Extension scripts under <addon>/extensions/*.js can now define `transform_corpus(corpus)` and call `mrdocs.set_brief(symbol_id, text)`, just like their Lua counterparts. The Lua and JS bindings now share a language-agnostic `setBriefImpl` helper that takes already-extracted `dom::Value` arguments. Each binding is a thin adapter: - Lua: the existing C closure registered on the raw `lua_State*`. - JS: a `dom::Function` exposed as a property of a `mrdocs` global object; the wrapper's `setGlobal` -> `toJsValue` -> `makeFunctionProxy` chain handles the rest. Discovery picks up both *.lua and *.js, sorted together so script ordering doesn't depend on the chosen language. A golden fixture mirrors the Lua test.
This completes the work on extensions by making both the read and the write view of scripts lay on the describe machinery already used by the rest of the code. Scripts can now read any described field, and write any field on an (intentionally small) allowlist.
This adds a `MRDOCS_DESCRIBE_KINDS` macro that registers the closed
set of derived classes ("kinds") of a polymorphic base. This lets
generic code dispatch over the closed set without the per-base
X-macro boilerplate every consumer would otherwise need.
The macro and its supporting machinery live in a dedicated
DescribeKinds.hpp header so consumers that only need
`MRDOCS_DESCRIBE_STRUCT`, `MRDOCS_DESCRIBE_CLASS`, and
`MRDOCS_DESCRIBE_ENUM` keep a slimmer include.
This applies `MRDOCS_DESCRIBE_KINDS` (added in the previous commit) to every polymorphic base in MrDocs. Each registration lives in a small private include under src/lib/Metadata/. The kind information is a compile-time consumer-side concern, so no public header exposes it; consumers include the relevant file directly. This commit only registers the relationships. The next one will introduce the first consumer.
The previous step's generic setter could navigate through `Optional`, `vector` and described structs, but not `Polymorphic<T>`, the value wrapper for polymorphic types used throughout MrDocs. It can now. The generic setter dispatches a `kind:` field against the closed set of derived classes registered via `MRDOCS_DESCRIBE_KINDS`, forwards the remaining DOM keys to the matched class, and writes the result into the `Polymorphic<T>`. The Lua adapter gains a recursive table-to-DOM converter; `doc` and `loc` are added to the allowlist. A new unary `is_polymorphic_v<T>` trait in `TypeTraits.hpp` answers "is `T` some `Polymorphic<...>`?" without naming the inner type, as needed by the dispatch above. The existing binary trait in `MergeReflectedType.hpp` is renamed to `is_polymorphic_for_v<T, U>` so the two coexist.
Some library types are better presented in documentation by their
semantic role than by their literal C++ form: e.g., a function returning
`capy::task<T>` is more useful to readers when documented as a coroutine
yielding `T` than when shown with its literal task-object return type.
Hence, this adds `returnType` to the script-side allowlist.
This is the substrate for semantic-rendering extensions, not the fully
declarative API one might eventually expose ("declare this type an
awaitable, MrDocs handles the rest"). That higher-level API can be built
on top of this hook.
…ce of truth Before: `kSettableFields` lived inline in `SetMember.cpp`, and the AsciiDoc reference table in `extensions.adoc` listed the same fields again, by hand. Adding a field meant touching both sites; mismatches were not caught by the build. Now: `src/lib/Extensions/AllowedFields.json` is the single source of truth. A CMake `add_custom_command` runs `util/generate_extension_allowed_fields.py` to emit both `include/mrdocs/Extensions/AllowedFields.gen.hpp` (the constexpr `kSettableFields[]` array `SetMember.cpp` now consumes) and `docs/modules/ROOT/partials/extensions-allowed-fields.adoc` (the allowlist table included from `extensions.adoc`). Adding a field is a single edit to the JSON. No semantic change to the script-facing API; the allowlist's contents are unchanged from the previous commit.
Replace the duplicated subset of symbol-fields with a clear "what `corpus` actually is" lead. Note explicitly that lookups, name searches and other queries are not exposed on `corpus`; scripts walk `corpus.symbols` and filter. For per-symbol fields, link out to the templates documentation rather than duplicate a subset.
Template helpers (generators.adoc) and corpus-mutation extensions (extensions.adoc) live at different layers but share the same addon-discovery machinery. Both pages were duplicating the explanation of `addons` and `addons-supplemental`, and neither covered the user journey of "what is an addon, where does it sit?". This adds `addons.adoc` as the canonical page for the addon concept: the built-in addon, replacement via `addons`, supplementation via `addons-supplemental`, the on-disk layout, and the discovery rule (last-wins for templates/helpers, full aggregation for extensions). `extensions.adoc` and `generators.adoc` drop their local explanations and cross-link to it.
This adds a "Design rationale" section to extensions.adoc.
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
from
db3824f to
6875227
Compare
3 participants
This mirrors the existing JavaScript helpers for Lua, and adds support for corpus-mutation extensions written either in Lua or Javascript.
Adds two related script-extension capabilities and the reflection plumbing they rely on:
<addons>/extensions/*.{lua,js}run between corpus finalisation and the first generator invocation. Each script may expose a globaltransform_corpus(corpus)function; the host calls it once with a read view of the corpus, and the script applies mutations through amrdocs.set(id, field, value)API. The script side is intentionally a narrow, allowlist-gated write surface (name,extraction,isCopyFromInherited,doc,loc,returnType). Reads piggy-back on the same DOM the generators already see, so scripts and templates have one shape to learn.The follow-up design for richer entry points (multiple capabilities per file, paired helpers, generator registration, enable/disable, auto-generated
mrdocs.*reference) lives in issue #1210; this PR ships the simplest rung of that ladder.To make the script-facing types stay in sync with the C++ model without a hand-maintained binding table, the extension layer is driven by reflection:
MRDOCS_DESCRIBE_KINDSmacro registers the closed set of concrete derived classes of a polymorphic base. The macro lives ininclude/mrdocs/Support/DescribeKinds.hpp. Each per-base list lives privately undersrc/lib/Metadata/and is fed by the existing*Nodes.incX-macro files, so the registered set has one source of truth.mrdocs.setbuilds aPolymorphic<Base>from a DOM object whosekind:field picks a concrete derived class registered withMRDOCS_DESCRIBE_KINDS.A
dom::Objectfield-lookup bug surfaced during the work (lookup returned the key instead of the value) and is fixed in the same PR.Changes
src/lib/Extensions/AddonDiscovery.{hpp,cpp}-- collect<root>/extensions/*.{lua,js}across addon roots (sorted by full path).src/lib/Extensions/SetMember.{hpp,cpp}-- the entire write surface: allowlist,assignFromDomoverload set,trySetMember,buildPolymorphic,ExtensionState,buildCorpusDom,setMemberImpl. This is the file that a maintainer touching the allowlist or the type machinery edits.src/lib/Extensions/LuaBinding.{hpp,cpp}-- Lua-side glue (state plumbing,mrdocs.setregistration,luaValueToDom).src/lib/Extensions/JsBinding.{hpp,cpp}-- JavaScript-side glue (dom::Function-backedmrdocs.set).src/lib/Extensions/RunExtensions.{hpp,cpp}-- orchestrator: collect scripts then dispatch to the right per-script entry point.CorpusImpl.cppinvokesrunExtensions()after finalisation.addonRoots()helper used by both the Handlebars layer and the extension layer was promoted tosrc/lib/Support/AddonRoots.hpp(a previous copy insrc/lib/Gen/hbs/AddonPaths.hppis gone).src/lib/Support/Lua.cpp'sdom::Arraymetatable is now a Lua-convention 1-indexed sequence (arr[1]is first,#arris the length,ipairs/pairswork).src/lib/Gen/hbs/Builder.cppupdated to load Lua helpers alongside JS ones; newinclude/mrdocs/Support/DescribeKinds.hppdefines theMRDOCS_DESCRIBE_KINDSmacro and its trait/query types; per-base kind lists undersrc/lib/Metadata/; small additions toinclude/mrdocs/Support/MergeReflectedType.hpp,Expected.hpp, andTypeTraits.hpp;dom::Objectfield-lookup fix.src/test/Extensions/SetMember.cppcovers themrdocs.seterror paths: argument-shape errors, unknown id, off-allowlist field, allowlisted-but-missing-on-this-symbol-kind, type mismatch for string/bool/enum fields, unknown enum name, polymorphic-write errors (not-an-object, missingkind, unknown kind, unknown sub-field), unknown sub-field on a struct field, and the happy path forOptional<T>null-reset.src/test/Support/DescribeKinds.cppcovers thehas_describe_kindstrait and thefor_each(describe_kinds<Base>{}, ...)iteration in both the variadic and the BEGIN/END forms of the registration macro.extensions/{js-set-name,lua-set-name,lua-set-return-type}/exercise rename, doc rewrite, and aPolymorphic<Type>write. Three more trees lock in claims the docs make:extensions/lua-extension-ordering/(alphabetical-by-full-path ordering across two addon roots),extensions/lua-clear-doc/(mrdocs.set(id, "doc", nil)clears the optional), andextensions/lua-empty-script/(an extension file with notransform_corpusis silently skipped). Newgenerator/hbs/{lua-helper,lua-helper-layering}/exercise Lua Handlebars helpers, including helper layering across multiple addon directories.docs/modules/ROOT/pages/addons.adocdocuments addon discovery (lookup paths,addons,addons-supplemental, override vs. layering semantics).docs/modules/ROOT/pages/extensions.adocdocuments the corpus-mutation extension model end to end, including a worked example, the writable-fields table, an "Invariants and operations" section spelling out which corpus properties the rest of Mr.Docs depends on and which structural operations are/aren't reachable from scripts, a Stability section (the extensibility contract scripts can rely on), and a Design rationale section comparing the four alternatives considered for the write surface and explaining why (D) was picked.docs/modules/ROOT/pages/generators.adocgains a[#custom-helpers]section covering JS and Lua helpers and the layering model.docs/mrdocs.ymlexcludes the reflection helpers from the generated reference.third-party/patches/lua/CMakeLists.txtupdated. The Lua source lives in-tree underthird-party/lua/and is built as part of MrDocs; this file is MrDocs's CMake override for that in-tree build (upstream Lua does not ship aCMakeLists.txt). The diff is to MrDocs's override, not to upstream Lua. The change drops two files from the Lua build:onelua.c(a unity-build entry point that defines its ownmain, which would conflict with MrDocs'smainwhen statically linked) andltests.c(Lua's internal test harness, only valid whenLUA_USER_Henables it -- not meant to ship in a library build).Testing
test-files/golden-tests/extensions/andtest-files/golden-tests/generator/hbs/lua-helper*/execute real Lua and JS scripts against fixed fixtures and assert the.xml,.html, and.adocoutput byte for byte. Any future regression in the helper or extension surface fails these tests.clang.mrdocs.Extensions.SetMember,clang.mrdocs.Support.DescribeKinds) cover, respectively, themrdocs.seterror paths and theMRDOCS_DESCRIBE_KINDStrait and iteration.test-files/golden-tests/on every build, so the new trees stay covered automatically.Documentation
extensions.adocdocuments the corpus-mutation extension model: a worked example (synthesising briefs from a naming convention), file layout, thetransform_corpushook, thecorpusargument, themrdocs.setAPI and its hand-maintained allowlist (with a note pointing at the future auto-generated table on issue Design the extension entry-point ladder beyondtransform_corpus#1210), the lifecycle (where extensions run relative to extraction and rendering), an "Invariants and operations" section, a "Stability" section spelling out the extensibility contract, and a "Design rationale" section comparing four shapes (A-D) considered for the write surface.generators.adocgains the[#custom-helpers]section covering JS and Lua Handlebars helpers, including the underscore-prefix utility convention, the options-object stripping rule, and resolution order.addons.adocis a new shared page covering lookup paths and the override-vs-layering semantics shared by both helpers and extensions.