Skip to content

Refresh meos-idl.json to MEOS-API canonical; regenerate bindings#2

Open
estebanzimanyi wants to merge 1 commit into
MobilityDB:mainfrom
estebanzimanyi:feat/meos-idl-refresh-extended-types
Open

Refresh meos-idl.json to MEOS-API canonical; regenerate bindings#2
estebanzimanyi wants to merge 1 commit into
MobilityDB:mainfrom
estebanzimanyi:feat/meos-idl-refresh-extended-types

Conversation

@estebanzimanyi
Copy link
Copy Markdown
Member

@estebanzimanyi estebanzimanyi commented May 22, 2026

Refreshes codegen/res/meos-idl.json to the MEOS-API canonical IDL and regenerates both outputs, now tracking the public MobilityDB surface.

Source of truth (reproducible)

  • MEOS-API: master, including #15 (PG-vendored type recovery)
  • MobilityDB: master at 2c4243a

The committed IDL is exactly what python setup.py && python run.py produces against those two states:

# fresh MEOS-API master checkout (post-#15)
python setup.py            # clones MobilityDB master
python run.py              # -> output/meos-idl.json : 2672 functions, 47 structs, 6 enums
diff <(jq -S . output/meos-idl.json) <(jq -S . codegen/res/meos-idl.json)   # empty

What changed vs the previous IDL

  1. ~512 signature drifts fixed — regenerating with #15's typerecover.py recovers GSERIALIZED / Datum / DateADT / GBOX / BOX3D / AFFINE slots that the earlier parse had collapsed to int / int *.
  2. 2 functions recoveredmindistance_tgeo_tgeo and tgeoarr_tgeoarr_mindist are present.
  3. Tracks public master (2672 functions). The not-yet-merged extended-type families (tbigint / th3index / extended tcbuffer / tpose) are dropped; they re-enter via a follow-up refresh once they land upstream, so the canonical binding never points at non-public code.

Datum stays out of the user API

The recovered types exposed that 171 functions carry a Datum in their signature — all internal (meos_internal.h). A Datum is an opaque tagged uintptr_t (a pointer for Float8 / text / geometry base types, 32-bit under WASM32) and cannot be marshalled to a JS scalar. The generator now skips every Datum-signature function (isDatum added to shouldSkip); the user-facing path is the typed *_meos.c wrappers, e.g. temporal_start_valuetint_start_value / tfloat_start_value / ttext_start_value. No Datum appears in any functions.generated.ts signature.

Wrapper + build sync

  • core/types/basic/{TFloat,TInt,TNumber}.ts: track MobilityDB's mult_*mul_* multiplication rename (the only core/types change — the symbols were renamed upstream).
  • Dockerfile: bump MOBILITYDB_COMMIT to 2c4243a so the WASM build matches the regenerated bindings.c.

Verify

npm run generate    # Generated: 2465  Skipped: 172  Manual: 35
npx tsc --noEmit    # clean

@Nyuke235
Copy link
Copy Markdown
Collaborator

Thank you for the refresh. I ran a final reproducibility check after MobilityDB/MEOS-API#15 landed, and the committed IDL doesn't match what regenerates from the current canonical state. Could you regenerate against master once more before we merge?

Pulled MEOS-API master (now with #15 merged), ran python run.py against a fresh setup.py clone of MobilityDB master, and diffed output/meos-idl.json against codegen/res/meos-idl.json from this branch:

Esteban's IDL: 2830 functions, 47 structs, 6 enums
Fresh IDL:     2672 functions, 47 structs, 6 enums

In Esteban but not Fresh: 160
In Fresh but not Esteban: 2
Common: 2670
Signature drift in common funcs: 512

tsc --noEmit still passes on this branch (I re-confirmed), because the drifts are in functions no core/types/**/*.ts wrapper currently consumes, but the IDL is the binding-generation source of truth, so the next wrapper that touches one of these will get an incorrect signature.

3 causes:

1. Older typerecover.py (the largest of the three)

Timestamps:

  • 23722ba (this PR's refresh commit): 2026-05-22T21:30:29Z
  • 9ef867a (the typerecover.py commit that landed in MEOS-API#15): 2026-05-24T19:18:42Z

The refresh was generated ~46h before _TYPE_MAP reached its final form. The merged version recovers GSERIALIZED, Datum, DateADT, GBOX, BOX3D, AFFINE; the IDL in this PR was generated before those entries existed. Concrete examples:

Function (slot) This PR's IDL Fresh from master
edwithin_tcbuffer_geo (gs param) const int * const GSERIALIZED *
geom_perimeter (gs param) const int * const GSERIALIZED *
temporal_start_value (return) int Datum
tsequence_max_val (return) int Datum
overbefore_set_date (d param) int DateADT

512 functions are affected: every GSERIALIZED * / Datum / DateADT / GBOX * / BOX3D * slot in scope is still spelled int/int * here.

2. MobilityDB headers were not master

The IDL contains 160 functions that don't exist in MobilityDB master:

  • 60 tbigint_*
  • 63 th3index_*
  • 121 tcbuffer_* (vs ~70 on master)
  • 54 tpose_*

These look like a feature branch with the extended type families. Could you confirm which MobilityDB commit/branch was used? If it's a not-yet-merged feature branch, the binding generator's source of truth would point at code that doesn't exist upstream, I'd prefer the canonical MEOS.js to track a state of MobilityDB that's already public.

3. Two functions present on master but missing here

mindistance_tgeo_tgeo and tgeoarr_tgeoarr_mindist — confirms the headers snapshot used was slightly behind current master.

What I'm asking for

  1. Rebase on current main (now with Initial commit #1 merged) and regenerate codegen/res/meos-idl.json via python run.py against:
    • MEOS-API at current master (after the #15 merge)
    • MobilityDB headers at a stable state: either master, or a specific feature branch you can document in the PR body
  2. Re-run npm run generate against the new IDL, commit the updated bindings.c + functions.generated.ts
  3. Force-push the amended 23722ba

The reproducibility check I ran is:

# in a fresh MEOS-API checkout, master at or after the #15 merge
python setup.py                         # or --branch <feature> if you used a branch
python run.py
diff <(jq -S . output/meos-idl.json) \
     <(jq -S . path/to/MEOS.js/codegen/res/meos-idl.json)

Once that diff is empty, I'll re-verify and approve.

Regenerate codegen/res/meos-idl.json from MEOS-API master (including #15's
typerecover fix) against MobilityDB master (2c4243a). This recovers the
PG-vendored C types the libclang parse had collapsed to int — GSERIALIZED,
Datum, DateADT, GBOX, BOX3D, AFFINE — fixing ~512 signature drifts, and picks
up mindistance_tgeo_tgeo / tgeoarr_tgeoarr_mindist. The IDL now tracks the
public MobilityDB surface (2672 functions); the not-yet-merged extended-type
families are dropped until they land on master.

Skip internal Datum-typed functions in the generator. A Datum is an opaque
tagged uintptr_t (a pointer for Float8/text/geometry base types, only 32-bit
under WASM32) that cannot be marshalled to a JS scalar. Such functions are
internal (meos_internal.h); the user-facing API is the typed *_meos.c wrappers
(e.g. temporal_start_value -> tint/tfloat/ttext_start_value). 171 Datum-signature
functions are now excluded from the generated surface so Datum never reaches the
user API.

Track MobilityDB's mult -> mul multiplication rename in the TFloat/TInt/TNumber
wrappers, and bump the Dockerfile MOBILITYDB_COMMIT pin to 2c4243a so the WASM
build stays consistent with the regenerated bindings.

  npm run generate -> Generated: 2465  Skipped: 172  Manual: 35
  npx tsc --noEmit  -> clean
  reproducibility: diff of a fresh run.py output vs the committed IDL is empty
@estebanzimanyi estebanzimanyi force-pushed the feat/meos-idl-refresh-extended-types branch from 23722ba to bbf7831 Compare May 26, 2026 21:00
@estebanzimanyi
Copy link
Copy Markdown
Member Author

Done — rebased onto main (now with #1) and force-pushed bbf7831. Regenerated against MEOS-API master (post-#15) and MobilityDB master 2c4243a, and your reproducibility diff is now empty:

$ diff <(jq -S . output/meos-idl.json) <(jq -S . codegen/res/meos-idl.json)
$       # (empty)
Done: 2672 functions, 47 structs, 6 enums

Point by point:

  1. typerecover (512 drifts) — fixed at the root by regenerating with #15. The GSERIALIZED / Datum / DateADT / GBOX / BOX3D / AFFINE slots are recovered; your cited examples now read const GSERIALIZED *, Datum, DateADT.

  2. Source state — I took your preferred option and pointed the regen at public master, so the 160 extended-type functions (tbigint / th3index / extended tcbuffer / tpose) are gone. They sit on a not-yet-merged feature branch; I'd rather refresh them in once they land upstream than have the canonical binding track non-public code.

  3. Missing functionsmindistance_tgeo_tgeo and tgeoarr_tgeoarr_mindist are present.

One thing your Datum examples surfaced: temporal_start_value (and 170 others) now correctly type as Datum — but Datum is internal (meos_internal.h) and can't be marshalled to a JS scalar (it's a pointer for Float8/text/geometry, and only 32-bit under WASM32). So the generator now skips every Datum-signature function, and the user-facing path stays the typed wrappers (tint_start_value / tfloat_start_value / …). functions.generated.ts carries no Datum in any signature.

I also bumped the Dockerfile MOBILITYDB_COMMIT pin to 2c4243a and tracked the upstream mult_*mul_* rename in the TFloat/TInt/TNumber wrappers (the only core/types change).

npm run generate → 2465 generated / 172 skipped / 35 manual; npx tsc --noEmit clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants