consClose only ever advanced currPtr by 1 byte. When the constructor
tag bits plus the already-consumed bits in the current byte totalled
16 or more (e.g. a 9-bit tag after 7 bits already used), usedBits
overflowed past 7 and the decoder state was silently corrupted.
The immediate symptom: dBool computes (128 >> usedBits) as 0 when
usedBits >= 8, so every bit reads as False. The Filler decoder then
loops forever building FillerBit constructors until memory runs out.
Replace the hand-rolled if/else with dropBits, which already uses
divMod to handle any number of bytes correctly.
Add three regression tests:
- control: 7 Bool fields + 8-bit tag (15 total bits, no overflow)
- bug trigger: 7 Bool fields + 9-bit tag (16 bits, infinite loop
without fix, guarded by a 5 s timeout)
- bounds check: 9-bit tag in a 1-byte buffer must fail with
NotEnoughSpace, not TooMuchSpace
We ran into this in plutus (IntersectMBO/plutus#7683), which vendors a copy of flat. Enums with more than 256 constructors were causing the decoder to eat all memory and hang.
consCloseonly advancescurrPtrby 1 byte. That's fine until the constructor tag bits plus the bits already used in the current byte add up to 16 or more. A 9-bit tag after 7 bits already consumed givesusedBits = 16, the subtraction leaves it at 8, and the decoder state is toast from there.dBoolcomputes128 >> 8as 0, reads every bit asFalse, and theFillerdecoder spins buildingFillerBits until you run out of memory.Fix is one line:
consClose = dropBits.dropBitsalready does thedivModto handle arbitrary bit counts, soconsClosedoesn't need its own logic.Added three tests to
testLargeEnum: a control case that stays under the boundary (7 Bools + 8-bit tag = 15 bits), the actual trigger (7 Bools + 9-bit tag = 16 bits, with a timeout), and a check that a 9-bit tag in a 1-byte buffer givesNotEnoughSpacenotTooMuchSpace.