Losing focus of the original UBJSON goal

[nebkat]

UBJSON was originally proposed to solve a problem with existing binary JSON formats:

Attempts to make using JSON faster through binary specifications like BSON, BJSON or Smile exist, but have been rejected from mass-adoption for two reasons:

1. Custom (Binary-Only) Data Types: Inclusion of custom data types that have no ancillary in the original JSON spec, leaving room for incompatibilities to exist as different implementations of the spec handle the binary-only data types differently.
2. Complexity: Some specifications provide higher performance or smaller representations at the cost of a much more complex specification, making implementations more difficult which can slow or block adoption. One of the key reasons JSON became as popular as it did was because of its ease of use.

BSON, for example, defines types for binary data, regular expressions, JavaScript code blocks and other constructs that have no equivalent data type in JSON. BJSON defines a binary data type as well, again leaving the door wide open to interpretation that can potentially lead to incompatibilities between two implementations of the spec and Smile, while the closest, defines more complex data constructs and generation/parsing rules in the name of absolute space efficiency. These are not short-comings, just trade-offs the different specs made in order to service specific use-cases.

The existing binary JSON specifications all define incompatibilities or complexities that undo the singular tenant that made JSON so successful: simplicity.

JSON’s simplicity made it accessible to anyone, made implementations in every language available and made explaining it to anyone consuming your data immediate.

Any successful binary JSON specification must carry these properties forward for it to be genuinely helpful to the community at large.

As an evolution of UBJSON, BJData originally fixed some basic quality-of-life issues with UBJSON, namely the lack of unsigned numbers and use of big-endian.

---

I believe the recent addition of extension types unfortunately bring us back to where the other binary specifications were in terms of complexity and risks reducing the further adoption of BJData as a generic JSON binary format.

It could be argued that other types such as char, half, and even the strongly-typed byte arrays I proposed had already deviated from the original JSON spec, but crucially these types all have an obvious/implicit representation in regular JSON, namely string, float and int array. This is not the case for extension types, where the context is completely lost if we turn them into plain numbers or strings.

I have wondered in hindsight whether it would have been better to stick with U based binary type as while it ended up solving a problem in some languages, it introduced further ambiguity in others e.g. Javascript which has a Uint8Array but no corresponding ByteArray. Extension types are likely to suffer the same fate, where some languages will have first-level support for timestamps, dates and UUIDs, while others will have to use proprietary types that make it more complicated to integrate fully.

---

Right now BJData is the closest thing we have to a one-to-one binary JSON representation, and I really don't want to lose that. Other than simply removing extension types from the spec, one option could be to branch BJData into two formats (which is ever so slightly better than introducing a new format...).

One would be a basic format that would stay true to the original goal - perhaps removing features such as no-op, N-dim arrays and any features deemed to be adding unnecessary complexity.

The other would be a feature-rich superset that could still parse the basic format but also introduce types that are not directly compatible with JSON.

This way those willing to commit to using "BJData++" exclusively could benefit from advanced features (though even there I would urge caution!), while BJData would remain a solid choice if you only care about JSON.

[fangq]

BJData spec has never set the goal of following UBJSON. It does prefer heavily towards JSON-like simplicity, which UBJSON follows.

@nebkat, the recent Draft-4 release of BJData adds two things:

1. packed objects {$... (or SoA struct-of-arrays or arrays-of-structs), and
2. extension type E

the first one has been something I have been hoping to add for a very long time - it mirrors the idea of packed array that I initially introduced to BJData that eventually departs BJData from the original UBJSON. it is also something being extensively proposed in the UBJSON community as well as after BJData starts its own path.

for extension E container, I consider it as the only way to stop BJData from inventing new type markers, and the only path towards creating a format that is both forward and backward compatible. As long as someone adds a new data type marker in the future, all existing parsers will break because it does not know how to handle it, or even how to ignore/skip it. E+typeid+payloadlen+payload provides such a way that parsers today can safely ignore unfamiliar data types, even data from future, yet still be able to read the remaining stream/data.

You don't really need to parse anything enclosed inside the E-marker. You can simply use the payloadlen to skip forward, or leave the payload buffer for other libraries to interpret. It is one of the simplest container that I can think of to provide such forward/backward compatibility. I don't know what else can be simpler, in the spirit of JSON, that can achieve this.

Again, extension types are custom types, if parsers do not want to handle customized data types, just ignore those. Perhaps I can add language to discourage the use of E for canonical data storage. was that what your concern? that people will abuse E and store function-critical data?

[nebkat]

BJData spec has never set the goal of following UBJSON.

I've always viewed it as the de-facto continuation of UBJSON after its abandonment but that is fair enough! I wouldn't want to suggest that it should take on that goal, but I hope you can consider the unique position that it is in right now.

---

1. packed objects {$... (or SoA struct-of-arrays or arrays-of-structs)

I am also very happy to see this implemented. It doesn't introduce any new types so I don't see it contributing to this problem. While there is some implementation complexity it is IMO completely justified by the increased efficiency. A well written encoder could even dynamically generate the schemas based on arbitrary input data so it's easy to benefit from it without making any BJData specific concessions (vs normal JSON).

---

2. extension type E

The beauty of UBJSON/BJData over all the other formats was always the one-to-one mapping to JSON.

• You could take any valid BJData and convert it to JSON and back without losing anything except maybe encoding efficiency.
• You could write APIs that accept as input and return as output both BJData and JSON and know that it will just work.
• You could (for the most part) follow existing language conventions on which language data types to use for which JSON data types when writing parsers.
• You were therefore never "locked in" to it as a format, making it easy to adopt in the first place.

Extension types:

• Are the only feature that doesn't have a direct representation in JSON, losing the one-to-one guarantee.
• Are the only type that encodes a purpose/meaning alongside its value, which is normally handled by a schema.
• Introduce (sensible, but) opinionated default data types e.g. 12 byte nanosecond epoch, 8 byte microsecond delta.
• Introduce ambiguity in terms of compatibility between different languages (particularly with the reserved extension types).

If the goal was just to provide an arbitrary length custom data type this was already possible with a byte array and application level schemas/conventions.

You don't really need to parse anything enclosed inside the E-marker. You can simply use the payloadlen to skip forward, or leave the payload buffer for other libraries to interpret. It is one of the simplest container that I can think of to provide such forward/backward compatibility. I don't know what else can be simpler, in the spirit of JSON, that can achieve this.

JSON is intentionally limited to a basic set of types to ensure universal compatibility. Standards like GeoJSON are possible by building a schema around those basic building blocks, not by introducing proprietary data types. I think this is a better approach to encoding e.g. complex numbers, especially with the addition of SoA making it 100% efficient.

Again, extension types are custom types, if parsers do not want to handle customized data types, just ignore those. Perhaps I can add language to discourage the use of E for canonical data storage. was that what your concern? that people will abuse E and store function-critical data?

It seems more likely to me that the parser would receive an extension type that it can't ignore precisely because it contains vital information. If the option is there, people will naturally use it for important data, and interoperability will depend on every implementation understanding the same extension semantics.

[flynn162]

I use the extensions E type in RPC protocol design.

When alignments are required, every byte counts. With a U8 array there is a 6 to 7-byte header representing the element type and the length: [$U#U. or [$U#u.. But there is no standard way to put the inner data type. I might want to declare a 8-bit schema ID preceding the payload, for example.

The E type allows that to be done compactly. You would write Eu<schemaId>u<len> That's 7 bytes with 1 byte of padding to spare.

Like OP mentioned, we can put raw data after the U8 array header [$U#u.. so that an entire bundle of data after that do not need to be parsed. I found that U8 arrays and extension types work well together when combined:

N[$U #u<u16OuterLen>              // begin packet content
Eu<u16typeId> u<u16InnerLen><pad> // first request
<request payload>
Eu<u16typeId> u<u16InnerLen><pad> // second request
<request payload>

In this configuration, all u16 fields do not cross the 8-byte boundary, assuming N[$U... is 8-byte aligned. My parser simply detects a fast-path pattern with 2 aligned u64 reads and some masking.

I am less focused on theoretical purity and more on the engineering side. IMO, Binary JData spec can have both ways of encapsulating raw bytes.