AVRO-2032: [Java] Add support for NaN, Infinity and -Infinity in JsonDecoder

[rroesch1]

What is the purpose of the change

JsonEncoder uses special string values to represent NaN, Infinity and -Infinity values for float and double values, but JsonDecoder does not accept these string values. This change adds support for these special values to JsonDecoder.

Verifying this change

This change added tests and can be verified as follows:

• The change adds an unit tests which verifies all 6 special cases:
  • NaN for float fields
  • NaN for double fields
  • Infinity for float fields
  • Infinity for double fields
  • -Infinity for float fields
  • -Infinity for double fields
• These values are defined in Jackson.
  StdDeserializer

Documentation

• This PR doesn't introduce a new feature but implements a behavior which is not documented at all.
• The Avro spec doesn't mention how these special cases are handled in JSON (RFC 8259 does not define numeric literals for NaN, Infinity and -Infinity)
• JsonEncoder/JsonDecoder resemble the behavior of Jackson

[martin-g]

Similar PR for the Rust SDK: #3051
Note: it also supports inf next to infinity.

[martin-g]

Note: it also supports inf next to infinity.

I see that the implementation here also supports Inf. The difference with the Rust PR is that here the values are case-sensitive.

[rroesch1]

@martin-g

• i added INF/-INF because these are the values supported by StdDeserializer in Jackson. (i don't know the reason for this). (JsonEncoder and JsonDecoder are based on Jackson)
• Please note this PR is just about the JSON encoding for records, not about the encoding for schemas (AVRO-4024: [Rust] support nan/inf/-inf as float/double default #3051 is about the JSON encoding the schemas)

[rroesch1]

fyi: looks like the C++ implementations also uses the literals "NaN", "Infinity", and "-Infinity" (see: JsonCodec.cc )

[xxchan]

The Avro spec doesn't mention how these special cases are handled in JSON

Indeed. I think we should fix that AVRO-4025.
Perhaps we could add some tests to demonstrate the current behavior (the de-facto standard) of schema / JsonEncoder, and them document the behavior into the spec. I will try to have a PR for that soon.

[xxchan]

Please note this PR is just about the JSON encoding for records, not about the encoding for schemas

Indeed. But "Except for unions, the JSON encoding is the same as is used to encode field default values (in schema)", so I think they are essentially the same (and SHOULD have the same behavior).

https://avro.apache.org/docs/1.11.1/specification/_print/#json-encoding

[zcsizmadia]

The same PR for C# #3070. We should specify the accepted strings. NaN, Infinity, -Infinity makes sense, however if if we need to support INF and -INF and/or case insensitivity, we should add all the supprted strings to the doc. I added tests for all of those cases.

[legal-s]

Hello,

Do you know when this pull request will be merged because it would fix a data deserialization problem with an entity containing the NaN value that i have?

Thanks

[gbecan]

We are also impacted by this issue and would love to see this PR merged.
The Rust and C# PRs are merged. Is there a reason why the Java one is stuck?
Happy to help if something is missing 🙏

[ajross]

We'd love to see this PR merged. Is there anything we can do to help to get this over the line?

[woppa684]

We are also impacted by this issue and would love to see this PR merged. The Rust and C# PRs are merged. Is there a reason why the Java one is stuck? Happy to help if something is missing 🙏

Same question indeed, why not merge this fix for Java? @martin-g or @xxchan is that something you can answer?

[ajross]

@RyanSkraba I saw you'd been able to get several PRs merged to this repo recently. Any chance you can help get this approved or point us to someone that can complete this PR?

[RyanSkraba]

Hello! I'm keen to have this straightened out, but my solution would be to remove JSON decoding from Avro entirely 😆 There's no reliable and portable way to represent floating point numbers in "pure" JSON, and everybody has their own unhappy workarounds and assumptions (looking at Jackson).

Seriously though, let's get this merged into Avro Java (and cherry-pick it into the next minor releases ASAP) so we can round-trip our data through a JSON encoding and decoding cycle without errors!

I created AVRO-4217 to track this discussion for the next major release.

[RyanSkraba]

Cherry-picked to branch-1.12.
Cherry-picked to branch-1.11.

[ajross]

Thanks very much @RyanSkraba !