`CelUnknownSet` when null checking a null value in a map

[RomanPodkovyrin]

Problem

We have an AVRO data that we want to evaluate with a rule value.pet == null to preserve int and doubles we have chosen to convert to Java Map rather that JSON before passing to CEL evaluator.

For this I have converted avro into a java map to be passed into CEL ENGINE.

The conversion logic looks something like this

  public static Map<String, Object> avroToMap(GenericRecord genericRecord) {
      Map<String, Object> map = new HashMap<>();
      genericRecord.getSchema().getFields().forEach(field -> {
          Object value = genericRecord.get(field.name());
          switch (value) {
              case null -> map.put(field.name(), null);
              case Integer i -> map.put(field.name(), i.longValue());
              case Utf8 utf8 -> map.put(field.name(), utf8.toString());
              case GenericRecord nested -> map.put(field.name(), avroToMap(nested));
              case GenericData.Array<?> array -> {
                  List<Object> list = new ArrayList<>();
                  for (Object elem : array) {
                      list.add(avroToMap((GenericRecord) elem));
                  }
                  map.put(field.name(), list);
              }
              default -> map.put(field.name(), value);
          }
      });
      return map;

It works fine for most cases accept null check eg value.pet == null when the value is set to null in the map.
This causes cel engine to return AutoValue_CelUnknownSet

The map does represent a null value in the map correctly and the expression evaluates

• has(value.pet) returns true
• 'pet' in value return false
• type(value.pet) == null_type returns CelUnknownSet
• value.pet == null returns CelUnknownSet

When looking at selectField I can see that it's extracting null from the map correcty.

Which then get's feed into IntermediateResult.create

But before that valueOrUnknown Checks if the value is null or instance of CelUnknownSet. Hence it why null value returns a CelUnknownSet?

What i don't understand is how do we set the field to a null value and get boolean evaluation on it.

Solution ?

After some further poking around and comparing JSON and JAVA MAP being passed into CEL. I have noticed the following difference.

When debugging inside CEL, and looking at the map type inside selectField

AVRO converted to JSON.

"pet" -> {NullValue}<NULL_VALUE>

---

AVRO to JAVA MAP with Java null

"pet" -> null

---

As we can see java null doesn't get recognised as a null value, but rather as an unknown field. This makes CEL return CelUnknownSet
when valueOrUnknown is called. So null values needs to be set to something else so that it is interpreted as null correctly

When using NullValue.NULL_VALUE from import com.google.protobuf.NullValue;

We get the expected null type in CEL

"pet" -> {NullValue}<NULL_VALUE>

This seems a bit counter intuitive and not explicitly clear when passing a JAVA Map, which can cause issues. Would this be a correct way of handling this or have I missed something?

[l46kok]

In general: CEL optional syntax is the canonical way of handling absence of a field value. To enable in CEL-Java, add CelOptionalLibrary to your environment.

I think has(value.pet) returning true for value map should be considered a bug, but I'd have to cross reference other stacks to see how it's handled there. The reason why the has check returns true today is that the map key for pet is populated, and we simply do a containsKey check on it, so the presence test will pass even though it's a null value. In any case, Java nulls isn't something that's supported in CEL-Java and you should generally avoid providing that as an input value to the Cel evaluator.

CEL null type (NullValue.NULL_VALUE), which is equivalent to google.protobuf.NullValue, is not the same as Java nulls: it is used to identify JSON null values and to distinguish empty well known type fields from default valued ones (see spec for details). So if the expression being evaluated is an actual JSON, value.pet == null is appropriate but otherwise, I'd strongly encourage leveraging optional syntax.