Add FFE test case for empty targeting key evaluation by leoromanovsky · Pull Request #6046 · DataDog/system-tests

leoromanovsky · 2026-01-16T13:31:22Z

Adds a new flag definition and test case to verify that flag evaluation succeeds when targeting key is an empty string. The flag should still match the allocation and return the expected value.

This tests that tracers properly handle empty targeting keys during evaluation (separate from exposure event handling).

Motivation

SDK Specification

OF.7
Empty string is a valid targeting key
Empty string shall be accepted as a targeting key and evaluation should proceed as normal.

While developing another system test to validate that exposures are being sent for empty targeting keys, found that in Java and NodeJS implementations an empty targeting key was causing evaluations to return the programmatic default.

__ Test_FFE_EXP_5_Missing_Targeting_Key.test_ffe_exp_5_missing_targeting_key ___

self = <tests.ffe.test_exposures.Test_FFE_EXP_5_Missing_Targeting_Key object at 0x7f5b1d4b90d0>

    def test_ffe_exp_5_missing_targeting_key(self):
        """EXP.5: Test that empty targeting key generates exposure with subject.id = ''."""
        assert self.response.status_code == 200, f"Flag evaluation failed: {self.response.text}"
    
        result = json.loads(self.response.text)
>       assert result["value"] == "value-a", f"Expected 'value-a', got '{result['value']}'"
E       AssertionError: Expected 'value-a', got 'default'
E       assert 'default' == 'value-a'
E         - value-a
E         + default

tests/ffe/test_exposures.py:958: AssertionError
------------------------------ Captured log call -------------------------------
03:49:31.761 INFO     weblog POST http://localhost:7777/ffe -> 200
- generated xml file: /home/runner/work/system-tests/system-tests/logs_feature_flagging_and_experimentation/reportJunit.xml -
=========================== short test summary info ============================
FAILED tests/ffe/test_exposures.py::Test_FFE_EXP_5_Missing_Targeting_Key::test_ffe_exp_5_missing_targeting_key
========== 1 failed, 12 passed, 2122 deselected in 124.34s (0:02:04) ===========
Error: Process completed with exit code 1.

Changes

tests/parametric/test_ffe/test-case-of-7-empty-targeting-key.json: New test case file for OF.7
tests/parametric/test_ffe/flags-v1.json: Added empty-targeting-key-flag fixture
tests/parametric/test_ffe/test_dynamic_evaluation.py:
- Added inline skip in test_ffe_flag_evaluation for Java/Node.js on OF.7 test case
- Added dedicated test_ffe_of7_empty_targeting_key test with @bug decorators

Decisions

Why both inline skip AND dedicated test?

Inline skip: Safeguard to prevent parametrized test failures for known-buggy libraries
Dedicated test with @bug decorators: Provides visibility in test reporting, registers skipped tests centrally, canonical pattern for tracking known bugs

This dual approach ensures the bugs are tracked properly while preventing CI failures.

Next Steps

Language: Java
JIRA: FFL-1729
Fix: open-feature/java-sdk#1807
Action: Merge upstream PR, then remove @bug decorator
────────────────────────────────────────
Language: Node.js
JIRA: FFL-1730
Fix: https://github.com/DataDog/openfeature-js-client/pull/new/leo/allow-empty-targeting-key
Action: Merge internal PR, then remove @bug decorator
Once fixes are merged:

Remove @bug decorators from test_ffe_of7_empty_targeting_key
Remove inline skip from test_ffe_flag_evaluation
Remove dedicated test

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-01-16T13:31:51Z

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/test-case-of-7-empty-targeting-key.json       @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
tests/parametric/test_ffe/flags-v1.json                                 @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core

Adds a new flag definition and test case to verify that flag evaluation succeeds when targeting key is an empty string. The flag should still match the allocation and return the expected value. This tests that tracers properly handle empty targeting keys during evaluation (separate from exposure event handling).

These libraries have known bugs where their OpenFeature SDKs reject empty targeting keys with a truthiness check: - Java: FFL-1729 (fix pending: open-feature/java-sdk#1807) - Node.js: FFL-1730 (fix: https://github.com/DataDog/openfeature-js-client/pull/new/leo/allow-empty-targeting-key)

@bug

- Added test_ffe_of7_empty_targeting_key with @bug decorators for Java (FFL-1729) and Node.js (FFL-1730) - Keeps inline skip in parametrized test as safeguard - Provides visibility in test reporting for known bugs

cbeauchesne · 2026-01-19T16:57:04Z

            )
+
+    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
+    @bug(context.library == "java", reason="FFL-1729")


Could you use manifest ? (ping me if you need any help)

Sure I didn't know this was undesired; there is a CI test that fails for the other test type.

nw, we just completed the scope of manifest, and the target is to use them every time it's possible.

typotter · 2026-01-20T04:48:56Z

+        is an empty string. The flag should still match allocations and return
+        the expected value, not fail with TARGETING_KEY_MISSING.
+
+        Temporary dedicated test until FFL-1729 (Java) and FFL-1730 (Node.js) are resolved.


might be useful to drop in a comment that the bugs are in the OF repositories.

Just Java; nodejs is ours. Sure doesn't hurt to add but the details are in the JIRAs and I expect this to be a short-lived comment.

typotter

LGTM for flags.

@bug

Move bug annotations for FFL-1729 (Java) and FFL-1730 (Node.js) from inline @bug decorators to manifest files as requested in code review.

cbeauchesne · 2026-01-20T09:22:54Z

+        # Skip OF.7 (empty targeting key) test for libraries with known bugs
+        # Java: FFL-1729 - OpenFeature Java SDK rejects empty targeting keys
+        # Node.js: FFL-1730 - OpenFeature JS SDK rejects empty targeting keys
+        if test_case_file == "test-case-of-7-empty-targeting-key.json":


FYI, this does not play nicely with all the downstream process that handles bug follow-up. But I do not have a clean solution to propose, so let's go with it. If I find a nice solution, I'll clean that.

Yea we end up needing to disable the entire test suite when we find a single corner case bug.

leoromanovsky added 3 commits January 16, 2026 23:14

Rename test file to include spec ID OF.7

a04a25c

leoromanovsky force-pushed the leo/ffe-empty-targeting-key-test branch from 3573388 to f3ce7cf Compare January 17, 2026 04:36

leoromanovsky added 2 commits January 16, 2026 23:40

Add dedicated OF.7 test with @bug decorators for visibility

101267b

- Added test_ffe_of7_empty_targeting_key with @bug decorators for Java (FFL-1729) and Node.js (FFL-1730) - Keeps inline skip in parametrized test as safeguard - Provides visibility in test reporting for known bugs

Fix: remove extraneous f-string prefix

e3c7467

leoromanovsky marked this pull request as ready for review January 17, 2026 05:22

leoromanovsky requested review from a team as code owners January 17, 2026 05:22

leoromanovsky assigned typotter and dd-oleksii Jan 17, 2026

cbeauchesne requested changes Jan 19, 2026

View reviewed changes

typotter reviewed Jan 20, 2026

View reviewed changes

typotter approved these changes Jan 20, 2026

View reviewed changes

typotter removed their assignment Jan 20, 2026

leoromanovsky added 2 commits January 20, 2026 00:34

Merge branch 'main' into leo/ffe-empty-targeting-key-test

40a8377

Migrate @bug decorators to manifests for empty targeting key test

f9c0656

Move bug annotations for FFL-1729 (Java) and FFL-1730 (Node.js) from inline @bug decorators to manifest files as requested in code review.

leoromanovsky requested review from a team as code owners January 20, 2026 05:42

leoromanovsky requested review from cbeauchesne, daniel-romano-DD, jandro996 and mhlidd January 20, 2026 05:42

Remove unused bug import

48deee3

cbeauchesne reviewed Jan 20, 2026

View reviewed changes

cbeauchesne approved these changes Jan 20, 2026

View reviewed changes

leoromanovsky merged commit 4683e7d into main Jan 22, 2026
400 checks passed

leoromanovsky deleted the leo/ffe-empty-targeting-key-test branch January 22, 2026 01:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add FFE test case for empty targeting key evaluation#6046

Add FFE test case for empty targeting key evaluation#6046
leoromanovsky merged 8 commits intomainfrom
leo/ffe-empty-targeting-key-test

leoromanovsky commented Jan 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jan 16, 2026 •

edited

Loading

Uh oh!

cbeauchesne Jan 19, 2026

Uh oh!

leoromanovsky Jan 20, 2026

Uh oh!

cbeauchesne Jan 20, 2026

Uh oh!

typotter Jan 20, 2026

Uh oh!

leoromanovsky Jan 20, 2026

Uh oh!

typotter left a comment

Uh oh!

cbeauchesne Jan 20, 2026

Uh oh!

leoromanovsky Jan 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

leoromanovsky commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Next Steps

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbeauchesne Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

leoromanovsky Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

cbeauchesne Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

typotter Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

leoromanovsky Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

typotter left a comment

Choose a reason for hiding this comment

Uh oh!

cbeauchesne Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

leoromanovsky Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

leoromanovsky commented Jan 16, 2026 •

edited

Loading

github-actions Bot commented Jan 16, 2026 •

edited

Loading