refactor(parametric): implement version-aware RC capability definitions

[watson]

Motivation

The test_default_capability_completeness test had a chicken-and-egg problem: when adding new Remote Config (RC) capabilities to tracers, we needed workarounds like @bug, @missing_feature decorators, and dynamic detection logic (e.g., for Node.js). This made the test brittle and difficult to maintain.

The core issue: System-tests must pass on main before a tracer PR lands, but also pass on the tracer PR with new capabilities, when the exact release version is unknown at merge time.

Changes

Replace static capability dictionary with version-aware YAML configuration to eliminate chicken-and-egg problems when adding new capabilities.

Added

• tests/parametric/capabilities.yml - Defines base and version-specific Remote Config capabilities per language using SemVer ranges
• tests/parametric/capabilities_schema.json - JSON schema for validating capabilities.yml structure
• tests/test_the_test/test_capabilities.py - Automated validation tests for capabilities.yml (runs in CI on every PR)
• Helper functions _load_capabilities_yaml() and get_expected_capabilities_for_version() in test_dynamic_configuration.py

Removed

• Hardcoded DEFAULT_SUPPORTED_CAPABILITIES_BY_LANG dictionary
• @bug and @missing_feature decorators from capability test
• Dynamic Node.js capability detection workaround

Benefits

• Pre-release versions can report new capabilities without breaking tests (lenient validation)
• Release versions enforce strict capability matching (prevents forgetting to update YAML)
• Declarative configuration easier to maintain than code + decorators
• Clear documentation of when each capability was introduced per language

Note for Reviewers

Pre-release vs Release Behavior:

• Dev/pre-release versions (e.g., 6.0.0-pre, 2.7.0-dev): Tests allow extra capabilities not listed in capabilities.yml. This solves the chicken-and-egg problem during development.
• Release versions (e.g., 5.72.0, 2.7.0): Tests fail if capabilities don't match exactly. This ensures you don't forget to update capabilities.yml when preparing a release.

This means: merge tracer PRs with new capabilities first, then update capabilities.yml in system-tests before the official release.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[watson]

• refactor(parametric): implement version-aware RC capability definitions #6100 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[watson]

@DataDog/dd-trace-go-guild I tried my best to verify this, but please review 😃

[github-actions]

CODEOWNERS have been resolved as:

tests/parametric/capabilities.yml                                       @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
tests/parametric/capabilities_schema.json                               @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
tests/test_the_test/test_capabilities.py                                @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
tests/parametric/test_dynamic_configuration.py                          @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
utils/scripts/compute_libraries_and_scenarios.py                        @DataDog/system-tests-core

[datadog-official]

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b89f284 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

[cbeauchesne]

Small RC, then all good.