[envoy][haproxy] Usage of scenario groups for appsec tests

[e-n-0]

Motivation

• Enabling more tests for proxies (envoy/haproxy) using the new manifests implemented in [envoy / haproxy] Create new components manifests #6024.
• Reducing decorators pain to include tests into similar scenarios (proxies, lambda)
• Catching all non "default" tagged tests

Changes

This PR introduces some change in decoration of tests to be included into scenarios:

• Previously tests were added into scenarios by their corresponding decorator, but here we had new scenarios that was including the same tests. So now tests have a scenario group decorator instead of a scenario. Scenarios now have to "subscribe" to their corresponding scenario group where all the matchings tests are selected.
• When a test doesn't have a scenario decorator, it was automatically added to the DEFAULT scenario. Now this behavior have a bit changed: it is actually bound to the DEFAULT scenario group. This doesn't change the existing default behavior but this allows default scenarios of proxies or lambda to also have these tests selected.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/envoy.yml                                                     @DataDog/system-tests-core
manifests/haproxy.yml                                                   @DataDog/system-tests-core
tests/appsec/api_security/test_api_security_rc.py                       @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/api_security/test_apisecurity_telemetry.py                 @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/api_security/test_custom_data_classification.py            @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/api_security/test_schemas.py                               @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_alpha.py                                              @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_asm_standalone.py                                     @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_automated_login_events.py                             @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_automated_user_and_session_tracking.py                @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_blocking_addresses.py                                 @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_conf.py                                               @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_extended_data_collection.py                           @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_extended_header_collection.py                         @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_fingerprinting.py                                     @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_ip_blocking_full_denylist.py                          @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_only_python.py                                        @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_remote_config_rule_changes.py                         @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_reports.py                                            @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_request_blocking.py                                   @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_trace_tagging.py                                      @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_traces.py                                             @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_user_blocking_full_denylist.py                        @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_versions.py                                           @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/waf/test_blocking.py                                       @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/waf/test_blocking_security_response_id.py                  @DataDog/asm-libraries @DataDog/system-tests-core
tests/remote_config/test_remote_configuration.py                        @DataDog/remote-config @DataDog/system-tests-core
tests/test_config_consistency.py                                        @DataDog/system-tests-core
tests/test_resource_renaming.py                                         @DataDog/system-tests-core
tests/test_scrubbing.py                                                 @DataDog/system-tests-core
tests/test_semantic_conventions.py                                      @DataDog/system-tests-core
tests/test_standard_tags.py                                             @DataDog/system-tests-core
tests/test_the_test/test_group_rules.py                                 @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_context/_scenarios/core.py                                       @DataDog/system-tests-core
utils/_context/_scenarios/go_proxies.py                                 @DataDog/system-tests-core
utils/build/docker/envoy/envoy.yaml                                     @DataDog/system-tests-core

[datadog-official]

⚠️ Tests

✨ Fix all issues with Cursor

⚠️ Warnings

❄️ 1 New flaky test detected

tests.test_resource_renaming.Test_Resource_Renaming_Stats_Aggregation_Keys.test_stats_aggregation_with_method_and_endpoint[haproxy-spoa] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Expected 5 hits for GET /resource_renaming/api/users/{param:int}, got 1
assert 1 == 5

self = <tests.test_resource_renaming.Test_Resource_Renaming_Stats_Aggregation_Keys object at 0x7fdc4488b830>

    def test_stats_aggregation_with_method_and_endpoint(self):
        """Test that stats are aggregated by method and endpoint"""
        stats_points = []
        for stats_data in interfaces.library.get_data("/v0.6/stats"):
            stats = stats_data.get("request", {}).get("content", {}).get("Stats", [])
...

ℹ️ Info

🧪 All tests passed

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 794be8d | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

[nccatoni]

I’m not entirely sure what you’re aiming to achieve here, could you please provide a bit more detail?

Additionally, the second point in your changes section doesn’t seem to align with the modifications you’ve made. The default scenario group is not a scenario group that is automatically applied to all tests without a scenario group; it is simply the scenario group that contains the default scenario. Moreover, it is deprecated and is currently being removed in this PR.

[e-n-0]

@nccatoni The idea of the PR you mentioned came up because of this current PR.
I talked with Charles and there was some misunderstanding about the default scenario and how the scenario for envoy/haproxy were made (weblog as infrastructure but still testing a go library behind).

I'll close this PR for now as it will be superseded by a rewrite of weblog "as customizable client infrastructure", where the current envoy/haproxy infra could live inside, and perform the current scenarios on these.