Improve policy coverage of tests

[baszoetekouw]

Add more policies to the tests.

I've bene tinkering around with the PolicyHarnassTest, but that's a little bit inconvenient to tests a large number of cases quickly, because it requires you to put in a policy, request and response for each usecase.

I would like to have a bit more advanced type of testing based on similar ideas as the HarnassTest.

I want to create a number of directories. Each of the dirs contains:

• a policy template policy.json
• a request template request.json
• test specifications tests.yaml

The policy template is a (jinja-like, but can be any template engine that supports loops and stuff) policy skeleton that allows substitution of template vars (which are defined in the test specification).
For example (using jinja syntax):

{
    "id": "abc",
    "policyId": "urn:surfconext:xacml:policy:id:some_name",
    "name": "some name",
    "description": "Testing templae",
    "serviceProviderIds": ["{{ policy_sps | join('", "') }}"],
    "serviceProvidersNegated": false,
    "identityProviderIds": [ "{{ policy_idp }}" ],
    "attributes": [
        {
            "name": "urn:mace:dir:attribute-def:eduPersonAffiliation",
            "value": "staff",
            "negated": false,
            "groupID": 0
        }
    ],
    "denyRule": {{ policy_is_deny | bool }},
    "allAttributesMustMatch": false,
    "created": "2025-11-25T09:33:28.216025782Z",
    "denyAdvice": "NOT ALLOWED",
    "denyAdviceNl": "MAG NIET",
    "active": true,
    "actionsAllowed": false,
    "type": "reg",
    "revisionNbr": 0,
    "activatedSr": false
}

Here we use three template vars: policy_sps (list of strings)m policy_idp (string) and policy_is_deny (bool). These are specified in the tests specification (see below).

The XACML request is a similar template:

{
   "Request": {
      "AccessSubject": {
         "Attribute": [
           {%- for a in affiliation -%}
           { "AttributeId": "urn:mace:dir:attribute-def:eduPersonAffiliation", "Value": "{{ a }}" }
           {% if not loop.last %},{% endif %}
           {%- endfor -%}
            ]
      },
      "Resource": {
         "Attribute": [
            {
               "AttributeId": "SPentityID",
               "Value": "{{ sp }}"
            },
            {
               "AttributeId": "IDPentityID",
               "Value": "{{ idp }}"
            },
            {
               "AttributeId": "ClientID",
               "Value": "EngineBlock"
            }
         ]
      }
   }
}

The test specification is a yaml file (could be json, is that's easier) in which the tests along with the template vars are defined. The file looks like this:

• One file can contain multiple tests.
• Each test contains a vars section specifying global template vars for all subtests
• Each test also contains a tests section defifing the subtests to run
• Each subtest has a vars section defining addition template variables
• Each subtest also has a result section, that defines a Decision that is to be matched against the PDP Response
• Each subtest can also have an Attribute section that defines attributes that are matched against

For example:

test_two_sps:
  vars:
    policy_idp: "http://idp1"
    policy_sp: ["http://sp1", "http://sp2"]
    idp: "http//idp1"
  tests:
    - vars: { "policy_is_deny": false, "sp": "http://sp1", "affiliation": ["student"] }
      result: { "Decision": "Permit" }
    - vars: { "policy_is_deny": false, "sp": "http://sp1", "affiliation": ["staff"] }
      result: { "Decision": "Deny", Attribute: {"DenyMessage:en": "NOT ALLOWED"  }
    - vars: { "policy_is_deny": false, "sp": "http://sp2", "affiliation": ["student"] }
      result: { "Decision": "NotApplicable" }
    - vars: { "policy_is_deny": true, "sp": "http://sp1", "affiliation": ["student"] }
      result: { "Decision": "Deny", Attribute: {"DenyMessage:en": "NOT ALLOWED"  }
    - vars: { "policy_is_deny": true, "sp": "http://sp1", "affiliation": ["staff"] }
      result: { "Decision": "Permit" }
    - vars: { "policy_is_deny": true, "sp": "http://sp2", "affiliation": ["student"] }
      result: { "Decision": "NotApplicable" }

test_one_sp:
  vars:
    policy_idp: "http://idp1"
    policy_sp: ["http://sp1"]
    idp: "http//idp1"
  tests:
    - vars: { "policy_is_deny": false, "sp": "http://sp1", "affiliation": ["student"] }
      result: { "Decision": "Permit" }
    - vars: { "policy_is_deny": false, "sp": "http://sp1", "affiliation": ["staff"] }
      result: { "Decision": "Deny", Attribute: {"DenyMessage:en": "NOT ALLOWED"  }
    - vars: { "policy_is_deny": false, "sp": "http://sp2", "affiliation": ["student"] }
      result: { "Decision": "NotApplicable" }
    - vars: { "policy_is_deny": true, "sp": "http://sp1", "affiliation": ["student"] }
      result: { "Decision": "Deny", Attribute: {"DenyMessage:en": "NOT ALLOWED"  }
    - vars: { "policy_is_deny": true, "sp": "http://sp1", "affiliation": ["staff"] }
      result: { "Decision": "Permit" }
    - vars: { "policy_is_deny": true, "sp": "http://sp2", "affiliation": ["student"] }

[baszoetekouw]

Seems this is getting too complex. Alternative solution: populate TestHarnass test cases from a python script.