Run zenko tests out of cluster by SylvainSenechal · Pull Request #2336 · scality/Zenko

SylvainSenechal · 2026-02-26T10:01:26Z

bert-e · 2026-02-26T10:01:30Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

Issue: ZENKO-5201

SylvainSenechal · 2026-03-23T14:10:03Z

.devcontainer/setup.sh

    )
 done

-(


No need for this anymore as we don't need the container, and config is handled differently

SylvainSenechal · 2026-03-23T14:19:21Z

.github/actions/deploy/action.yaml

-      working-directory: ./.github/scripts/end2end
-    - name: Linting
-      shell: bash
-      run: bash run-e2e-test.sh "end2end" ${E2E_IMAGE_NAME}:${E2E_IMAGE_TAG} "lint" "default"


No need to do the linting in the deploy action..

this deploy action is run multiple times in the ci

it was building an image, taking time, all that just for linting
Now you will see later in end2end.yaml there is a simple yarn run lint

SylvainSenechal · 2026-03-23T14:20:45Z

.github/scripts/end2end/configure-e2e-ctst.sh

 DIR=$(dirname "$0")

+# Set up ingress endpoints and /etc/hosts for out-of-cluster access
+source "$DIR/configure-e2e-endpoints.sh"


This setup will be common to ctst and end2end tests.
Hopefully we will soon be able to have a common setup for both type of tests, to avoid having to go from configure-ctst to configure e2e, with many things that are common

SylvainSenechal · 2026-03-23T14:21:18Z

.github/scripts/end2end/configure-e2e-endpoints.sh

@@ -0,0 +1,139 @@
+#!/bin/bash


This should be idempotent. I used it multiple times in Codespace without issues

SylvainSenechal · 2026-03-23T14:22:07Z

.github/scripts/end2end/configure-e2e.sh

-
-SERVICE_ACCOUNT="${ZENKO_NAME}-config"
-POD_NAME="${ZENKO_NAME}-config"
-MANAGEMENT_ENDPOINT="http://${ZENKO_NAME}-management-orbit-api:5001"


This is the kind of things that we have to change now that the test doesn't run in a pod within the Zenko cluster

SylvainSenechal · 2026-03-23T14:23:10Z

.github/scripts/end2end/configure-e2e.sh

+pip3 install --break-system-packages -r "$ZENKO_TESTS_DIR/requirements.txt"
+
+cd "$ZENKO_TESTS_DIR"
+
+envsubst < e2e-config.yaml.template > e2e-config.yaml


This used to be setup in the Dockerfile that I deleted

SylvainSenechal · 2026-03-23T14:24:05Z

.github/scripts/end2end/setup-e2e-env.sh

@@ -0,0 +1,130 @@
+#!/usr/bin/env bash


I came up with this to deal with the env variables.
It's a bit tricky as I wanted it to work for both github ci and codespace

Variables setup here are pretty much the same as the ones that were setup in the (now deleted) run-e2e-test.sh file

SylvainSenechal · 2026-03-23T14:26:24Z

.github/workflows/end2end.yaml

      - name: Deploy second Zenko for PRA
        run: bash deploy-zenko.sh end2end-pra default './configs/zenko.yaml'
-        env:
-          ZENKO_MONGODB_DATABASE: pradb


setup in prepare-pra.sh directly

SylvainSenechal · 2026-03-23T14:27:20Z

.github/workflows/end2end.yaml

          cache-from: type=gha,scope=kafka-connect-${{ env.KAFKA_CONNECT_TAG }}
          cache-to: type=gha,mode=max,scope=kafka-connect-${{ env.KAFKA_CONNECT_TAG }}

-  build-test-image:


No more image, just a new lint step as I removed it from the deploy action

SylvainSenechal · 2026-03-23T14:29:56Z

.github/workflows/end2end.yaml

-        working-directory: ./.github/scripts/end2end
+        run: |
+          source .github/scripts/end2end/setup-e2e-env.sh
+          yarn run test_operator


I prefer this, simple, direct use of yarn commands instead of going through scripts with FOUR arguments 🧐

I just don't really like having to source the env var for each run so if you have suggestion I'm open

you can instead add a first step which updates the env for followup steps

c.f. https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-commands#setting-an-environment-variable

SylvainSenechal · 2026-03-23T14:31:21Z

tests/zenko_tests/e2e_config/accounts.py

        RoleSessionName='end2end',
        WebIdentityToken=token,
-        DurationSeconds=60 * 60 * 12, # 12 hrs
+        DurationSeconds=60 * 60 * 12, # 12 hrs (max allowed by STS for assume role)


I wanna keep this comment, I tried to use 30 days because in my codespace it was expiring after 12h, but when I used 30 days it broke because the api only accepts 12h

SylvainSenechal · 2026-03-23T14:32:21Z

tests/zenko_tests/node_tests/package.json

    "mocha": "^10.0.0",
    "mocha-junit-reporter": "^2.2.1",
    "mocha-multi-reporters": "^1.1.7",
-    "mocha-tags": "^1.0.1",


Never used,

Could be useful in theory,

Will never be used in practice, so just add again if needed (will never be needed 🌚 )

SylvainSenechal · 2026-03-23T14:35:34Z

tests/zenko_tests/node_tests/package.json

+    "test_operator": "mocha --exit -t 10000 --reporter mocha-multi-reporters --reporter-options configFile=config.json,cmrOutput=mocha-junit-reporter+testsuitesTitle+test_operator ./init_test.js",
+    "test_smoke": "mocha --exit -t 10000 --reporter mocha-multi-reporters --reporter-options configFile=config.json,cmrOutput=mocha-junit-reporter+testsuitesTitle+test_smoke --recursive smoke_tests",
+    "test_iam_policies": "mocha --exit -t 15000 --reporter mocha-multi-reporters --reporter-options configFile=config.json,cmrOutput=mocha-junit-reporter+testsuitesTitle+test_iam_policies --recursive iam_policies",
    "test_all_extensions": "run-p --aggregate-output test_crr test_aws_crr test_expiration test_transition test_ingestion_oob_s3c",


Btw, I just had a look at end2end-sharded. The step "Run backbeat end to end tests", which is running test_all_extensions, now runs within ~12 min, against ~28min before.

I had to double check, as I thought some tests were simply not run, but we run the same tests. I think previously the parallel run was not working, maybe because of some weird stuff happening when this command is run in a pod, the pod can't run stuff in parallel or something.

it seems this step perform a yarn install: by having it running directly in the runner, it will now be cached (automatically!) by setup-node, and would thus be much faster?

Yeah also possible

francoisferrand · 2026-03-23T18:28:39Z

.github/scripts/end2end/setup-e2e-env.sh

+    else
+        trap "kill ${_MONGO_PF_PID} 2>/dev/null || true" EXIT
+    fi
+    sleep 2


the port forward isn't instant, but yeah this is a flaky risk, i changed it so we wait till its ready instead

.github/scripts/end2end/setup-e2e-env.sh

.devcontainer/setup.sh

Issue: ZENKO-5201

…ster Issue: ZENKO-5201

… out of cluster Issue: ZENKO-5201

Issue: ZENKO-5201

bert-e · 2026-03-26T16:40:59Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

delthas · 2026-03-30T09:14:37Z

.github/scripts/end2end/configure-e2e.sh

-  --command -- python3 configuration.py
+# Run configuration.py directly
+ZENKO_TESTS_DIR="$DIR/../../../tests/zenko_tests"
+pip3 install --break-system-packages -r "$ZENKO_TESTS_DIR/requirements.txt"


I wouldn't want to break my system packages 😁 If we need to install dependencies, could we perhaps use a venv (maybe use uv?)

I was thinking about using uv, maybe in another pr, but i could add it now 🤔

SylvainSenechal · 2026-03-30T16:12:57Z

/approve

bert-e · 2026-03-30T16:13:07Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: approve

bert-e · 2026-03-31T15:12:16Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.14

The following branches will NOT be impacted:

development/2.10
development/2.11
development/2.12
development/2.13
development/2.5
development/2.6
development/2.7
development/2.8
development/2.9

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2026-03-31T17:43:56Z

Queue build failed

The corresponding build for the queue failed:

Checkout the status page.
Identify the failing build and review the logs.
If no issue is found, re-run the build.
If an issue is identified, checkout the steps below to remove
the pull request from the queue for further analysis and maybe rebase/merge.

Remove the pull request from the queue

Add a /wait comment on this pull request.
Click on login on the status page.
Go into the manage page.
Find the option called Rebuild the queue and click on it.
Bert-E will loop again on all pull requests to put the valid ones
in the queue again, while skipping the one with the /wait comment.
Wait for the new queue to merge, then merge/rebase your pull request
with the latest changes to then work on a proper fix.
Once the issue is fixed, delete the /wait comment and
follow the usual process to merge the pull request.

bert-e · 2026-04-01T08:38:19Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.14

The following branches have NOT changed:

development/2.10
development/2.11
development/2.12
development/2.13
development/2.5
development/2.6
development/2.7
development/2.8
development/2.9

Please check the status of the associated issue ZENKO-5201.

Goodbye sylvainsenechal.

SylvainSenechal force-pushed the improvement/ZENKO-5201 branch 5 times, most recently from d6a0f4c to 245e27d Compare February 26, 2026 22:10

SylvainSenechal force-pushed the improvement/ZENKO-5201 branch from 77b703d to 8d5ede4 Compare March 20, 2026 11:47

document token 12h lifespan in codespace, and logs observability

1b3fbc6

Issue: ZENKO-5201

SylvainSenechal force-pushed the improvement/ZENKO-5201 branch from 8d5ede4 to 18c02d8 Compare March 23, 2026 09:26

SylvainSenechal added 2 commits March 23, 2026 10:42

remove useless mocha tags

cbbcb61

Issue: ZENKO-5201

simplify linting and remove it from deploy action

ed605fc

Issue: ZENKO-5201

SylvainSenechal force-pushed the improvement/ZENKO-5201 branch from 18c02d8 to 733afdf Compare March 23, 2026 11:51

SylvainSenechal commented Mar 23, 2026

View reviewed changes

scality deleted a comment from bert-e Mar 23, 2026

SylvainSenechal requested review from a team, delthas, francoisferrand and maeldonn March 23, 2026 14:41

SylvainSenechal removed the request for review from maeldonn March 23, 2026 14:41

SylvainSenechal marked this pull request as ready for review March 23, 2026 14:41

francoisferrand approved these changes Mar 23, 2026

View reviewed changes

scality deleted a comment from bert-e Mar 26, 2026

SylvainSenechal added 5 commits March 26, 2026 17:30

run zenko node tests out of zenko cluster

fac0083

Issue: ZENKO-5201

remove unused files following zenko tests that are now run out of clu…

b87ae37

…ster Issue: ZENKO-5201

update dev container documentation and setup as the tests are now run…

1348694

… out of cluster Issue: ZENKO-5201

improve sourcing of env var for ci

64c1ca3

Issue: ZENKO-5201

reduce flakiness risk on port forward setup

fd8ba68

Issue: ZENKO-5201

SylvainSenechal force-pushed the improvement/ZENKO-5201 branch from 733afdf to fd8ba68 Compare March 26, 2026 16:39

scality deleted a comment from bert-e Mar 26, 2026

delthas reviewed Mar 30, 2026

View reviewed changes

delthas approved these changes Mar 31, 2026

View reviewed changes

bert-e merged commit e584f2e into development/2.14 Apr 1, 2026
39 of 42 checks passed

bert-e deleted the improvement/ZENKO-5201 branch April 1, 2026 08:38

                   )
               done
-              (

Conversation

SylvainSenechal commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Feb 26, 2026

Hello sylvainsenechal,

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bert-e commented Mar 26, 2026

Waiting for approval

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal commented Mar 30, 2026

Uh oh!

bert-e commented Mar 30, 2026

Waiting for approval

Uh oh!

bert-e commented Mar 31, 2026

In the queue

Uh oh!

bert-e commented Mar 31, 2026

Queue build failed

Uh oh!

bert-e commented Apr 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SylvainSenechal commented Feb 26, 2026 •

edited

Loading

SylvainSenechal Mar 23, 2026 •

edited

Loading