Skip to content

CLDSRV-835: Retry docker pull flakiness #6101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
BourgoisMickael wants to merge 1 commit into
base: development/9.2
Choose a base branch
from
+109 −0
Open

CLDSRV-835: Retry docker pull flakiness #6101

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions .github/workflows/tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,10 @@ env:
# https://github.com/git-lfs/git-lfs/issues/5749
GIT_CLONE_PROTECTION_ACTIVE: 'false'
AWS_SDK_JS_SUPPRESS_MAINTENANCE_MODE_MESSAGE: '1'
# Sometimes CI fails to pull image: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
# Nothing is configurable in docker, so we use a custom shell retry pull step.
DOCKER_PULL_RETRIES: 3
DOCKER_PULL_RETRY_DELAY: 15

permissions:
contents: read
Expand Down Expand Up @@ -264,6 +268,15 @@ jobs:
password: ${{ github.token }}
- name: Setup CI environment
uses: ./.github/actions/setup-ci
- name: Pull CI services
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose --profile sproxyd --profile mongo pull && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
Copy link

@claude claude bot Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the last retry attempt, if the pull fails the loop sleeps for 15s before hitting exit 1. You can skip the sleep on the final attempt to avoid wasting CI time.

Same applies to all other occurrences of this pattern.

— Claude Code

exit 1
Copy link

@claude claude bot Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retry block is duplicated 12 times across jobs. Consider extracting it into a reusable composite action under .github/actions/ (like the existing setup-ci), accepting compose-args as an input. This would reduce each call site to ~3 lines and make future changes to retry logic a single-point edit.

— Claude Code

Copy link
Contributor Author

@BourgoisMickael BourgoisMickael Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are all different because the image pulled are copied from the docker compose up line

Copy link
Contributor

@francoisferrand francoisferrand Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would side with Claude: this is the same code, the only difference are some docker compose params.... which can be passed in argument to a reusable script or github action

working-directory: .github/docker
- name: Setup CI services
run: docker compose --profile sproxyd --profile mongo up -d
working-directory: .github/docker
Expand Down Expand Up @@ -326,6 +339,15 @@ jobs:
sudo mkdir -p /logs
sudo chmod 0777 /logs
sudo touch /logs/server-access.log && sudo chmod 0666 /logs/server-access.log
- name: Pull CI services
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose --profile mongo pull && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
exit 1
working-directory: .github/docker
- name: Setup CI services
run: docker compose --profile mongo up -d
working-directory: .github/docker
Expand Down Expand Up @@ -387,6 +409,15 @@ jobs:
sudo mkdir -p /logs
sudo chmod 0777 /logs
sudo touch /logs/server-access.log && sudo chmod 0666 /logs/server-access.log
- name: Pull CI services
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose --profile mongo pull && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
exit 1
working-directory: .github/docker
- name: Setup CI services
run: docker compose --profile mongo up -d
working-directory: .github/docker
Expand Down Expand Up @@ -460,6 +491,15 @@ jobs:
sudo mkdir -p /logs
sudo chmod 0777 /logs
sudo touch /logs/server-access.log && sudo chmod 0666 /logs/server-access.log
- name: Pull CI services
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose pull && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
exit 1
working-directory: .github/docker
- name: Setup CI services
run: docker compose up -d
working-directory: .github/docker
Expand Down Expand Up @@ -552,6 +592,15 @@ jobs:
sed -i 's/\("METADATA_NEW_BUCKETS_VFORMAT":\s*\)"[^"]*"/\1"${{ matrix.vformat }}"/' .github/docker/md-config.json
- name: Copy S3C config
run: cp .github/docker/config.s3c.json tests/functional/sse-kms-migration/config.json
- name: Pull CI services
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose pull redis sproxyd metadata-standalone vault-sse-before-migration cloudserver-sse-before-migration && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
exit 1
working-directory: .github/docker
- name: Setup CI services
run: docker compose up -d --quiet-pull redis sproxyd metadata-standalone vault-sse-before-migration cloudserver-sse-before-migration
working-directory: .github/docker
Expand Down Expand Up @@ -638,6 +687,15 @@ jobs:
uses: actions/checkout@v4
- name: Setup CI environment
uses: ./.github/actions/setup-ci
- name: Pull CI services
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose pull && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
exit 1
working-directory: .github/docker
- name: Setup CI services
run: docker compose up -d
working-directory: .github/docker
Expand Down Expand Up @@ -694,6 +752,15 @@ jobs:
uses: actions/checkout@v4
- name: Setup CI environment
uses: ./.github/actions/setup-ci
- name: Pull CI services
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose --profile mongo pull && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
exit 1
working-directory: .github/docker
- name: Setup CI services
run: docker compose --profile mongo up -d
working-directory: .github/docker
Expand Down Expand Up @@ -754,6 +821,15 @@ jobs:
- name: Copy KMIP certs
run: cp -r ./certs /tmp/ssl-kmip
working-directory: .github/pykmip
- name: Pull CI services
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose --profile pykmip pull && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
exit 1
working-directory: .github/docker
- name: Setup CI services
run: docker compose --profile pykmip up -d
working-directory: .github/docker
Expand Down Expand Up @@ -820,6 +896,15 @@ jobs:
configs/kmip-cluster.json \
> config.json
working-directory: tests/functional/sse-kms-migration
- name: Pull CI services
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose pull redis pykmip cloudserver-sse-before-migration && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
exit 1
working-directory: .github/docker
- name: Setup CI services
run: docker compose up -d --quiet-pull redis pykmip cloudserver-sse-before-migration
working-directory: .github/docker
Expand Down Expand Up @@ -926,6 +1011,18 @@ jobs:
- name: Copy base config
run: cp configs/base.json config.json
working-directory: tests/functional/sse-kms-migration
- name: Pull CI services (with old cloudserver image before sse migration)
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose pull redis vault-sse-before-migration cloudserver-sse-before-migration && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
exit 1
working-directory: .github/docker
env:
CLOUDSERVER_IMAGE: ${{ env.CLOUDSERVER_IMAGE_BEFORE_SSE_MIGRATION }}
VAULT_IMAGE: ${{ env.VAULT_IMAGE_BEFORE_SSE_MIGRATION }}
- name: Setup CI services (with old cloudserver image before sse migration)
run: docker compose up -d --quiet-pull redis vault-sse-before-migration cloudserver-sse-before-migration
working-directory: .github/docker
Expand Down Expand Up @@ -967,6 +1064,18 @@ jobs:
configs/sseMigration.json \
> config.json
working-directory: tests/functional/sse-kms-migration
- name: Pull CI services (migration cloudserver image)
run: |
for i in $(seq 1 $DOCKER_PULL_RETRIES); do
docker compose pull ${{ matrix.kms.container }} vault-sse-migration cloudserver-sse-migration && exit 0
echo "Attempt $i failed, retrying in ${DOCKER_PULL_RETRY_DELAY}s..."
sleep $DOCKER_PULL_RETRY_DELAY
done
exit 1
working-directory: .github/docker
env:
S3KMS: ${{ matrix.kms.provider }} # S3
KMS_BACKEND: ${{ matrix.kms.provider == 'aws' && 'aws' || '' }} # vault only supports aws
- name: Replace old cloudserver image with current one
run: |-
docker compose down cloudserver-sse-before-migration vault-sse-before-migration
Expand Down
Loading