`DOCKER_AUTH_CONFIG` takes precedence over `docker login`

[fabriceclementz]

Description

First of all, apologies if this is not the appropriate place to raise this — please feel free to redirect me if this should be handled elsewhere.

Context

Since the merge of PR #6008, we're experiencing issues when pushing to a private registry from a GitLab CI job using the docker:dind image.

Our GitLab CI pipelines pull images from a private registry using a read-only service account. To support this, we initialize the DOCKER_AUTH_CONFIG environment variable with a base64-encoded JSON config containing the read-only credentials (following the principle of least privilege). This is mandatory to allow the GitLab Runner to pull our job image.

Later in the pipeline, we perform a docker login with another service account that has write permissions, in order to push new images. The login correctly updates the ~/.docker/config.json file.

Issue

After the change introduced by this PR #6008, it seems that the DOCKER_AUTH_CONFIG environment variable continues to take precedence, even after a successful docker login. As a result, docker push fails with permission denied errors because it is still using the read-only credentials from DOCKER_AUTH_CONFIG.

Furthermore, there is no message or warning from the Docker CLI indicating that DOCKER_AUTH_CONFIG is being used in preference to the updated login credentials. This makes it especially difficult to diagnose the root cause, as one would expect the docker login command to override or be honored for subsequent operations.

Previously, docker login would override the current auth context, allowing the push to succeed using the updated credentials.

Maybe I misunderstood this change and we could do otherwise? Or maybe this is a bug introduced with this PR. Thanks for your help.

Reproduce

1. Use docker:dind in GitLab CI.
2. Set DOCKER_AUTH_CONFIG with read-only credentials to access a private registry.
3. Run docker login private-registry.xxx.com with credentials that have write access.
4. Attempt to run docker push private-registry.xxx.com/repo/image:tag.

unauthorized: unauthorized to access repository: repo/image, action: push: unauthorized to access repository: repo/image, action: push

Expected behavior

docker push should succeed using the credentials updated via docker login.

docker version

Client:
 Version:    28.3.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.25.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.37.3
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 28.3.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.6-0-ge89a299
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.16.14-1.el8.elrepo.x86_64
 Operating System: Alpine Linux v3.22 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.8GiB
 Name: eae77d15590e
 ID: 024926f0-57c7-4049-b2a5-c3e33d4dca88
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false

Additional Info

No response

[julien-sysadmin-bzh]

Same problem here, we had to downgrade to previous docker version.

[carlos-vl]

I also had to downgrade to a previous version for this exact reason

[gdegoulet]

Nothing should have precedence when you run explictly "docker login -u" !

[julien-sysadmin-bzh]

Problem reproduced with following gitlab-ci.yml example:

stages:
  - build_image

variables:
  DOCKER_AUTH_CONFIG:
    value: "{\"auths\": {\"myrepo.mydomain.com\":{\"auth\": \"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"}}}"
    description: "docker auth"

build_test:
  extends:
    - .container-image:base
  stage: build_image
  variables:
    CONTAINER_IMAGE_NAME: test
    CONTAINER_IMAGE_TAG: 1.0.0

.container-image:base:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $CI_REGISTRY/$CI_PROJECT_PATH/$CONTAINER_IMAGE_NAME:$CONTAINER_IMAGE_TAG .
    - docker push $CI_REGISTRY/$CI_PROJECT_PATH/$CONTAINER_IMAGE_NAME:$CONTAINER_IMAGE_TAG

• DOCKER_AUTH_CONFIG is defined as global variable
• docker login is performed and should take precedence over the variable according to previous comment

case 1: DOCKER_AUTH_CONFIG is a valid token but not for current repository

docker push is refused with the following message

denied: requested access to the resource is denied

case 2: DOCKER_AUTH_CONFIG is not a valid token

docker push is performed with credentials from docker login

Failed to create credential store from DOCKER_AUTH_CONFIG:  illegal base64 data at input byte 44

[tobiasbaehr]

My workaround unset DOCKER_AUTH_CONFIG where I use docker login.

[fabriceclementz]

My workaround unset DOCKER_AUTH_CONFIG where I use docker login.

Thanks for the suggestion — I'm actually using the same workaround (unset DOCKER_AUTH_CONFIG) before docker login, and it does solve the issue for now.

However, from my point of view, this workaround isn't ideal or sustainable in the long run, as it requires modifying all affected pipelines. The current behavior also feels a bit unintuitive: without any warning or log indicating that DOCKER_AUTH_CONFIG takes precedence over docker login, it can lead to a lot of wasted debugging time for developers trying to understand why their push fails despite having valid credentials.

It would be helpful if either the precedence was more clearly documented or if the CLI emitted a warning when credentials from docker login are being ignored in favor of DOCKER_AUTH_CONFIG.

If needed and acceptable, I would be happy to help contribute a patch — whether to improve the behavior or to make the precedence and fallback logic more transparent through clearer messaging or documentation.

[carlos-vl]

Yes, at least a warning on the command line output would be helpful. My pipelines started failing suddenly and I had to spend some time trying ot understand the cause. Such a warning would have made it obvious and saved me some time.

[Benehiko]

Hi all, apologies for the disruption this has caused you 😔.

We tried to align the variable name with what is currently in use in the ecosystem and took inspiration from GitLab documentation as well as Testcontainers. Unfortunately I did not predict that CI workflows would be affected if they also relied upon docker login after setting DOCKER_AUTH_CONFIG - which was an oversight on my part.

The intent was to natively support DOCKER_AUTH_CONFIG in the CLI so that CI workflows wouldn't need to write to the Docker config file (./docker/config.json) or do a docker login. In most cases a docker login to a registry only does a credential check and does not issue a token. That means setting DOCKER_AUTH_CONFIG and doing a docker login would be the same. The DOCKER_AUTH_CONFIG environment variable takes precedence over the config file to keep the behavior consistent with how other software on Unix function (env > config).

That said, we are looking into a fix so that we can still move forward with supporting DOCKER_AUTH_CONFIG while preserving backwards compatibility with docker login usage.

[wdoekes]

Thanks for acknowledging the disruption 👍

My 2 cents: don't change behaviour now. We've found it and have done the appropriate workarounds. (And will happily use the new feature, which is nice.)

As @carlos-vl mentioned: a welcome addition would be a warning on docker login ... alerting people of the new/changed behaviour.

[heresie]

Hi @Benehiko, thank you for your response.

I respectfully disagree with @wdoekes regarding the urgency of the patch release.

The current workaround - temporarily updating all CI/CD DOCKER_AUTH_CONFIG credentials to use read/write accounts on registries - should be avoided as much as possible. It violates the principle of least privilege, which we consider essential to uphold.

Furthermore, modifying all CI/CD pipelines across multiple projects and GitLab instances to include an unset DOCKER_AUTH_CONFIG command is not a viable solution on our side. It would impact too many teams and repositories, making the change both costly and error-prone.

More broadly, this approach contradicts the widely accepted configuration precedence model used in most open-source tools. Typically, configuration sources are prioritized as follows (from lowest to highest):

1. Built-in defaults
2. Global configuration files (e.g., /etc/, ~/.config/)
3. Environment variables
4. User or project-specific configuration files
5. Explicit commands (e.g., docker login)

In this model, explicit user commands like docker login are expected to take precedence over all other sources, including environment variables such as DOCKER_AUTH_CONFIG. Ignoring that priority breaks consistency and undermines user expectations.

As @gdegoulet rightly pointed out, this behavior should remain authoritative and predictable.

[wdoekes]

@heresie

modifying all CI/CD pipelines across multiple projects and GitLab instances to include an unset DOCKER_AUTH_CONFIG command is not a viable solution on our side. It would impact too many teams and repositories, making the change both costly and error-prone.

That’s a fair point — I can’t reasonably estimate how much work this would be for others.

@gdegoulet

Nothing should have precedence when you run explicitly "docker login -u" !

Here, I respectfully disagree. In many Unix/Linux tools and workflows, environment variables are conventionally considered to have higher precedence than configuration files — precisely to enable runtime overrides without persistent changes.

I believe the confusion comes from differing interpretations of what docker login is supposed to represent:

• If we treat it as starting a session, then yes — the credentials provided should "win".
• But if it merely modifies ~/.docker/config.json, and subsequent commands might use different environment context, then environment variables should take precedence.

In the Unix philosophy, commands are typically stateless and composable. We're not starting sessions — we're executing independent processes. So from that perspective, it makes sense to me that an environment variable should override what's in the config file.