bug: nginx env variable for private key is not resolved in upstream with mtls

[cybersec-jochenarnold]

Current Behavior

Using mTLS for HTTPS Upstream with $env:// for Certificate and Private Key does not work. I think it does not resolve the variable "$env://API_UPSTREAM_KEY]". It tries to decrypt it instead.

I haven't been able to figure out where the variables for Upstream TLS are resolved in the code yet.

I also configured upstream tls client directly (not via extra SSL ID) - this works as expected. Maybe the resolving of the env vars is missing if the SSL config is loaded via get_by_id(cert_id)?

I will try to add some debug logs in the next days.

init.lua: handle_upstream()

    if api_ctx.matched_upstream and api_ctx.matched_upstream.tls and
        api_ctx.matched_upstream.tls.client_cert_id then

        local cert_id = api_ctx.matched_upstream.tls.client_cert_id
        local upstream_ssl = router.router_ssl.get_by_id(cert_id)
        if not upstream_ssl or upstream_ssl.type ~= "client" then
            local err  = upstream_ssl and
                "ssl type should be 'client'" or
                "ssl id [" .. cert_id .. "] not exits"
            core.log.error("failed to get ssl cert: ", err)

            if is_http then
                return core.response.exit(502)
            end

            return ngx_exit(1)
        end

        core.log.info("matched ssl: ",
                  core.json.delay_encode(upstream_ssl, true))
        api_ctx.upstream_ssl = upstream_ssl
    end

Expected Behavior

Using nginx env variable should be working for secrets like private key.

Error Logs

ssl.lua:162: aes_decrypt_pkey(): base64 decode ssl key failed. key[$env://API_UPSTREAM_KEY] ...

Steps to Reproduce

1. Run APISIX binary on Ubuntu
2. Create Service with Upstream and refer to a SSL ID
3. Create SSL with type=client

Used ADC but curl to Admin API does not work too.

services:
  - id: apigw-service
    name: apigw-service
    upstream:
      name: apigw-upstream
      type: roundrobin
      scheme: https
      pass_host: "node"
      nodes:
        - host: "example.com"
          port: 443
          weight: 1
      tls:
        client_cert_id: "UPSTREAM_CLIENT_CERT_ID"
    routes:
      - name: apigw-route
        plugins:
          proxy-rewrite:
            headers:
              set:
                apikey: "REDACTED"
        uris:
          - "/api/example-api"
        methods:
          - GET

ssls:
  - id: "UPSTREAM_CLIENT_CERT_ID"
    certificates:
      - certificate: "$env://API_UPSTREAM_CERT"
        key: "$env://API_UPSTREAM_KEY"
    snis:
      - "example.com"
    ssl_protocols:
      - TLSv1.2
      - TLSv1.3
    type: client

Environment

• APISIX version (run apisix version): 3.15.0
• Operating system (run uname -a): Ubuntu
• OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.27.1.2
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): -
• APISIX Dashboard version, if relevant: not used
• Plugin runner version, for issues related to plugin runners: -
• LuaRocks version, for installation issues (run luarocks --version): -

[Baoyuantop]

Hi @cybersec-jochenarnold, could you provide the configuration information for the service?

[cybersec-jochenarnold]

Hi @Baoyuantop ,

of course, I have added my service configuration.

[Baoyuantop]

The code at https://github.com/apache/apisix/blob/master/apisix/init.lua#L557 directly uses api_ctx.upstream_ssl = upstream_ssl, without calling secret.fetch_secrets to resolve environment variables and key references as in server-side SSL processing. This part of the logic needs to be improved by submitting code.

[cybersec-jochenarnold]

I'm happy to do that.

[Satish11012007]

Hi @Baoyuantop, @cybersec-jochenarnold

I’ve been looking into this issue and reviewing the relevant code in init.lua. It appears that when router_ssl.get_by_id() loads the upstream SSL object, it is directly assigned to api_ctx.upstream_ssl without going through secret.fetch_secrets().

Because of that, references such as $env://API_UPSTREAM_KEY are not resolved before key processing, which results in the base64 decode failure during aes_decrypt_pkey().

If this issue is not already being worked on, I would be glad to explore a potential improvement by aligning the upstream SSL handling with the existing server-side secret resolution flow.

Please let me know if it would be appropriate for me to proceed with a draft fix.

Thank you for your guidance.

[cybersec-jochenarnold]

Hi @Satish11012007 ,

I'm still unsure, as I would of course like to contribute something. However, I first need to set up the build development environment. But I would still like to try.

Edit: Setup of build development environment was successfully. I will provide the fix soon.

[cybersec-jochenarnold]

Fixes including test cases have been implemented. I will now deploy it in a test environment and (if successful) create the PR.