Random SSL error

[shawnchapla-8451]

We're using an OSS plugin for Kong that is using the openidc.lua module to provide relying party functionality. We have configured it to use Azure Redis as the session store for the cluster, and it's been working just fine until a few days ago. What appears to be happening is that, randomly a request with the session cookie to the plugin results in a redirect to the IdP rather than using the cookie to find a valid access token in Redis (resulting in a CORS error). When this happens, this message is written to the Kong error log:

2023/05/08 10:54:17 [alert] 2741596#0: *368 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt), client: <client IP>, server: kong, request: "GET <requested URL> HTTP/1.1", host: "<hostname>", referrer: "<referring URL>"

I've been able to trace it down to when the authenticate() function is called in openidc.lua by the plugin code, and further this error is generated somewhere down in resty.session after this statement is called.

As stated, this is happening randomly, because the same plugin on the same endpoint on the same Kong node can successfully process a request from the same client right before the failed one. And when running this test multiple times, it will be different request URLs that trigger the error.

I had to manually edit openidc.lua to add debug log statements to get this far, and looking at the resty.session code, I have no idea how far down the rabbit hole this goes. I'm looking for recommendations on figuring out what is the cause of the error.

[bungle]

@shawnchapla-8451, what version of lua-resty-session do you have? For that library it needs to be <= 3.10. Let's hope that openidc.lua is upgraded at some point to 4.0.3 of this library. But if I assume you have 3.10, is that correct?

[bungle]

On 3.10 the line is this probably this:
https://github.com/bungle/lua-resty-session/blob/v3.10/lib/resty/session/ciphers/aes.lua#L101

and then you find it here:
https://github.com/openresty/lua-resty-string/blob/master/lib/resty/aes.lua#L283-L285

On 4.0.3 I think it is this:
https://github.com/bungle/lua-resty-session/blob/v4.0.3/lib/resty/session/utils.lua#L713

lua-resty-openssl seems to use different APIs:
https://github.com/fffonion/lua-resty-openssl/blob/master/lib/resty/openssl/include/evp/cipher.lua

[shawnchapla-8451]

@shawnchapla-8451, what version of lua-resty-session do you have? For that library it needs to be <= 3.10. Let's hope that openidc.lua is upgraded at some point to 4.0.3 of this library. But if I assume you have 3.10, is that correct?

Our build process which creates our Kong images uses the command:

luarocks install lua-resty-session

The file /usr/local/share/lua/5.1/resty/session/ciphers/aes.lua matches what you've described for v3.10:

decrypted_data, err = cip:decrypt(data, tag)

And at the top of openidc.lua is this:

local openidc = {
  _VERSION = "1.7.6"
}

But if it's a version mismatch, wouldn't every call to resty.session.start() fail with the same error, not random ones?

[shawnchapla-8451]

What data is it attempting to decrypt? Is it what is stored in Redis?

[shawnchapla-8451]

Some additional context ... this plugin is being used to secure the content for a single page app behind Kong (there's a requirement that the end user has to authenticate before the SPA code is accessible by the browser). If the request for the app home page doesn't have the session cookie from resty.session, then the auth code flow is initiated to authenticate the user, and once that's completed, the plugin (actually, resty.session) stores their JWT in Redis and redirects them back to the home page with the session cookie.

What appears to be happening is when the browser processes the page with all its "child" elements (Javascript files, etc.), it sends the requests for all of those back to Kong with the cookie. But randomly, one or more of those requests results in a 302 back to the IdP. It looks like that's happening because of the decryption error above, which is interpreted by openidc.lua as there not being a valid session. Because the IdP redirect is not in the same origin, the result is a CORS error.

Update: it appears when the client is working with just one Kong node (vs. requests being load balanced across a cluster), the error doesn't appear to happen; at least I've not been able to reproduce it. Could this have something to do with timing such that the write to Redis from one cluster node hasn't completed (a lazy write?) before the next request at a different node tries to read/decrypt it?

[shawnchapla-8451]

Last update: I've tested switching the plugin to use cookie session mgmt rather than Redis, and am unable to replicate the issue. While not irrefutable evidence, it would seem to suggest Azure Redis in a clustered Kong env may play a role.