Unable to connect to WSGI daemon process

[Srushtipatilp9]

Hi,
I am currently facing an issue with the WSGI daemon process connection. The WSGI file created on the server and all required Python packages being installed are in the attachment for reference. But the application is not functioning as expected and is throwing 50X errors frequently.

python packages.txt

wsgi.txt

Please find the error logs below for reference:

`2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636700 2025] [wsgi:error] [pid 25398:tid 26174] self._conn.request("POST", "v0.7/config", payload, self._headers)	[Tue Mar 25 03:40:06.636700 2025] [wsgi:error] [pid 25398:tid 26174] self._conn.request("POST", "v0.7/config", payload, self._headers)
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636708 2025] [wsgi:error] [pid 25398:tid 26174] File "/usr/lib64/python3.7/http/client.py", line 1281, in request
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636715 2025] [wsgi:error] [pid 25398:tid 26174] self._send_request(method, url, body, headers, encode_chunked)
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636723 2025] [wsgi:error] [pid 25398:tid 26174] File "/usr/lib64/python3.7/http/client.py", line 1327, in _send_request
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636729 2025] [wsgi:error] [pid 25398:tid 26174] self.endheaders(body, encode_chunked=encode_chunked)
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636735 2025] [wsgi:error] [pid 25398:tid 26174] File "/usr/lib64/python3.7/http/client.py", line 1276, in endheaders
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636741 2025] [wsgi:error] [pid 25398:tid 26174] self._send_output(message_body, encode_chunked=encode_chunked)
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636747 2025] [wsgi:error] [pid 25398:tid 26174] File "/usr/lib64/python3.7/http/client.py", line 1036, in _send_output
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636754 2025] [wsgi:error] [pid 25398:tid 26174] self.send(msg)
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636760 2025] [wsgi:error] [pid 25398:tid 26174] File "/usr/lib64/python3.7/http/client.py", line 976, in send
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636766 2025] [wsgi:error] [pid 25398:tid 26174] self.connect()
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636771 2025] [wsgi:error] [pid 25398:tid 26174] File "/usr/lib64/python3.7/http/client.py", line 948, in connect
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636778 2025] [wsgi:error] [pid 25398:tid 26174] (self.host,self.port), self.timeout, self.source_address)
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636783 2025] [wsgi:error] [pid 25398:tid 26174] File "/usr/lib64/python3.7/socket.py", line 728, in create_connection
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636790 2025] [wsgi:error] [pid 25398:tid 26174] raise err
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636796 2025] [wsgi:error] [pid 25398:tid 26174] File "/usr/lib64/python3.7/socket.py", line 716, in create_connection
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636802 2025] [wsgi:error] [pid 25398:tid 26174] sock.connect(sa)
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.636808 2025] [wsgi:error] [pid 25398:tid 26174] ConnectionRefusedError: [Errno 111] Connection refused
	2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.014205 2025] [wsgi:error] [pid 31924:tid 31924] (2)No such file or directory: [client 10.116.0.150:61886] mod_wsgi (pid=31924): Unable to connect to WSGI daemon process 'revo6004dompe' on '/run/httpd/wsgi.22725.0.5.sock' as user with uid=48.`

Could you please help investigate the issue and provide guidance on resolving it?
Please help to verify the wsgi content as well created on the server.

Let me know if you need any additional details from my end.

Thanks,
Srushti Patil

[GrahamDumpleton]

The only error in that log output which is specifically related to mod_wsgi is:

2025-03-25T03:40:06.000Z[Tue Mar 25 03:40:06.014205 2025] [wsgi:error] [pid 31924:tid 31924] (2)No such file or directory: [client 10.116.0.150:61886] mod_wsgi (pid=31924): Unable to connect to WSGI daemon process 'revo6004dompe' on '/run/httpd/wsgi.22725.0.5.sock' as user with uid=48.`

This is occurring because something has triggered Apache to do a reload of the configuration and you have keep alive connections enabled.

If this occurs the same time every day it is likely due to the operating system triggering a reload in order to cause log file rotation for Apache.

That linux handles log file rotation this way is a bit of a pain because Apache has its own log file rotation mechanism but linux distributions choose to ignore it and force use of their own less than optimal solution which forces a reload of Apache, thus causing potential interruptions to Apache service.

That said, you can see that message on any Apache reload when you update configuration and trigger Apache to reread configuration. So is not specifically related to log file rotation but log file rotation is when most people see it as Apache otherwise isn't usually being reloaded.

If the code base for your web application is pretty stable and mod_wsgi configuration is not changing, you can avoid that specific error by setting:

WSGISocketRotation Off

This stops mod_wsgi from using a different socket path for communicating to daemon processes after a configuration reload meaning connections to the daemon process from Apache workers which are still running during a reload, but where a keep alive connection is still active, will be able to connect to the new instance of the daemon process since will use the same socket path.

As to all the other errors I have no idea as mod_wsgi doesn't use pure Python code to do anything, so those tracebacks are coming from your own application code, so you would need to investigate what your code is trying to connect to.

[Srushtipatilp9]

Thank you @GrahamDumpleton for your response.
WSGISocketRotation Off will it have any impact on the stale data stored?

also can we have below parameters in our wsgi file code in
"WSGIDaemonProcess abcd user=ec2-user threads=5 python-home=/var/venv/dd/ddabc request-timeout=60"

• keepAliveTimeout=60
• socket-timeout=120
• queue-timeout=30
• inactivity-timeout=300

what will be the impact of them on stale data or server?

[GrahamDumpleton]

Turning off socket rotation should in general be okay.

KeepAliveTimeout of 60 seconds is excessively high. Although a long time ago KeepAliveTimeout used to default to 60 seconds, the default for KeepAliveTimeout these days is 5 seconds. I would not run with a value of 60 seconds unless you had very good reasons to and understand the implications. In mod_wsgi-express it uses 2 seconds.

The socket-timeout for daemon processes usually defaults to value of Apache Timeout directive, which is 60 seconds. Even 60 seconds is too high in my mind. In mod_wsgi-express it uses 45 seconds.

I suggest you watch:

• https://www.youtube.com/watch?v=H6Q3l11fjU0

and learn a bit from defaults used by mod_wsgi-express talked about in it. Source code where mod_wsgi-express defaults can be seen is at:

• https://github.com/GrahamDumpleton/mod_wsgi/blob/develop/src/server/__init__.py#L1963

You might also find this talk interesting if you want to start playing with configuration.

• https://www.youtube.com/watch?v=SGleKfigMsk

I don't really know what you mean by "stale data or server".

[Srushtipatilp9]

thank you @GrahamDumpleton for your response.

The term "stale data" in this context refers to a stale WSGI socket. If the wsgi-socket-rotation setting is turned off, it means that the WSGI server will not rotate or regenerate the socket periodically.

I would like to understand whether disabling socket rotation could lead to potential long-term issues on the server, such as connection failures, resource exhaustion, or degraded performance over time.

Could you please provide insights into the possible impact of keeping wsgi-socket-rotation disabled and whether it is recommended to enable it for optimal server health?

[GrahamDumpleton]

The original reason for socket rotation was so that when the Apache configuration was changed, that you didn't have a situation where a request received by worker process based on old configuration didn't go to a new daemon process. This was done thinking that security wise it was best thing to do.

In practice this was being overly cautious in the design.

If your configuration and application code is stable there would be no risk in not changing the socket name.

So the reasons are nothing to do with performance and was just being overly cautious on security. I know of no detrimental effects from disabling socket rotation.

[Srushtipatilp9]

Hi @GrahamDumpleton,

1. Without WSGISocketRotation Off :
   We have also observed that the issue of being unable to connect to the WSGI daemon process is occurring consistently at the same time each day when socket rotation is not disabled. Upon analyzing CPU and memory utilization during the affected time frame, no anomalies were observed—resource usage remained within normal thresholds.
   Below are the logs captured during the same time frame—please refer to the timestamps for verification.

`[Tue May 20 03:44:05.371783 2025] [wsgi:error] [pid 22002:tid 22002] (2)No such file or directory: [client 1.1.1.1:15826] mod_wsgi (pid=22002): Unable to connect to WSGI daemon process 'revo6004abc' on '/run/httpd/wsgi.28402.19.24.sock' as user with uid=48.

[Mon May 19 03:42:02.136960 2025] [wsgi:error] [pid 4311:tid 4311] (2)No such file or directory: [client 1.1.1.2:36056] mod_wsgi (pid=4311): Unable to connect to WSGI daemon process 'revo6004bcd on '/run/httpd/wsgi.12252.18.18.sock' as user with uid=48.

[Thu May 15 03:43:05.301843 2025] [wsgi:error] [pid 32695:tid 32695] (2)No such file or directory: [client 1.1.1.3:64348] mod_wsgi (pid=32695): Unable to connect to WSGI daemon process 'revo6004def' on '/run/httpd/wsgi.12252.14.8.sock' as user with uid=48.

[Wed May 14 03:20:03.811221 2025] [wsgi:error] [pid 12677:tid 12677] (2)No such file or directory: [client 1.1.1.4:59204] mod_wsgi (pid=12677): Unable to connect to WSGI daemon process 'revo6004efg' on '/run/httpd/wsgi.12252.13.19.sock' as user with uid=48.`

Is this related any sort of WSGI daemon/socket lifecycle or cleanup issue?

2. With WSGISocketRotation Off:
   We are observing continuous error "HTTP error status 500, reason Internal Server Error", can we please get some insights on what can be the cause for this error?
   Our application is not impacted with the error seen, but the loggers for internal server error are causing alerts for 5XX.

[GrahamDumpleton]

@Srushtipatilp9 As explained above, (1) is due to system forced log rotation. It should be transient and go away, although may appear worse if you have a long keep alive timeout.

As to (2), I cannot know unless see the log errors from Apache error logs, especially anything from Python application code.

[Srushtipatilp9]

Hi @GrahamDumpleton
Thank you for your guidance.
We have implemented the following configuration changes on our application for more than a month in an effort to resolve the recurring 503 errors, but the issue still persists:

1. Disabled WSGISocketRotation to prevent potential disruption from socket reallocation.
2. Increased KeepAliveTimeout to allow for longer idle connections and again disabled WSGISocketRotation to rule out socket rotation as a cause.

Despite these changes, we are still consistently encountering 503 Service Unavailable errors at approximately around 3:37 AM UTC on a daily basis. At this time, the application becomes unresponsive and enters a 503 state, indicating a potential underlying issue unrelated to the attempted wsgi configuration updates.

We suspect there might be a scheduled process or resource constraint triggering this behavior, and would appreciate any further guidance or suggestions for investigation.

Please find the logs below:

`[Mon Jul 14 03:37:03.921796 2025] [wsgi:error] [pid 29973:tid 29973] (2)No such file or directory: [client 10.10.0.0:39936] mod_wsgi (pid=29973): Unable to connect to WSGI daemon process 'revo6004abcde' on '/run/httpd/wsgi.12374.u1000.22.sock' as user with uid=48., referer: https://abc.com/abcede-6004/

[Mon Jul 14 03:44:05.371783 2025] [wsgi:error] [pid 22002:tid 22002] (2)No such file or directory: [client 1.1.1.1:15826] mod_wsgi (pid=22002): Unable to connect to WSGI daemon process 'revo6004abc' on '/run/httpd/wsgi.28402.19.24.sock' as user with uid=48.

[Mon Jul 14 03:42:02.136960 2025] [wsgi:error] [pid 4311:tid 4311] (2)No such file or directory: [client 1.1.1.2:36056] mod_wsgi (pid=4311): Unable to connect to WSGI daemon process 'revo6004bcd on '/run/httpd/wsgi.12252.18.18.sock' as user with uid=48.

[Mon Jul 14 03:43:05.301843 2025] [wsgi:error] [pid 32695:tid 32695] (2)No such file or directory: [client 1.1.1.3:64348] mod_wsgi (pid=32695): Unable to connect to WSGI daemon process 'revo6004def' on '/run/httpd/wsgi.12252.14.8.sock' as user with uid=48.

[Mon Jul 14 03:20:03.811221 2025] [wsgi:error] [pid 12677:tid 12677] (2)No such file or directory: [client 1.1.1.4:59204] mod_wsgi (pid=12677): Unable to connect to WSGI daemon process 'revo6004efg' on '/run/httpd/wsgi.12252.13.19.sock' as user with uid=48.``

[GrahamDumpleton]

It doesn't look like you have WSGISocketRotation defined properly in a config file that is being read. The code which calculates the path is:

        /*
         * Calculate path for socket to accept requests on and
         * create the socket.
         */

        entry->socket_rotation = wsgi_server_config->socket_rotation;

        if (entry->socket_rotation) {
            entry->socket_path = apr_psprintf(p, "%s.%d.%d.%d.sock",
                                         wsgi_server_config->socket_prefix,
                                         getpid(), mpm_generation, entry->id);
        }
        else {
            entry->socket_path = apr_psprintf(p, "%s.%d.u%d.%d.sock",
                                         wsgi_server_config->socket_prefix,
                                         getpid(), entry->uid, entry->id);
        }

The path the logs is outputting has:

/run/httpd/wsgi.12252.13.19.sock

Note how there is no u prefix before the second number. This indicates (based on looking at code), that socket rotation is still enabled because otherwise that part of the path should be a u followed by the user ID.

Check your Apache configuration files to make sure WSGISocketRotation Off is in a file that is being read. Change it to WSGISocketRotation XXX which should trigger an error since value is not either On or Off. If you don't get an error, it isn't being read.

[Srushtipatilp9]

@GrahamDumpleton We tried suggested WSGISocketRotation XXX.

Please find the logs below:

Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xeu httpd.service" for details.
[ec2-user@ip-1-1-1-134 conf.d]$ systemctl status httpd.service
× httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; preset: disabled)
Drop-In: /etc/systemd/system/httpd.service.d
└─override.conf
Active: failed (Result: exit-code) since Tue 2025-07-15 06:12:52 UTC; 52s ago
Duration: 14h 4min 7.981s
Docs: man:httpd.service(8)
Process: 239839 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
Main PID: 239839 (code=exited, status=1/FAILURE)
Status: "Reading configuration..."
CPU: 66ms

Jul 15 06:12:52 ip-1-1-1-134.ec2.internal systemd[1]: Starting httpd.service - The Apache HTTP Server...
Jul 15 06:12:52 ip-1-1-1-134.ec2.internal httpd[239839]: AH00526: Syntax error on line 5 of /etc/httpd/conf.d/rdh.conf:
Jul 15 06:12:52 ip-1-1-1-134.ec2.internal httpd[239839]: WSGISocketRotation must be one of: Off | On
Jul 15 06:12:52 ip-1-1-1-134.ec2.internal systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Jul 15 06:12:52 ip-1-1-1-134.ec2.internal systemd[1]: httpd.service: Failed with result 'exit-code'.
'Jul 15 06:12:53 ip-1-1-1-134.ec2.internal systemd[1]: Failed to start httpd.service - The Apache HTTP Server.`

[GrahamDumpleton]

When you change to WSGISocketRotation Off, did you do a complete Apache stop and start, or did you only do a reload/restart?

If you only did the latter, I can imagine you might see the wrong path in a transition period as old processes cycled out, but eventually it should switch to different path naming as mentioned.

What do you get if run:

ls -las /run/httpd/wsgi.*.sock

Are there any files in there with that u prefix I mention.

[Srushtipatilp9]

@GrahamDumpleton Please find the details below

ec2-user@ip-1-1-1-134 httpd]$ ls -las /run/httpd/wsgi.*.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.1.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.10.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.11.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.12.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.2.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.3.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.4.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.5.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.6.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.7.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.8.sock
0 srwx------. 1 apache root 0 Jul 15 06:17 /run/httpd/wsgi.240989.u1000.9.sock

[GrahamDumpleton]

Which means it should be working now. Those last errors were probably just from the change over period. Monitor it for a few days and see if everything is okay now.

[Srushtipatilp9]

Hi @GrahamDumpleton ,
Thank you for the inputs.
We are using blue-green deployments, and following each deployment, new servers are provisioned with WSGISocketRotation Off configured.

However, despite these being fresh server setups, we are still encountering the same errors.

Could you please help us in this?

[GrahamDumpleton]

Where are you putting the WSGISocketRotation in your configuration files?

It is quite possible that it must be specified before any WSGIDaemonProcess directive is encountered while parsing the Apache configuration.

The best place if using Linux systems which use it, is to put it in the wsgi.load file immediately after the LoadModule line which loads mod_wsgi.