fix(flask): wrap wsgi_app call in try/except to prevent active_requests gauge leak#4433
fix(flask): wrap wsgi_app call in try/except to prevent active_requests gauge leak#4433alliasgher wants to merge 3 commits intoopen-telemetry:mainfrom
Conversation
…ts gauge leak If wsgi_app() raises an uncaught exception, the active_requests_counter decrement at the end of _wrapped_app was never reached, causing the gauge to permanently read high. Kubernetes HPA and similar systems would see phantom load. Add a bare try/except that decrements the counter and re-raises on exception, matching the pattern already used in the WSGI instrumentation. Fixes open-telemetry#4431 Signed-off-by: alliasgher <alliasgher123@gmail.com>
|
Thanks for the quick fix! We hit this exact bug in production — the leaked gauge was preventing our Kubernetes HPA from scaling down. One suggestion: consider using The WSGI instrumentation already uses this pattern: # wsgi/__init__.py
try:
...
return _end_span_after_iterating(iterable, span, token)
except Exception as ex:
raise
finally:
self.active_requests_counter.add(-1, active_requests_count_attrs)For Flask it would look like: try:
result = wsgi_app(wrapped_app_environ, _start_response)
# ...duration recording unchanged...
return result
finally:
active_requests_counter.add(-1, active_requests_count_attrs)This also removes the original decrement at line 458, so there's exactly one decrement path. Also worth adding a test to prevent regression — I have one in #4437 ( |
Signed-off-by: alliasgher <alliasgher123@gmail.com>
|
Good catch — updated to use |
Signed-off-by: alliasgher <alliasgher123@gmail.com>
bb98692 to
5c2b07a
Compare
|
Added the regression test test_active_requests_counter_decremented_on_error. It hits /hello/500 (which raises ValueError internally), collects http.server.active_requests, and asserts the value is back to 0 after both requests complete. |
Fixes #4431. If
wsgi_app()raises, theactive_requests_counter.add(-1,...)at the end was never reached. Add a bare try/except that decrements and re-raises, matching the existing WSGI instrumentation pattern.