fix(client): log push-mode request deserializer failures#672
Closed
untitaker wants to merge 1 commit into
Closed
Conversation
In push mode the worker runs a gRPC server. When a `PushTask` request body fails to deserialize (e.g. proto mismatch or corruption), gRPC swallows the exception in `grpc._common._transform` — it logs only to the `grpc._common` logger and returns `None` — and the server then aborts the call with an opaque `INTERNAL "Exception deserializing request!"`. Unless the worker happens to route gRPC's internal loggers somewhere, the actual exception is invisible: no record on our own `taskbroker_client` logger, no metric, just an `INTERNAL` on the broker side. Wrap the deserializer body in the signature interceptor so the exception is logged on the `taskbroker_client.worker.client` logger before it is re-raised and swallowed by gRPC. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
Author
|
bogus actually. i'll just check the grpc logger setup in sentry. |
untitaker
added a commit
to getsentry/sentry
that referenced
this pull request
Jun 3, 2026
…6806) sentry's `LOGGING` uses `disable_existing_loggers: True` and has no `grpc` logger entry, so grpc's own logging is dropped or nondeterministically routed depending on import order. grpc catches a lot of exceptions internally (e.g. request deserialization failures in the push-mode taskworker) and logs them on the `grpc._common` logger, so we want those to be reliably visible. Configure `grpc` explicitly — `ERROR`, `console`, `propagate: False` — matching how other third-party loggers (`arroyo`, `taskbroker_client`, `urllib3`) are handled. Context: getsentry/taskbroker#672 (closed) — originally attempted to log these in the client before realizing the fix belongs here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In push mode the Python worker runs a gRPC server (
PushTaskWorker→WorkerServicer). When aPushTaskrequest body fails to deserialize — proto schema mismatch between broker and worker, or a corrupted payload — the underlying exception is silently swallowed and the broker only sees an opaqueINTERNALstatus. Nothing lands on our own logger and no metric is emitted, so the failure is effectively invisible in worker logs.This PR logs the exception on the
taskbroker_client.worker.clientlogger before re-raising.Where the exception is swallowed
The request deserializer (our
RequestSignatureServerInterceptorwraps it) is invoked by gRPC, and gRPC catches any exception it raises. Links below are pinned to thegrpcio==1.75.1tag (the version this repo locks).The swallow —
grpc/_common.py_transform(), lines 88–92:deserialize()calls this withexception_message="Exception deserializing message!".The opaque status —
grpc/_server.py_receive_message(), lines 333–343:So the real traceback exists only on the
grpc._commonlogger atERROR. Unless the worker explicitly routes gRPC's internal loggers to a handler, it is lost — and the broker just getsINTERNAL "Exception deserializing request!"with no detail about why.Fix
Wrap the deserializer body in the interceptor and log on our own logger before re-raising (gRPC still produces the
INTERNALstatus for the caller, but now there's a visible, attributable record on the worker side):Test
test_request_signature_server_interceptor_logs_deserialization_errorsstands up a realgrpc.serverwith the interceptor, sends a correctly-signed but unparseable body, and asserts:StatusCode.INTERNALtaskworker.grpc_server.request_deserialization_failedrecord is emitted on our loggerVerified the test fails without the fix (no such log record) and passes with it. A real server is required because the deserializer runs on gRPC's event-polling thread, not the servicer thread — a mocked channel cannot exercise this path.
🤖 Generated with Claude Code