feat(worker): add taskworker.task.failed log for failed tasks

[s-starostin]

Adds a structured taskworker.task.failed log event when task execution fails in
run_worker.

For non-retriable failure paths, this replaces direct
sentry_sdk.capture_exception(err) calls with:

logger.exception("taskworker.task.failed", extra={...})

inside the existing isolation_scope blocks where they exist. This gives
operators a structured stdout log with the task identity and original exception
details. In environments where the Sentry SDK LoggingIntegration captures
error logs, the same call also produces the corresponding Sentry event.

The log includes:

• task_id
• taskname
• namespace
• processing_pool
• exception_type
• exception_message

Failure paths covered

• ProcessingDeadlineExceeded, gated by task_func.report_timeout_errors
• General exception fallback, gated by task_func.silenced_exceptions and retry state
• Retry-exhausted (unconditional, per fix(iswf): Reverts change to reporting logic that silences NoRetriesRemaining exceptions #627 - retry exhaustion is a terminal state worth surfacing even for silenced exception types)

On the retry-exhausted path, the structured log uses the original task exception
(err) and fires alongside the existing synthetic NoRetriesRemainingError
capture. This keeps retry-exhausted failures visible in worker logs in the same
shape as other task failures, while preserving the existing retry-exhaustion
Sentry grouping/fingerprint.

Existing behavior preserved

• task_func.report_timeout_errors opt-out continues to silence both the
  Sentry-reporting path and the new structured worker log on the timeout path
• task_func.silenced_exceptions opt-out continues to silence both surfaces on the general-exception path (non-fexhausted). Retry-exhausted is intentionally unconditional per fix(iswf): Reverts change to reporting logic that silences NoRetriesRemaining exceptions #627.
• NoRetriesRemainingError synthetic Sentry capture and fingerprint are unchanged
• Ambient Sentry scope on the general-exception fallback is unchanged; the call
  is not wrapped in a new isolation_scope

Note on retry-exhausted reporting

When Sentry SDK LoggingIntegration is configured to capture error-level logs
(the default), retry-exhausted failures will now produce two Sentry events:

1. the existing synthetic NoRetriesRemainingError event for retry-exhaustion
   alerting/grouping
2. an event for the original exception from taskworker.task.failed

These represent different exception objects and have distinct grouping behavior.
The synthetic event preserves the existing retry-exhaustion signal, while the
original-exception event/log makes the underlying failure visible to operators.

Refs #610

[markstory]

Why is this being introduced? This change was recently reverted in #627

[s-starostin]

I missed #627. Restored the unconditional capture and dropped the gate.

[markstory]

I don't think we need this test, the behavior for silenced errors shouldn't be changed.

[s-starostin]

Agreed, removed.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 3782285. Configure here.}

[cursor]

Retry-exhausted log not gated by silenced_exceptions

Medium Severity

On the retry-exhausted path, _log_task_failed is called unconditionally without checking should_capture_error. The PR description explicitly states this path is "gated by task_func.silenced_exceptions," but the code doesn't implement that gating. When a silenced exception exhausts retries, the original exception will now appear in structured logs (and potentially as a Sentry event via LoggingIntegration), defeating the purpose of silenced_exceptions on this path.

 

^{Reviewed by Cursor Bugbot for commit 3782285. Configure here.}

[s-starostin]

Bugbot is right that the PR description is stale, but I think the suggested code change is wrong. Retry-exhausted reporting is intentionally unconditional per #627, so I’ll update the PR description instead of gating _log_task_failed.