Skip to content

feat: add exponential backoff retry to MQ consumer on receive errors#775

Merged
alexluong merged 1 commit intomainfrom
consumer-retry
Mar 23, 2026
Merged

feat: add exponential backoff retry to MQ consumer on receive errors#775
alexluong merged 1 commit intomainfrom
consumer-retry

Conversation

@alexluong
Copy link
Collaborator

@alexluong alexluong commented Mar 20, 2026

Summary

  • Adds receiveWithRetry() to the generic MQ consumer with exponential backoff (200ms → 5s cap, max 5 consecutive errors)
  • Transient Receive() errors (e.g. GCP oauth2 token refresh timeouts, network blips) now self-heal instead of killing the worker
  • Persistent errors still fail after exhausting retries — same end behavior, just slightly delayed
  • Pattern matches existing scheduler retry logic in scheduler.go

Test plan

  • Existing consumer tests pass (single handler, concurrent handler)
  • New test: transient errors retried through successfully
  • New test: persistent errors exhaust retry budget and fail

🤖 Generated with Claude Code

Previously, any error from subscription.Receive() immediately killed the
consumer worker with no recovery. Transient errors like GCP oauth2 token
refresh timeouts would take down the entire worker until a manual restart.

Add receiveWithRetry() with exponential backoff (200ms -> 5s, max 5
consecutive errors) matching the existing scheduler retry pattern. Transient
errors now self-heal; persistent errors still fail after the retry budget.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@vercel
Copy link

vercel bot commented Mar 20, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
outpost-docs Ready Ready Preview, Comment Mar 20, 2026 5:37pm
outpost-website Ready Ready Preview, Comment Mar 20, 2026 5:37pm

Request Review

@alexluong alexluong merged commit 135ccdc into main Mar 23, 2026
5 checks passed
@alexluong alexluong deleted the consumer-retry branch March 23, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants