chore: remove delivery event concept #653
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
- Rename interface methods: InsertManyDeliveryEvent -> InsertMany, ListDeliveryEvent -> ListDelivery, RetrieveDeliveryEvent -> RetrieveDelivery - Rename request types: ListDeliveryEventRequest -> ListDeliveryRequest, etc. - Add DeliveryRecord type for query results with Event and Delivery - Update memlogstore, pglogstore, chlogstore implementations - Update all API handlers and tests to use new interface - Remove DeliveryEventID field from Delivery struct Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add DeliveryTask struct with IdempotencyKey() and RetryID() methods - Update deliverymq to publish/consume DeliveryTask instead of DeliveryEvent - Update publishmq to create and enqueue DeliveryTask - Update RetryTask to convert to DeliveryTask - Update API handlers, eventtracer, alert, emetrics to use DeliveryTask - Fix Delivery fields (TenantID, Attempt, Manual) not being set before logging - Add :manual suffix to idempotency key for manual retries Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update chlogstore/README.md method names and SQL examples - Update pglogstore/README.md method names - Update tracer_test.go comment to reference DeliveryTask Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Preserves Event-Delivery pairing through the insert flow, eliminating the need for eventMap reconstruction in ClickHouse implementation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
alexluong
force-pushed
the
model-delivery-event
branch
from
04c3ebd to
9277a3f
Compare
* refactor: add event fetching to retry scheduler Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: align mock eventGetter with logstore behavior Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: remove logStore from messagehandler Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: remove dead eventGetter code from messagehandler tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: verify manual retry publishes full event data Extends TestRetryDelivery to verify that the manual retry API publishes a DeliveryTask with complete event data to deliveryMQ. This ensures the deliverymq handler receives full event data for manual retries, consistent with the scheduled retry flow which fetches event data in the retry scheduler. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add failing tests for retry race condition Add unit and e2e tests verifying that retries are not lost when the retry scheduler queries logstore before the event has been persisted. Test scenario: 1. Initial delivery fails, retry is scheduled 2. Retry scheduler queries logstore for event data 3. Event is not yet persisted (logmq batching delay) 4. Retry should remain in queue and be reprocessed later Tests added: - TestRetryScheduler_RaceCondition_EventNotYetPersisted (unit test) - TestE2E_Regression_RetryRaceCondition (e2e test) Also adds: - RetryVisibilityTimeoutSeconds config option - WithRetryVisibilityTimeout scheduler option - mockDelayedEventGetter for simulating delayed persistence Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: return error when event not found in logstore during retry Return error instead of nil so the retry message stays in queue and will be reprocessed after the visibility timeout. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: improve flaky tests * chore: dev yaml * chore: `make test` skip cmd/e2e by default --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Update test files to use the new Attempt naming: - logstore/drivertest/*.go: AttemptFactory, ListAttempt, RetrieveAttempt - deliverymq/*_test.go: AttemptStatus*, entry.Attempt - apirouter/*_test.go: AttemptFactory, /attempts API paths - logmq/batchprocessor_test.go: AttemptFactory, Attempt struct fields All unit tests pass (1665 tests). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Revert attempt_metadata → delivery_metadata in OpenAPI (matches Go code) - Fix config: delivery_prefix → attempt_prefix, example att → atm - Fix variable names: deliveryID → attemptID, attErr → atmErr - Fix test names: TestListDeliveries → TestListAttempts, etc. - Update doc links for renamed API endpoints - Update comments and test descriptions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove comments that simply restate what the next line of code does, such as "// Create client" before NewClient() or "// Check if exists" before .Exists(). Preserves all meaningful comments including godoc, section headers, WHY explanations, and unit clarifications. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
refactor: rename delivery -> attempt
There was a problem hiding this comment.
Attempts belong to a destination, unclear to me why we removed /tenants/:tenant_id,/destinations/:dest_id/attempts
There was a problem hiding this comment.
I was thinking that this is the same as /tenants/:tenant_id/attempts?destination_id=dest_id, so it's effectively syntactic sugar. I thought it was simpler to have only 1 way of doing things.
Not an issue to add this back, so let me know which you prefer.
There was a problem hiding this comment.
I think from a REST principle attempts belong to a destination and so have a explicit /tenants/:tenant_id,/destinations/:dest_id/attempts is more idiomatic? 🤔
There was a problem hiding this comment.
added
GET /:tenantID/destinations/:destinationID/attempts
GET /:tenantID/destinations/:destinationID/attempts/:attemptID
POST /:tenantID/destinations/:destinationID/attempts/:attemptID/retry
internal/models/event.go
Outdated
| TenantID string `json:"tenant_id"` | ||
| EventID string `json:"event_id"` | ||
| DestinationID string `json:"destination_id"` | ||
| AttemptNumber int `json:"attempt"` |
There was a problem hiding this comment.
That seems a bit odd label, what about just Number?
There was a problem hiding this comment.
will also include a migration to change field attempt -> number, right?
FWIW Hookdeck also calls it attempt_number (ref)
There was a problem hiding this comment.
The json in the line commented on is just attempt though. I could also go with attempt_number I just think attempt is vague
There was a problem hiding this comment.
yes, I’ll change to “attempt_number” to be consistent with Hookdeck API
“number” is a tad vague
There was a problem hiding this comment.
i'm working on this attempt_number thing and i noticed some inconsistency & things to discuss. Updated the naming in this PR and created this issue #662 here
| const [retrying, setRetrying] = useState<boolean>(false); | ||
|
|
||
| const retryDelivery = useCallback( | ||
| const retryAttempt = useCallback( |
There was a problem hiding this comment.
I'm hesitant on that terminology, you aren't really retrying an attempt, you retry a delivery with generates and attempt. Thoughts?
There was a problem hiding this comment.
i changed this on the FE but I assume your feedback is more general. My concern here is how would API for 'retry delivery' work since we don't expose the concept of delivery?
so should we do 'retry event to a destination' as a requirement?
# instead of
POST /tenants/:tenant_id/attempts/:attempt_id/retry
# do something along this line
POST /tenants/:tenant_id/destinations/:destination_id/events/:event_id/retry
#or
POST /tenants/:tenant_id/events/:event_id/retry?destination_id=...
it just feels a bit awkward I think
| { label: "Overview", path: "" }, | ||
| { label: "Settings", path: "/settings" }, | ||
| { label: "Deliveries", path: "/deliveries" }, | ||
| { label: "Attempts", path: "/attempts" }, |
There was a problem hiding this comment.
I don't think Attempts is the right term for the UI, end-users don't think in terms of Attempts. I would call it "Event deliveries" which shows a list of delivery attemtps. This is what Stripe does
There was a problem hiding this comment.
I changed to "Event Deliveries" but kept the path /attempts, or do you think /deliveries make more sense?
There was a problem hiding this comment.
For the front end yes
Remove DeliveryEvent concept
This PR replaces the monolithic
DeliveryEventtype with purpose-specific types that better represent the different stages of event delivery.New Types
DeliveryTask- Message type for delivery processing queues. Flows frompublishmq→deliverymqand from retry →deliverymq. Contains theEvent,DestinationID,Attempt, andManualflag. ProvidesIdempotencyKey()for deduplication.RetryTask- Message type for scheduling retries. Contains only the metadata needed to re-fetch the event and re-attempt delivery (EventID,TenantID,DestinationID,Attempt). Converts toDeliveryTaskwhen the retry fires.LogEntry- Message type for the log queue. Contains bothEventandDeliveryfor persistence to the logstore.DeliveryRecord- Query result type returned by logstore. ContainsDeliverywith optionalEventpopulation for API responses.Logstore Interface Changes
InsertManyDeliveryEvent→InsertMany(events, deliveries)ListDeliveryEvent→ListDelivery(returnsDeliveryRecord)RetrieveDeliveryEvent→RetrieveDelivery(returnsDeliveryRecord)Other Changes
delivery_event_idcolumn from database schema/delivery-eventsAPI endpointsevent_id:destination_idformat (with:manualsuffix for manual retries)