Skip to content

Add event receiver framework for webhook-driven graph updates #508

@ravisuhag

Description

@ravisuhag

Context

Not every metadata change needs a full extraction cycle. Many sources can push notifications — a GitHub webhook on a schema migration merge, a PagerDuty alert when an incident opens, a cloud event when a resource is created. Meteor should be able to receive these events and convert them into graph updates.

Scope

  • Lightweight HTTP server for receiving webhooks (meteor receive or built into meteor run)
  • Pluggable event handler interface — map incoming events to asset/relationship mutations
  • Built-in handlers for common sources:
    • GitHub webhooks (push, PR merge, release)
    • Generic cloud events (GCP Pub/Sub, AWS EventBridge)
  • Feed received events into the same processing pipeline as extracted records
  • Authentication and validation for incoming webhooks

Design Considerations

  • Event receivers complement extractors, they don't replace them. Full extraction provides the baseline; events provide real-time updates.
  • Should support running alongside batch extraction in the same Meteor process
  • Events should carry enough metadata to route to the right graph update

Why

Batch extraction on a 1-hour schedule means the graph can be up to 1 hour stale. Event receivers close this gap for sources that support push notifications, making the graph near-real-time for critical changes.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions