Skip to content

fix(linear): orchestration reconciler loses terminal events during outages (no stranded-orchestration backstop) #303

@isadeks

Description

@isadeks

Component

API or orchestration

Describe the bug / gap

The #247 Mode A orchestration reconciler reacts to TaskTable-stream terminal-state events to release dependency-unblocked children. If the reconciler is unavailable when a child reaches terminal state (deploy window, throttle, a crash like the 256MB OOM found during the first dev smoke), that stream event is lost and never reprocessed — the dependent children are never released and the orchestration stalls forever with no recovery.

Observed live on dev (2026-06-09): child A reached COMPLETED during a reconciler OOM window; after the fix deployed, A's completion event was gone, so B stayed blocked until the record was manually nudged to re-emit a stream event.

Proposed solution

Add a scheduled stranded-orchestration reconciler (mirrors the existing reconcile-stranded-tasks pattern): a periodic sweep that scans OrchestrationTable for orchestrations with children in non-terminal states whose predecessors are already terminal-success, and releases them (idempotent — releaseChild is already idempotency-keyed). Also surfaces orchestrations stuck with a released child whose task is long-terminal but whose row never advanced.

Acceptance criteria

  • A child completing during a reconciler outage is eventually released (bounded by the sweep interval).
  • Sweep is idempotent — no duplicate child tasks, no regressed statuses.
  • Stuck/stranded orchestrations are logged/metriced for operator visibility.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    adaptersThird-party integrations: Linear, Slack, GitHub App, notification/deploy providersbugSomething isn't workingorchestrationTask lifecycle, REST API handlers, orchestrator Lambdas, durable execution

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions