feat(table): live cell updates via SSE + per-table event buffer (#4508)

* feat(table): live cell updates via SSE + per-table event buffer

Replaces the polling-based row refetch with a push-based SSE stream that
patches the React Query cache directly as cell-state events arrive.

Architecture:
- New per-table event buffer in apps/sim/lib/table/events.ts. Redis sorted-set
  with monotonic eventId, 1h TTL, 5000-event cap, in-memory fallback. Modeled
  after apps/sim/lib/execution/event-buffer.ts but stripped of complexity
  tables don't need (no per-execution lifecycle, no id-batching, no write
  queue serialization). ~150 lines instead of 700.
- writeWorkflowGroupState appends a fat event after each successful 'wrote'.
  Status transitions carry executionId + jobId; terminal/partial transitions
  also include the new output values inline so the client can patch row data
  without a follow-up refetch.
- New SSE route at /api/table/[tableId]/events/stream?from=<lastEventId>.
  Replays from buffer on connect, polls at 500ms (mirrors workflow execution
  stream), heartbeat every 15s, signals 'pruned' if the caller fell off the
  back of the buffer.
- Client hook useTableEventStream subscribes via EventSource. Reconnect-resume
  with last-seen eventId. On 'pruned', invalidates the rows query and resumes
  from the new earliest. Cache patches walk every cached query under
  rowsRoot(tableId) so filter/sort variants all stay live.
- Removes refetchInterval from useTableRows and the per-page polling effect
  from useInfiniteTableRows. React Query's refetchOnWindowFocus +
  refetchOnReconnect cover the durability gap if any push is dropped.

Out of scope:
- Bulk-cancel events (cancellation path is being redesigned separately).
- Generalizing the workflow event-buffer module to a shared primitive (defer
  until a third use case appears; for now the table buffer is the simpler
  cousin of the workflow one).

* fix(table): drop run-mutation refetch so SSE patches aren't overwritten

useRunColumn.onSettled was canceling in-flight queries and invalidating the
rows query — leftover behavior from the polling era. With the SSE stream
now keeping the cache live via incremental patches, this refetch races the
stream and snaps the cache back to whatever DB shows at the refetch moment,
which can lag the just-arrived queued/running events. Cells appeared stuck
on the optimistic 'pending' even though the SSE was delivering the real
transitions.

* chore(table): simplify SSE plumbing — reuse helpers, drop dead polling code

- Reuse snapshotAndMutateRows for SSE cache patches instead of reimplementing
  the page-walk + cache-shape detection. Adds a {cancelInFlight: false} opt
  for the SSE caller (mutations still cancel as before).
- Drop client-side type duplication in use-table-event-stream — import
  TableEvent and TableEventEntry from lib/table/events directly.
- Drop the now-dead mergePagePreservingIdentity + rowEqual from tables.ts;
  their only caller was the polling effect that was removed earlier.
- Drop the defensive try/catch around appendTableEvent in cell-write — the
  function is documented as never-throwing (returns null on failure).
- Combine INCR + ZADD into one Lua eval in events.ts. Halves Redis RTT per
  cell-write. Lua returns the new eventId; the script splices it into the
  pre-built entry JSON.
- Trim refs to plain let bindings inside the effect; trim stale
  comments referencing the old polling implementation.

* fix(table): address PR review on SSE buffer

- TTL-expiry silent miss: when all keys expire, hgetall(meta) returns empty
  so earliestEventId is undefined and the prune branch was skipped. Reconnect
  with non-zero afterEventId now checks the seq counter — its absence (TTL
  expired) signals pruned so the client refetches. Memory fallback mirrors.
- Unbounded ZRANGEBYSCORE: cap reads at TABLE_EVENT_READ_CHUNK = 500 events
  per call. The route's 500ms poll loop drains chunks across ticks instead of
  flushing 5000 entries (multi-MB) in one tick after a long disconnect.
- Pruned handler closes EventSource client-side: server-side close was firing
  onerror and routing through the 500ms backoff path. Now we close
  proactively, reset the reconnect attempt counter, and reconnect immediately
  from the new earliest.

* improvement(table): persist SSE lastEventId in sessionStorage

Tab refresh / navigate-away-and-back now resume the stream from where the
previous mount left off instead of replaying from from=0. Mirrors the
useExecutionStream pattern (saveExecutionPointer / loadExecutionPointer).
First-ever mounts and new tabs still start at 0 — sessionStorage is
per-tab, so the safe default applies. On a 'pruned' fallback the new
earliestEventId is also persisted so the next reconnect starts there.

* fix(table): include runningBlockIds + blockErrors in SSE event payload

The cell renderer's 'queued' vs 'running' vs 'pending-upstream' decision
reads exec.runningBlockIds + exec.blockErrors. Without those fields the
inFlight branch falls through to 'pending-upstream' (amber Pending pill)
even when the worker has already written status=running. The worker writes
both fields to DB; the SSE event was stripping them. Thread them through
events.ts → cell-write.ts → use-table-event-stream.ts.

* fix(table): show value once column output has landed mid-run

The cell renderer treated any `status: 'running'` event as in-flight,
even when the column's own output had already been written. During a
multi-block group run, partial-write events for a later block carry
the earlier block's outputs but tag only the later block as running
— that flipped the finished column back to the amber Pending pill
until the terminal `completed` event arrived.

Re-order the priority chain so the column's value wins over
`pending-upstream`. Active re-run of the column itself
(`blockRunning`) still wins over the stale value, so a re-run on a
previously-completed cell still surfaces the running pill before the
new value overwrites.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(table): address PR review nits on tables.ts

- Merge duplicate JSDoc on snapshotAndMutateRows into a single block
- Remove unused useQueryClient() calls from useTableRows and
  useInfiniteTableRows (leftover from polling-era code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(table): apply biome formatting fixes

CI lint job flagged import order in two files and over-wrapped union
in lib/table/events.ts. No behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: TheodoreSpeaks

apps/sim/app/api/table/[tableId]/events/stream/route.ts

@@ -0,0 +1,161 @@
+import { createLogger } from '@sim/logger'
+import { toError } from '@sim/utils/errors'
+import { sleep } from '@sim/utils/helpers'
+import { type NextRequest, NextResponse } from 'next/server'
+import { tableEventStreamContract } from '@/lib/api/contracts/tables'
+import { parseRequest } from '@/lib/api/server'
+import { checkSessionOrInternalAuth } from '@/lib/auth/hybrid'
+import { generateRequestId } from '@/lib/core/utils/request'
+import { SSE_HEADERS } from '@/lib/core/utils/sse'
+import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
+import { readTableEventsSince, type TableEventEntry } from '@/lib/table/events'
+import { accessError, checkAccess } from '@/app/api/table/utils'
+
+const logger = createLogger('TableEventStreamAPI')
+
+const POLL_INTERVAL_MS = 500
+const HEARTBEAT_INTERVAL_MS = 15_000
+const MAX_STREAM_DURATION_MS = 4 * 60 * 60 * 1000 // 4 hours; client reconnects past this
+
+export const runtime = 'nodejs'
+export const dynamic = 'force-dynamic'
+
+interface RouteContext {
+  params: Promise<{ tableId: string }>
+}
+
+/** GET /api/table/[tableId]/events/stream?from=<lastEventId>
+ *
+ *  SSE stream of cell-state transitions. Replay-on-reconnect via `from`.
+ *  Pruning (buffer cap exceeded or TTL expired) sends a `pruned` event and
+ *  closes; the client responds with a full row-query refetch and reconnects
+ *  from the new earliest. */
+export const GET = withRouteHandler(async (req: NextRequest, context: RouteContext) => {
+  const requestId = generateRequestId()
+  const parsed = await parseRequest(tableEventStreamContract, req, context)
+  if (!parsed.success) return parsed.response
+  const { tableId } = parsed.data.params
+  const { from: fromEventId } = parsed.data.query
+
+  const auth = await checkSessionOrInternalAuth(req, { requireWorkflowId: false })
+  if (!auth.success || !auth.userId) {
+    return NextResponse.json({ error: 'Authentication required' }, { status: 401 })
+  }
+
+  const access = await checkAccess(tableId, auth.userId, 'read')
+  if (!access.ok) return accessError(access, requestId, tableId)
+
+  logger.info(`[${requestId}] Table event stream opened`, { tableId, fromEventId })
+
+  const encoder = new TextEncoder()
+  let closed = false
+
+  const stream = new ReadableStream<Uint8Array>({
+    async start(controller) {
+      let lastEventId = fromEventId
+      const deadline = Date.now() + MAX_STREAM_DURATION_MS
+      let nextHeartbeatAt = Date.now() + HEARTBEAT_INTERVAL_MS
+
+      const enqueue = (text: string) => {
+        if (closed) return
+        try {
+          controller.enqueue(encoder.encode(text))
+        } catch {
+          closed = true
+        }
+      }
+
+      const sendEvents = (events: TableEventEntry[]) => {
+        for (const entry of events) {
+          if (closed) return
+          enqueue(`data: ${JSON.stringify(entry)}\n\n`)
+          lastEventId = entry.eventId
+        }
+      }
+
+      const sendPrunedAndClose = (earliestEventId: number | undefined) => {
+        enqueue(
+          `event: pruned\ndata: ${JSON.stringify({ earliestEventId: earliestEventId ?? null })}\n\n`
+        )
+        if (!closed) {
+          closed = true
+          try {
+            controller.close()
+          } catch {}
+        }
+      }
+
+      const sendHeartbeat = () => {
+        // SSE comment line — keeps proxies (ALB default 60s idle) from closing
+        // the connection during quiet periods.
+        enqueue(`: ping ${Date.now()}\n\n`)
+      }
+
+      try {
+        // Initial replay from buffer.
+        const initial = await readTableEventsSince(tableId, lastEventId)
+        if (initial.status === 'pruned') {
+          sendPrunedAndClose(initial.earliestEventId)
+          return
+        }
+        if (initial.status === 'unavailable') {
+          throw new Error(`Table event buffer unavailable: ${initial.error}`)
+        }
+        sendEvents(initial.events)
+
+        // Stream loop — poll the buffer and forward new events. Workflow
+        // execution stream uses the same shape; pub/sub wakeups are an
+        // optimization we can add later if 500ms polling becomes a problem.
+        while (!closed && Date.now() < deadline) {
+          await sleep(POLL_INTERVAL_MS)
+          if (closed) return
+
+          const result = await readTableEventsSince(tableId, lastEventId)
+          if (result.status === 'pruned') {
+            sendPrunedAndClose(result.earliestEventId)
+            return
+          }
+          if (result.status === 'unavailable') {
+            throw new Error(`Table event buffer unavailable: ${result.error}`)
+          }
+          if (result.events.length > 0) {
+            sendEvents(result.events)
+          }
+
+          if (Date.now() >= nextHeartbeatAt) {
+            sendHeartbeat()
+            nextHeartbeatAt = Date.now() + HEARTBEAT_INTERVAL_MS
+          }
+        }
+
+        // Reached the defensive duration ceiling — close cleanly so the client
+        // reconnects with the latest lastEventId.
+        if (!closed) {
+          enqueue(`event: rotate\ndata: {}\n\n`)
+          closed = true
+          try {
+            controller.close()
+          } catch {}
+        }
+      } catch (error) {
+        logger.error(`[${requestId}] Table event stream error`, {
+          tableId,
+          error: toError(error).message,
+        })
+        if (!closed) {
+          try {
+            controller.error(error)
+          } catch {}
+        }
+      }
+    },
+    cancel() {
+      closed = true
+      logger.info(`[${requestId}] Client disconnected from table event stream`, { tableId })
+    },
+  })
+
+  return new NextResponse(stream, {
+    headers: { ...SSE_HEADERS, 'X-Table-Id': tableId },
+  })
+})

apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/cell-render.tsx

@@ -76,23 +76,27 @@ export function resolveCellRender({
 
     if (blockError) return { kind: 'block-error' }
 
-    // In-flight wins over the existing value: when the group is being re-run,
-    // the current value is about to be overwritten — surface the run state so
-    // the user sees the cell is changing. Without this, a queued / running
-    // re-run on a previously-completed cell looks like nothing happened until
-    // the new value lands.
+    // Active re-run of THIS column wins over its prior value — the value is
+    // about to be overwritten and the user should see the cell is changing.
     const inFlight =
       exec?.status === 'running' || exec?.status === 'queued' || exec?.status === 'pending'
+    if (inFlight && blockRunning) return { kind: 'running' }
+
+    // Value wins over `pending-upstream`: once this column's output has
+    // landed, the cell is done from the user's perspective — even if the
+    // group is still running other blocks downstream. Without this, mid-run
+    // partial-write events (`status: 'running'` carrying outputs but tagging
+    // a different block as running) would flip a finished column back to the
+    // amber Pending pill until the terminal `completed` event arrives.
+    if (!isNull) return { kind: 'value', text: stringifyValue(value) }
+
     if (inFlight && !(groupHasBlockErrors && !blockRunning)) {
-      if (blockRunning) return { kind: 'running' }
       if (exec?.status === 'queued' || exec?.status === 'pending') return { kind: 'queued' }
-      // `running` with this block not in `runningBlockIds` = upstream block
-      // still going; surface as the amber Pending pill per logs convention.
+      // `running` with this block not in `runningBlockIds` and no value yet =
+      // upstream block still going; surface as the amber Pending pill.
       return { kind: 'pending-upstream' }
     }
 
-    if (!isNull) return { kind: 'value', text: stringifyValue(value) }
-
     // Waiting wins over a stale terminal state: if deps are unmet right now,
     // the prior `cancelled` / `error` is informational at best — the cell
     // can't actually run until the user fills the missing input. Surface the

apps/sim/app/workspace/[workspaceId]/tables/[tableId]/hooks/index.ts

@@ -1,2 +1,3 @@
 export * from './use-context-menu'
 export * from './use-table'
+export * from './use-table-event-stream'