feat(workflow-executor): workflow executor package (rebased)

[matthv]

Summary

Rebased version of #1493 onto current main (was 76 commits behind).

• Complete @forestadmin/workflow-executor package: step executors, HTTP server, RunStore (in-memory + Sequelize), adapters, graceful shutdown
• AI-powered step execution: condition, read-record, update-record, trigger-action, load-related-record, MCP
• buildDatabaseExecutor / buildInMemoryExecutor factory functions
• Example setup with Docker + PostgreSQL
• 427 tests passing

Replaces #1493 (protected branch prevented force-push after rebase).

Test plan

• yarn workspace @forestadmin/workflow-executor test — 427 tests pass
• yarn workspace @forestadmin/workflow-executor lint — clean
• yarn workspace @forestadmin/ai-proxy build — compiles after rebase conflict resolution

🤖 Generated with Claude Code

Note

Add workflow-executor package with AI-driven step executors and HTTP server

• Introduces a new @forestadmin/workflow-executor package that polls for pending workflow steps, executes them via AI-backed executors, and exposes a Koa HTTP server for run access and triggering.
• Adds step executor implementations for condition, read-record, update-record, trigger-action, load-related-record, and mcp step types, each supporting automatic execution or user-confirmation flows with persistence via RunStore.
• Adds InMemoryStore and DatabaseStore (Sequelize + Umzug migrations) run store backends, exposed through buildInMemoryExecutor and buildDatabaseExecutor factory functions.
• Refactors ai-proxy to use a ToolProvider interface, adding ForestIntegrationClient for Zendesk tools, BraveToolProvider, and a createToolProviders factory that replaces the previous MCP-only mcpConfigs with a unified toolConfigs map.
• Extends the chart rendering pipeline (renderChart on datasource, collection, and decorators) to accept and forward an optional parameters record, parsed from request query/body by a new QueryStringParser.parseChartParameters method.
• Risk: mcpConfigs is renamed to toolConfigs across route args and public interfaces; consumers passing mcpConfigs directly will break.

📊 Macroscope summarized 2ea7e95. 55 files reviewed, 10 issues evaluated, 1 issue filtered, 2 comments posted

🗂️ Filtered Issues

packages/workflow-executor/src/adapters/agent-client-agent-port.ts — 0 comments posted, 2 evaluated, 1 filtered

• line 98: The condition limit !== null on line 98 does not handle undefined. If limit is omitted from the query (making it undefined), undefined !== null evaluates to true, causing { pagination: { size: undefined, number: 1 } } to be passed to the API. The check should use loose equality limit != null to correctly handle both null and undefined. [ Failed validation ]

[linear]

PRD-214 Setup du package workflow-executor dans agent-nodejs

[qltysh]

16 new issues

Tool	Category	Rule	Count
qlty	Structure	Function with high complexity (count = 14): createWorkflowExecutor	7
qlty	Structure	Function with many returns (count = 4): getAiConfiguration	5
qlty	Structure	Deeply nested control flow (level = 4)	2
qlty	Duplication	Found 16 lines of similar code in 2 locations (mass = 92)	2

[macroscopeapp]

🟡 Medium adapters/forest-server-workflow-port.ts:14

When runId contains URL-reserved characters (like /, ?, or #), ROUTES.updateStepExecution produces a malformed path because the parameter is not encoded. Unlike ROUTES.pendingStepExecutionForRun which correctly uses encodeURIComponent, the unencoded runId in line 14 causes incorrect routing or 404 errors when the value is interpolated into the path.

-  updateStepExecution: (runId: string) => `/liana/v1/workflow-step-executions/${runId}/complete`,
+  updateStepExecution: (runId: string) => `/liana/v1/workflow-step-executions/${encodeURIComponent(runId)}/complete`,

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file packages/workflow-executor/src/adapters/forest-server-workflow-port.ts around line 14:

When `runId` contains URL-reserved characters (like `/`, `?`, or `#`), `ROUTES.updateStepExecution` produces a malformed path because the parameter is not encoded. Unlike `ROUTES.pendingStepExecutionForRun` which correctly uses `encodeURIComponent`, the unencoded `runId` in line 14 causes incorrect routing or 404 errors when the value is interpolated into the path.

Evidence trail:
packages/workflow-executor/src/adapters/forest-server-workflow-port.ts lines 12-14 at REVIEWED_COMMIT:
- Line 12-13: `pendingStepExecutionForRun` uses `encodeURIComponent(runId)`
- Line 14: `updateStepExecution` does NOT use `encodeURIComponent(runId)`

[macroscopeapp]

🟢 Low src/build-workflow-executor.ts:77

The stopTimeoutMs option is accepted in ExecutorOptions and passed through buildCommonDependencies, but createWorkflowExecutor never receives it. The force exit timeout on line 115 always uses the hardcoded FORCE_EXIT_DELAY_MS (5000ms), making the user-configured value ineffective.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file packages/workflow-executor/src/build-workflow-executor.ts around line 77:

The `stopTimeoutMs` option is accepted in `ExecutorOptions` and passed through `buildCommonDependencies`, but `createWorkflowExecutor` never receives it. The force exit timeout on line 115 always uses the hardcoded `FORCE_EXIT_DELAY_MS` (5000ms), making the user-configured value ineffective.

Evidence trail:
packages/workflow-executor/src/build-workflow-executor.ts lines 20, 37, 70, 73, 108-111, 145, 165 at REVIEWED_COMMIT. Key evidence: `stopTimeoutMs` defined in ExecutorOptions (line 37), included in buildCommonDependencies return (line 70), but createWorkflowExecutor signature only accepts (runner, server, logger) (line 73), and uses hardcoded FORCE_EXIT_DELAY_MS=5000 (lines 20, 108-111).

[macroscopeapp]

🟠 High src/runner.ts:236

The finally block deletes the step key from inFlightSteps before updateStepExecution reports the outcome. During this window, a concurrent runPollCycle can fetch the same pending step, see it's not in-flight, and spawn a duplicate execution. Move the deletion to after the outcome is successfully reported.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file packages/workflow-executor/src/runner.ts around line 236:

The `finally` block deletes the step key from `inFlightSteps` before `updateStepExecution` reports the outcome. During this window, a concurrent `runPollCycle` can fetch the same pending step, see it's not in-flight, and spawn a duplicate execution. Move the deletion to after the outcome is successfully reported.

Evidence trail:
packages/workflow-executor/src/runner.ts lines 237-268: `doExecuteStep` method shows `finally` block at line 253 with `this.inFlightSteps.delete(key)` executing before `updateStepExecution` call at line 256. packages/workflow-executor/src/runner.ts lines 200-206: `runPollCycle` filters steps using `!this.inFlightSteps.has(Runner.stepKey(s))` check.

[macroscopeapp]

🟢 Low adapters/agent-client-agent-port.ts:92

In getRelatedData, resolveSchema(relation) passes the relation name (e.g., "orders") to the schema cache, but relation names often differ from their target collection names. When the lookup fails, it silently falls back to primaryKeyFields: ['id'] on line 155. This causes extractRecordId on line 104 to extract wrong primary key values when the actual related collection has a composite key or differently-named primary key field, producing RecordData with incorrect recordId values.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file packages/workflow-executor/src/adapters/agent-client-agent-port.ts around line 92:

In `getRelatedData`, `resolveSchema(relation)` passes the relation name (e.g., "orders") to the schema cache, but relation names often differ from their target collection names. When the lookup fails, it silently falls back to `primaryKeyFields: ['id']` on line 155. This causes `extractRecordId` on line 104 to extract wrong primary key values when the actual related collection has a composite key or differently-named primary key field, producing `RecordData` with incorrect `recordId` values.

Evidence trail:
packages/workflow-executor/src/adapters/agent-client-agent-port.ts:87-106 (getRelatedData method, line 92 calls resolveSchema(relation)), packages/workflow-executor/src/adapters/agent-client-agent-port.ts:148-157 (resolveSchema method with fallback to primaryKeyFields: ['id']), packages/workflow-executor/src/schema-cache.ts:15,26 (cache is keyed by collectionName), packages/workflow-executor/src/types/record.ts:11-13 (FieldSchema has relatedCollectionName property that should be used), packages/workflow-executor/test/adapters/agent-client-agent-port.test.ts:55-65 (test setup shows 'orders' collection with composite key ['tenantId', 'orderId'])

[macroscopeapp]

🟢 Low types/step-execution-data.ts:38

ReadRecordStepExecutionData marks executionResult as required, but every other step type has executionResult?: (optional). This breaks the incremental data-building pattern used elsewhere—code that creates a ReadRecordStepExecutionData before execution completes will fail TypeScript compilation. Consider adding ? to make it optional like the other step types.

-  executionResult: { fields: FieldReadResult[] };
+  executionResult?: { fields: FieldReadResult[] };

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file packages/workflow-executor/src/types/step-execution-data.ts around lines 38-41:

`ReadRecordStepExecutionData` marks `executionResult` as required, but every other step type has `executionResult?:` (optional). This breaks the incremental data-building pattern used elsewhere—code that creates a `ReadRecordStepExecutionData` before execution completes will fail TypeScript compilation. Consider adding `?` to make it optional like the other step types.

Evidence trail:
packages/workflow-executor/src/types/step-execution-data.ts at REVIEWED_COMMIT:
- Line 40: `executionResult: { fields: FieldReadResult[] };` (required, no `?`)
- Line 16: `executionResult?: { answer: string };` (optional)
- Line 49: `executionResult?: { updatedValues: Record<string, unknown> } | { skipped: true };` (optional)
- Line 72: `executionResult?: { success: true; actionResult: unknown } | { skipped: true };` (optional)
- Line 93: `executionResult?: ...` (optional)
- Line 103: `executionResult?: Record<string, unknown>;` (optional)
- Line 125: `executionResult?: { relation: RelationRef; record: RecordRef } | { skipped: true };` (optional)

[macroscopeapp]

🟢 Low executors/load-related-record-step-executor.ts:442

The runtime validation in selectBestRecordIndex only checks range, not integer type. If the AI returns a float like 1.5, it passes the range check but candidates[1.5] returns undefined (JavaScript arrays only accept integer indices), causing a TypeError when callers access .recordId on the result.

-    if (recordIndex < 0 || recordIndex > maxIndex) {
+    if (!Number.isInteger(recordIndex) || recordIndex < 0 || recordIndex > maxIndex) {

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file packages/workflow-executor/src/executors/load-related-record-step-executor.ts around line 442:

The runtime validation in `selectBestRecordIndex` only checks range, not integer type. If the AI returns a float like `1.5`, it passes the range check but `candidates[1.5]` returns `undefined` (JavaScript arrays only accept integer indices), causing a TypeError when callers access `.recordId` on the result.

Evidence trail:
packages/workflow-executor/src/executors/load-related-record-step-executor.ts lines 410-419 (Zod schema with .int()), lines 440-445 (comment stating Zod is not validated + runtime guard without integer check), lines 233-236 (caller using returned index for array access), lines 203-210 (preRecordedArgs path showing integer check pattern that's missing from AI path), lines 447-454 (toRecordRef function that would receive undefined).