Phase 1: Foundation - OpenAdapt Capture Integration

[abrichr]

Phase 1: Foundation

Integrate openadapt-capture for real recording and replay functionality.

Goals

1. Real recording: WorkflowRecorder uses openadapt-capture to record actual GUI events
2. Real replay: WorkflowExecutor replays recordings with action execution
3. Parameter substitution: Replace literal values (file paths, coordinates) with parameters
4. Basic testing: Comprehensive test suite for core functionality

Tasks

1. Integrate openadapt-capture Recording

• Update WorkflowRecorder.__enter__ to start openadapt_capture.Recorder
• Store recording in recordings/{workflow_name}/recording.db
• Add workflow metadata to recording
• Update WorkflowRecorder.__exit__ to stop recorder and save manifest

Files: workflows/base.py

2. Implement Replay from Recordings

• Load recording from recording.db
• Parse events into actions (clicks, typing, etc.)
• Execute actions via platform-specific automation (pyautogui, pynput, etc.)
• Add timing/delays between actions

Files: workflows/base.py, new playback/executor.py

3. Parameter Substitution

• Identify parameterizable values in recordings (file paths, text inputs)
• Extract parameters during recording
• Substitute parameters during replay
• Support environment variables

Example:

# Recording: Click file dialog, type "/absolute/path/to/file.html"
# Replay: Click file dialog, type parameters["html_path"]

Files: workflows/base.py, playback/parameter_substitution.py

4. Screenshot Workflow Real Implementation

• Update ScreenshotWorkflow to record actual browser automation
• Record opening HTML, resizing window, capturing screenshots
• Replay workflow to generate screenshots
• Validate generated screenshots match expected viewports

Files: workflows/screenshot_workflow.py

5. Comprehensive Testing

• Test recording workflow end-to-end
• Test replay workflow end-to-end
• Test parameter substitution
• Test error handling (missing files, failed actions)
• Add CI/CD pipeline

Files: tests/test_recording.py, tests/test_replay.py, tests/test_screenshot_workflow.py

Dependencies

• openadapt-capture: Already in dependencies
• pyautogui or pynput: For action execution during replay
• Platform-specific automation libraries

Success Criteria

✅ Record a screenshot workflow manually
✅ Replay workflow to generate screenshots automatically
✅ Parameter substitution works (different HTML files)
✅ 95%+ test coverage
✅ Documentation updated

Estimated Effort

8-12 hours (recording integration, replay implementation, testing)

Related Issues

• Depends on: None (foundation)
• Blocks: Phase 2: Claude Code Integration - Autonomous Workflow Execution #3 (Phase 2: Claude Code Integration)

Resources

• openadapt-capture docs
• Architecture doc