Phase 2: Claude Code Integration - Autonomous Workflow Execution

[abrichr]

Phase 2: Claude Code Integration

Enable autonomous decision-making during workflow replay via Claude Code API integration.

Goals

1. Prompt handling: Claude Code makes decisions when workflow encounters prompts
2. Error recovery: Automatic retry, adaptation, or graceful failure
3. UI adaptation: Handle UI changes that break exact replay
4. Background execution: Run workflows without user intervention

Tasks

1. ClaudeCodeIntegration Class

• Create playback/claude_code_integration.py
• Implement handle_prompt() method
• Implement handle_error() method
• Implement adapt_action() for UI changes

API Design:

class ClaudeCodeIntegration:
    def handle_prompt(
        self,
        prompt: str,
        context: dict,
        screenshot: Image.Image | None = None
    ) -> str:
        """Ask Claude Code what to do when encountering a prompt."""
        pass

    def handle_error(
        self,
        error: Exception,
        context: dict
    ) -> Action:
        """Propose fix for error: retry, skip, or abort."""
        pass

    def adapt_action(
        self,
        original_action: Action,
        current_screenshot: Image.Image
    ) -> Action:
        """Adapt action coordinates if UI has changed."""
        pass

2. Integrate with WorkflowExecutor

• Add claude_code_enabled parameter to WorkflowExecutor
• Call Claude Code during replay when needed
• Log Claude Code decisions
• Add timeout/fallback for API failures

Files: workflows/base.py

3. Prompt Handling

Scenarios:

• File save dialog: "What filename should I use?"
• Confirmation dialog: "Should I proceed?"
• Dropdown menu: "Which option should I select?"

Implementation:

• Capture screenshot when prompt detected
• Send to Claude Code with context
• Parse response into action
• Execute action

Files: playback/claude_code_integration.py, playback/executor.py

4. Error Recovery

Error types:

• Element not found (UI changed)
• Timeout waiting for element
• Unexpected dialog/popup
• File/directory not found

Recovery strategies:

• Retry: Try action again with backoff
• Adapt: Use vision to find similar element
• Skip: Continue without action (if non-critical)
• Abort: Fail gracefully with detailed error

Files: playback/error_recovery.py

5. UI Adaptation

Challenge: Recorded actions use exact coordinates, which break if UI changes

Solution: Use vision models to locate UI elements

• Extract element description from original recording
• Use Claude Code vision to find element in current screenshot
• Translate to new coordinates
• Execute adapted action

Example:

Original: CLICK(x=500, y=300) on "Save button"
Current UI: Save button moved to (x=550, y=320)
Adapted: CLICK(x=550, y=320)

Files: playback/ui_adaptation.py

6. Background Execution

• Run workflows in background (no user interaction needed)
• Log execution progress
• Send notifications on completion/failure
• Post results to GitHub (for mobile review)

Integration with openadapt-tray:

• Use tray notifications for workflow status
• "Screenshot workflow completed: 12 images generated"

Files: playback/background_executor.py

7. Testing

• Mock Claude Code API for testing
• Test prompt handling scenarios
• Test error recovery strategies
• Test UI adaptation
• End-to-end test with real Claude Code API

Files: tests/test_claude_code_integration.py, tests/mocks.py

Dependencies

• anthropic: Already in dependencies
• ANTHROPIC_API_KEY: Environment variable
• Phase 1 complete (recording/replay working)

Success Criteria

✅ Claude Code handles prompts during replay
✅ Error recovery adapts to UI changes
✅ Workflows run in background without user
✅ Results posted to GitHub for mobile review
✅ Comprehensive tests with mocked API

Estimated Effort

12-16 hours (Claude Code integration, error handling, UI adaptation, testing)

Related Issues

• Depends on: Phase 1: Foundation - OpenAdapt Capture Integration #2 (Phase 1: Foundation)
• Blocks: Phase 4: Recursive Self-Improvement - Bootstrap Achievement #4 (Phase 3: GitHub Integration)

Resources

• Anthropic API docs
• Vision API guide
• openadapt-tray