Skip to content

fix(ai-red-teaming): save_workflow verification and overwrite detection (ENG-6812)#15

Merged
rdheekonda merged 1 commit into
mainfrom
fix/save-workflow-verification-eng-6812
May 17, 2026
Merged

fix(ai-red-teaming): save_workflow verification and overwrite detection (ENG-6812)#15
rdheekonda merged 1 commit into
mainfrom
fix/save-workflow-verification-eng-6812

Conversation

@rdheekonda
Copy link
Copy Markdown
Contributor

@rdheekonda rdheekonda commented May 16, 2026

Summary

Fixes save_workflow reporting success without actually overwriting files, causing AI agents to operate under stale assumptions when workflow scripts fail to write correctly.

  • Root cause: pathlib.Path.write_text() can silently fail due to permissions, disk space, file locking, or network filesystem issues
  • Impact: AI agents think they wrote new code but file remains unchanged, leading to incorrect behavior and debugging issues
  • Solution: Add write verification by reading back content and comparing with expected

Linear: https://linear.app/dreadnode/issue/ENG-6812/save-workflow-reports-success-without-overwriting-existing-files

Changes

  • tools/workflows.py: Add content verification logic to save_workflow() function
  • scripts/workflow_helper.py: Same verification logic for legacy implementation
  • tests/test_workflow_helper.py: Test overwrite detection and content validation scenarios
  • capability.yaml: Bump version to 1.4.0 (minor version for significant bug fix)

Verification Logic

# Read existing content (if any) for comparison
existing_content = ""
if filepath.exists():
    try:
        existing_content = filepath.read_text()
    except Exception:
        pass

# Attempt write
try:
    filepath.write_text(code)
except Exception as e:
    return f"Error writing file: {e}"

# Verify write succeeded by reading back
try:
    written_content = filepath.read_text()
    if written_content != code:
        return f"Error: File write incomplete..."
    if existing_content and existing_content == written_content and existing_content != code:
        return f"Warning: File exists but content unchanged..."
except Exception as e:
    return f"Error verifying write: {e}"

Test Coverage

  • ✅ Normal file creation and overwrite scenarios
  • ✅ Silent write failure detection (mocked scenarios)
  • ✅ Partial write detection
  • ✅ Content verification edge cases
  • ✅ All existing tests continue to pass

Test plan

  • Run all existing workflow tests: python -m pytest tests/test_workflow_helper.py -v
  • Verify new overwrite detection test passes
  • Test both tools/workflows.py and scripts/workflow_helper.py implementations
  • Confirm capability version bump to 1.4.0

🤖 Generated with Claude Code

@rdheekonda rdheekonda force-pushed the fix/save-workflow-verification-eng-6812 branch from 56049b1 to 54d1852 Compare May 17, 2026 01:18
…on (ENG-6812)

Fixes save_workflow reporting success without actually overwriting files,
causing AI agents to operate under stale assumptions when workflow scripts
fail to write correctly.

**Root Cause:**
- pathlib.Path.write_text() can silently fail due to permissions, disk space,
  file locking, or network filesystem issues
- No verification that content was actually written
- Agent continues with incorrect assumptions about file state

**Solution:**
- Add write verification by reading back content and comparing with expected
- Detect when file content doesn't change during overwrite attempts
- Enhanced error reporting for write failures and verification issues
- Comprehensive test coverage for edge cases and silent failures

**Changes:**
- tools/workflows.py: Add content verification logic to save_workflow()
- scripts/workflow_helper.py: Same verification logic for legacy implementation
- tests/test_workflow_helper.py: Test overwrite detection and content validation
- capability.yaml: Bump version to 1.4.0 (minor version for significant bug fix)

**Testing:**
- All existing tests pass
- New tests verify silent failure detection
- Edge cases covered: permission issues, partial writes, unchanged content

This ensures AI agents get accurate feedback about file operations and can
respond appropriately to write failures instead of operating under stale
assumptions.
@rdheekonda rdheekonda force-pushed the fix/save-workflow-verification-eng-6812 branch from 54d1852 to 9d69166 Compare May 17, 2026 01:20
@rdheekonda rdheekonda merged commit 3cb5224 into main May 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant