fix(ai-red-teaming): save_workflow verification and overwrite detection (ENG-6812)#15
Merged
Merged
Conversation
56049b1 to
54d1852
Compare
…on (ENG-6812) Fixes save_workflow reporting success without actually overwriting files, causing AI agents to operate under stale assumptions when workflow scripts fail to write correctly. **Root Cause:** - pathlib.Path.write_text() can silently fail due to permissions, disk space, file locking, or network filesystem issues - No verification that content was actually written - Agent continues with incorrect assumptions about file state **Solution:** - Add write verification by reading back content and comparing with expected - Detect when file content doesn't change during overwrite attempts - Enhanced error reporting for write failures and verification issues - Comprehensive test coverage for edge cases and silent failures **Changes:** - tools/workflows.py: Add content verification logic to save_workflow() - scripts/workflow_helper.py: Same verification logic for legacy implementation - tests/test_workflow_helper.py: Test overwrite detection and content validation - capability.yaml: Bump version to 1.4.0 (minor version for significant bug fix) **Testing:** - All existing tests pass - New tests verify silent failure detection - Edge cases covered: permission issues, partial writes, unchanged content This ensures AI agents get accurate feedback about file operations and can respond appropriately to write failures instead of operating under stale assumptions.
54d1852 to
9d69166
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes
save_workflowreporting success without actually overwriting files, causing AI agents to operate under stale assumptions when workflow scripts fail to write correctly.pathlib.Path.write_text()can silently fail due to permissions, disk space, file locking, or network filesystem issuesLinear: https://linear.app/dreadnode/issue/ENG-6812/save-workflow-reports-success-without-overwriting-existing-files
Changes
save_workflow()functionVerification Logic
Test Coverage
Test plan
python -m pytest tests/test_workflow_helper.py -vtools/workflows.pyandscripts/workflow_helper.pyimplementations🤖 Generated with Claude Code