fix(app): preserve custom policy strategy previews#8328
fix(app): preserve custom policy strategy previews#8328
Conversation
![promptfoo-scanner[bot]](https://avatars.githubusercontent.com/in/2172945?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 All Clear
I reviewed the front-end changes around red team test generation and plugin selection. The PR primarily refactors plugin representation to preserve config and switches the selection UI to index-based values, without altering backend capabilities or execution paths. Tracing shows no new privileged tool access or unsafe prompt/data flows introduced. No LLM security vulnerabilities were identified.
Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more
ianw-oai
changed the title
…cy-strategy-previews # Conflicts: # src/app/src/pages/redteam/setup/components/strategies/useStrategyTestGeneration.ts
ianw-oai
requested review from
faizan-oai,
mldangelo-oai and
wholley-oai
as code owners
There was a problem hiding this comment.
Pull request overview
This PR fixes strategy test-case previews when the configured plugin set contains configured plugin objects (e.g., custom policy/intent entries) by preserving plugin configuration through the preview flow and making duplicate plugin IDs distinguishable in the UI.
Changes:
- Preserve configured plugin objects (id + config) instead of flattening to string IDs for preview plugin selection.
- Update the strategy preview dialog to select plugins by index (vs id) and render distinct labels/keys for multiple
policyentries. - Add/adjust tests to cover preservation of custom policy configuration during strategy preview generation.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/app/src/pages/redteam/setup/components/TestCaseGenerationProvider.tsx | Keeps availablePlugins as TargetPlugin[] and regenerates using the selected plugin’s full config. |
| src/app/src/pages/redteam/setup/components/TestCaseDialog.tsx | Updates plugin dropdown to use option indices + adds custom labeling for policy entries. |
| src/app/src/pages/redteam/setup/components/TestCaseDialog.test.tsx | Updates dialog test fixtures to pass TargetPlugin[] instead of string[]. |
| src/app/src/pages/redteam/setup/components/strategies/useStrategyTestGeneration.ts | Ensures strategy preview generation uses configured plugin objects (preserving config). |
| src/app/src/pages/redteam/setup/components/strategies/useStrategyTestGeneration.test.tsx | Adds tests verifying custom policy config is forwarded and default fallback behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
📝 WalkthroughWalkthroughThis pull request refactors the plugin system in red-team test case generation from using simple string identifiers to full plugin descriptor objects. The changes update Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/app/src/pages/redteam/setup/components/TestCaseDialog.test.tsx`:
- Around line 220-224: Add a regression fixture in TestCaseDialog.test.tsx that
uses duplicate plugin ids to exercise the index-based selection path: update the
test data where availablePlugins is defined to include two configured policy
entries with the same id (e.g., two 'policy' entries) and render the
TestCaseDialog component; then assert the dropdown shows distinct labels for
each configured policy entry and that when selecting one the onRegenerate prop
receives the selected index (not the id). Reference the test utilities and props
used in this file (availablePlugins, policy entries, TestCaseDialog, and
onRegenerate) so the new case verifies index-based selection behavior.
In `@src/app/src/pages/redteam/setup/components/TestCaseDialog.tsx`:
- Around line 160-176: The dropdown labels for non-`policy` plugins need
disambiguation when the same plugin `id` appears multiple times: update
getAvailablePluginLabel to compute an instance index for any option.id (not just
'policy') by counting occurrences of that id in availablePlugins up to the
current index (similar to the existing policyIndex logic) and append a suffix
(e.g., " `#2`" or " (2)") to the base label derived from
displayNameOverrides[option.id as Plugin] || categoryAliases[option.id as
Plugin] || option.id whenever that count is greater than 1; this preserves
current behavior for unique ids while making duplicate instances distinguishable
in the UI.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0ac72c0a-a02d-44d3-9024-2a52437d7cd5
📒 Files selected for processing (5)
src/app/src/pages/redteam/setup/components/TestCaseDialog.test.tsxsrc/app/src/pages/redteam/setup/components/TestCaseDialog.tsxsrc/app/src/pages/redteam/setup/components/TestCaseGenerationProvider.tsxsrc/app/src/pages/redteam/setup/components/strategies/useStrategyTestGeneration.test.tsxsrc/app/src/pages/redteam/setup/components/strategies/useStrategyTestGeneration.ts
Summary
This change fixes strategy test case previews when the configured plugin set includes custom policies or other configured plugin objects. Before this patch, the strategy preview flow collapsed configured plugins down to raw plugin ids and then rebuilt a preview request with an empty config object. For built-in string plugins that happened to work, but for configured plugins such as
policyorintentit meant the preview request lost the data required to generate a test case.For users, that showed up in two ways. First, previewing a strategy could fail with the generic "Failed to generate test case" error because the backend received
policywithout the requiredconfig.policypayload. Second, when multiple custom policies were present, the strategy preview dialog rendered duplicatepolicyoptions, which triggered the React duplicate-key warning and made the dropdown unable to distinguish one custom policy from another during regeneration.Root Cause
The root cause was that the strategy preview path was preserving only plugin identity, not plugin configuration. The hook that starts strategy previews selected a plugin by id and always sent an empty config object. Separately, the dialog/provider path represented the available preview plugins as an array of strings, so multiple configured instances of the same plugin id were indistinguishable in the UI.
Fix
The fix keeps configured preview plugins as full plugin objects instead of flattening them to strings. The strategy preview hook now forwards the selected plugin's real config when calling the generation provider, which preserves custom policy and intent definitions. The provider now keeps the available preview plugins as
TargetPlugin[], and the dialog uses the option index as the select value and part of the React key so multiple configured plugins with the same id remain distinct. The dialog also derives a readable label for custom policy entries so the selector is understandable when more than one policy is configured.Validation
I installed dependencies with
npm ciunder the repo's pinned Node version from.nvmrcand reran the focused frontend tests covering this flow. The following command passed:That run completed with 3 test files passed and 34 tests passed.