Skip to content

Add Verification Stack use case group with QA Changes page#501

Draft
xingyaoww wants to merge 3 commits into
mainfrom
add-verification-stack-docs
Draft

Add Verification Stack use case group with QA Changes page#501
xingyaoww wants to merge 3 commits into
mainfrom
add-verification-stack-docs

Conversation

@xingyaoww
Copy link
Copy Markdown
Contributor

Summary

Creates a Verification Stack use case group in the docs, grouping the code review and QA agents under a unified section that explains the full verification pipeline. This supports the upcoming blog post by providing a permanent home for detailed setup instructions.

Changes

New pages

  • verification-stack.mdx — Overview of the layered verification system:

    • Layer 1: Trajectory-level verifier (critic model) — links to existing critic docs, placeholder for upcoming configuration integration
    • Layer 2: Repo-level verifier (ReviewBot + QA agent) — setup instructions for both GitHub Actions and OpenHands Automations (beta)
    • Effectiveness summary (key metrics from production usage on software-agent-sdk)
    • The iterate skill — how it closes the feedback loop between verifiers and the agent
    • Related resources linking to blog posts, critic docs, and automations
  • qa-changes.mdx — QA agent use case page:

    • Four-phase methodology (Understand → Setup → Exercise → Report)
    • What it does vs. doesn't (vs. CI and code review)
    • Quick start for GitHub Actions and in-conversation usage
    • QA report format example
    • Customization via repo-specific QA guidelines
    • Troubleshooting section

Navigation restructuring

  • Moved code-review.mdx into a new "Verification Stack" sub-group alongside verification-stack.mdx and qa-changes.mdx
  • Added Verification Stack and QA Testing cards to the use-cases overview page

Navigation structure (under Use Cases)

Use Cases/
  Vulnerability Remediation
  Verification Stack/          ← new group
    The Verification Stack     ← new overview
    Automated Code Review      ← moved here
    Automated QA Testing       ← new page
  Incident Triage
  COBOL Modernization
  Dependency Upgrades
  Spark Migrations

Context

This PR is part of the work for AGE-512. The goal is to provide a permanent docs home for verification stack content so the companion blog post can stay high-level and link here for detailed instructions.


This PR was created by an AI agent (OpenHands) on behalf of the user.

@xingyaoww can click here to continue refining the PR

- Create verification-stack.mdx: overview of the layered verification system
  covering Layer 1 (trajectory-level critic), Layer 2 (ReviewBot + QA agent),
  setup instructions for both GitHub Actions and OpenHands Automations,
  effectiveness summary, and the iterate skill
- Create qa-changes.mdx: QA agent use case page with methodology, quick start,
  report format, customization, and troubleshooting
- Move code-review.mdx under a new 'Verification Stack' nav group alongside
  qa-changes and the overview page
- Update overview.mdx with Verification Stack and QA Testing cards
- Update docs.json navigation to nest the three pages under a
  'Verification Stack' sub-group within Use Cases

Co-authored-by: openhands <openhands@all-hands.dev>
@mintlify
Copy link
Copy Markdown

mintlify Bot commented May 11, 2026

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
all-hands-ai 🟢 Ready View Preview May 11, 2026, 1:50 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

…tions

Add a 'Setup Checklist for GitHub Event Automations' section that covers:
- GitHub App installation requirements
- Team org vs personal org event routing (common pitfall)
- Service account setup for bot-driven automations
- Complete PR review bot walkthrough example
- Troubleshooting tips for when automations don't trigger

This addresses a non-obvious gotcha where automations created under a
personal org silently never receive events for org repos, because GitHub
events are routed to the team org that claimed the GitHub organization.

Co-authored-by: openhands <openhands@all-hands.dev>
…nt automations

Rewrites the 'Setup Checklist' section (from previous commit) into a more
complete 'Prerequisites for GitHub Event Automations' section based on
real-world debugging of a silent event delivery failure.

Key changes:
- Adds 'Claim Your GitHub Organization' as the #1 prerequisite with a
  Warning callout — this was the actual root cause (missing OrgGitClaim
  meant events had nowhere to route)
- Adds 'Create an OpenHands Team Organization' as a prerequisite step
- Links to the existing claiming docs at
  /openhands/usage/cloud/organizations/settings#claiming-git-organizations
- Replaces the flat Tip block with an AccordionGroup troubleshooting
  section covering the 4 most common failure modes
- Updates the summary table to link to prerequisites instead of saying
  'None — just create the automation'

Co-authored-by: openhands <openhands@all-hands.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants