Skip to content

[Feature] gitagent test -- run against example scenarios #5

@shreyas-lyzr

Description

@shreyas-lyzr

Summary

Add a gitagent test command that runs an agent against predefined test scenarios.

Problem

There is no standard way to verify that an agent behaves correctly before deploying it. Agent developers need a testing framework analogous to unit tests for code.

Proposed Solution

# Run all tests for the current agent
gitagent test

# Run a specific test scenario
gitagent test --scenario greeting

# Run with a specific adapter
gitagent test --adapter openai

Test scenarios would live in tests/ or examples/ within the agent repo:

# tests/greeting.yaml
name: greeting-test
input: "Hello, who are you?"
expect:
  contains: ["agent name"]
  tone: professional
  max_tokens: 500

Acceptance Criteria

  • gitagent test command implemented
  • YAML-based test scenario format defined
  • Works with multiple adapters
  • Clear pass/fail output
  • Documentation added

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions