Add LLM judge evaluator factory

[fcogidi]

Summary

This PR adds a reusable LLM-as-a-judge evaluator factory for the evaluation harness, including request/retry configuration, shared grader utilities, and comprehensive unit tests for success and error paths.

Clickup Ticket(s): N/A

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📝 Documentation update
• 🔧 Refactoring (no functional changes)
• ⚡ Performance improvement
• 🧪 Test improvements
• 🔒 Security fix

Changes Made

• Added make_llm_as_judge_evaluator with structured response parsing and mapping to Langfuse-compatible Evaluation objects.
• Added grader config/util modules for OpenAI-compatible parse calls, retry handling, prompt/rubric rendering, markdown loading, and deterministic error metrics.
• Added unit tests covering successful metric mapping, default/custom rubric behavior, error fallback paths, and confidence validation boundaries.

Testing

• Tests pass locally (uv run pytest tests/)
• Type checking passes (uv run mypy <src_dir>)
• Linting passes (uv run ruff check src_dir/)
• Manual testing performed (describe below)

Manual testing details:
N/A

Screenshots/Recordings

N/A

Related Issues

N/A

Deployment Notes

N/A

Checklist

• Code follows the project's style guidelines
• Self-review of code completed
• Documentation updated (if applicable)
• No sensitive information (API keys, credentials) exposed

[Copilot]

Pull request overview

Adds a reusable LLM-as-a-judge evaluator factory to the evaluation harness with shared OpenAI-parse utilities, rubric/prompt rendering, deterministic error metrics, and unit tests for success/error scenarios.

Changes:

• Introduces make_llm_as_judge_evaluator with structured response parsing and mapping to Evaluation objects
• Adds shared grader utilities for structured parse calls (with retry), markdown loading, prompt rendering, and prompt serialization
• Adds pytest coverage for successful mapping, default/custom rubric behavior, and deterministic error metrics

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
aieng-eval-agents/aieng/agent_evals/evaluation/graders/llm_judge.py	Adds the LLM-judge evaluator factory and response-to-evaluations mapping
aieng-eval-agents/aieng/agent_evals/evaluation/graders/_utils.py	Adds shared helpers for structured parse calls, retry logic, rubric rendering, and serialization
aieng-eval-agents/aieng/agent_evals/evaluation/graders/config.py	Introduces request/retry configuration for LLM grader calls
aieng-eval-agents/aieng/agent_evals/evaluation/graders/init.py	Exposes grader factory/types at the package level
aieng-eval-agents/tests/aieng/agent_evals/evaluation/graders/test_llm_judge.py	Adds comprehensive unit tests for the evaluator factory and helpers
aieng-eval-agents/tests/aieng/agent_evals/evaluation/init.py	Establishes test package structure for evaluation modules
aieng-eval-agents/tests/aieng/agent_evals/evaluation/graders/init.py	Establishes test package structure for grader tests

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.