Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

The LLM Evaluation Framework

A framework for few-shot evaluation of language models.

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

This is the repository of our article published in RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" and of several follow-up studies.

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

Data-Driven Evaluation for LLM-Powered Applications

PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.

Python SDK for running evaluations on LLM generated responses

The official evaluation suite and dynamic data release for MixEval.

build and benchmark deep research

A research library for automating experiments on Deep Graph Networks

AI Data Management & Evaluation Platform

MedEvalKit: A Unified Medical Evaluation Framework

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.

A framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings.

Expressive is a cross-platform expression parsing and evaluation framework. The cross-platform nature is achieved through compiling for .NET Standard so it will run on practically any platform.