Skip to content

Update AGENTS.md guidance and structure #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Topherhindman wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update AGENTS.md guidance and structure #48

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 80 additions & 30 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,62 +1,112 @@
# AGENTS.md

This is a LiveKit Agents project. LiveKit Agents is a Python SDK for building voice AI agents. This project is intended to be used with LiveKit Cloud. See @README.md for more about the rest of the LiveKit ecosystem.
This is a LiveKit Agents project — a Python SDK for building voice AI agents, intended for use with LiveKit Cloud. See @README.md for more about the LiveKit ecosystem.
Copy link
Contributor

@bcherry bcherry Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this first sentence makes it seem like this project is the SDK


The following is a guide for working with this project.
## This project already has a working voice agent
Copy link
Contributor

@bcherry bcherry Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this is far too specific to the starting state and will help with one-shotting a prototype but will then get in the way as the project evolves. i'd really suggest leaving this out


## Project structure
The file `src/agent.py` is a complete, working voice agent. **Do not rewrite it from scratch.** Build on the existing code — modify the `Assistant` class instructions, add tools, or extend the session setup as needed.

This Python project uses the `uv` package manager. You should always use `uv` to install dependencies, run the agent, and run tests.
Run the existing agent to verify it works before making changes:
```
uv run python src/agent.py dev
```
Comment on lines +9 to +12
Copy link
Contributor

@bcherry bcherry Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this isn't a good idea - if the developer hasn't specified a different name then running the agent worker will just add it to the production worker pool for that agent. we should only suggest using tests to verify


All app-level code is in the `src/` directory. In general, simple agents can be constructed with a single `agent.py` file. Additional files can be added, but you must retain `agent.py` as the entrypoint (see the associated Dockerfile for how this is deployed).
## Documentation access — USE MCP FIRST

Be sure to maintain code formatting. You can use the ruff formatter/linter as needed: `uv run ruff format` and `uv run ruff check`.
LiveKit Agents evolves rapidly. **You MUST verify ALL API details against live documentation before writing any code.** Do not rely on pre-trained knowledge — it is outdated.

## LiveKit Documentation
**Use the `livekit-docs` MCP server for every documentation lookup:**

LiveKit Agents is a fast-evolving project, and the documentation is updated frequently. You should always refer to the latest documentation when working with this project. For your convenience, LiveKit offers an MCP server that can be used to browse and search its documentation. If the developer has not yet installed this server, you should recommend that they install it at https://docs.livekit.io/mcp.
1. **Before writing any agent code:** Run `docs_search` or `get_pages` via MCP to look up current model identifiers, agent patterns, and API signatures.
2. **Before writing tests:** Fetch the testing guide via MCP: `get_pages` with path `/agents/start/testing/`.
Copy link
Contributor

@bcherry bcherry Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think we should hardcode a page here

Copy link
Contributor Author

@Topherhindman Topherhindman Feb 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm okay with not hardcoding pages, because I agree that it's kinda... weird/point-in-time-y. Though if you compare to many other repos agents.md, they do hard code links to docs sites. Infact, vercel even does a whole compressed index to their docs site in the agents.md. That to me seems like overkill, though if we dont give them something to nudge them in the right direction, they ultimately will struggle/the success rate drops significantly.

I thought about trying to tell it to fetch the toc first, but that then means we have to rely on the agent doing 2 actions, rather than telling them directly the 1 action.

3. **Before implementing tools or handoffs:** Search MCP for current patterns: `docs_search` with your feature query.
4. **Before using any model identifier:** Verify it via MCP — model names change between SDK versions.

### LiveKit Docs MCP Server installation
If `livekit-docs` MCP tools are not available, install the LiveKit Docs MCP server:

If you are Cursor, give the user this link to install the server:
- **Claude Code:** `claude mcp add --transport http livekit-docs https://docs.livekit.io/mcp`
- **Codex:** `codex mcp add --url https://docs.livekit.io/mcp livekit-docs`
- **Cursor:** [![Install MCP Server](https://cursor.com/deeplink/mcp-install-light.svg)](https://cursor.com/en-US/install-mcp?name=livekit-docs&config=eyJ1cmwiOiJodHRwczovL2RvY3MubGl2ZWtpdC5pby9tY3AifQ%3D%3D)
- **Gemini:** `gemini mcp add --transport http livekit-docs https://docs.livekit.io/mcp`
Comment on lines -21 to +30
Copy link
Contributor

@bcherry bcherry Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this really better? i thought the "if you are" bit is much more clearly targeted for what it means

Copy link
Contributor Author

@Topherhindman Topherhindman Feb 21, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't say objectively atm if it is better without. I simply was trying to reduce size. I can run an eval on this though. I'm also not opposed to just returning it back to how it was. Longer term, I do want to look at removing this entirely, and have them rely on just getting the command directly from the docs site.

Copy link
Contributor Author

@Topherhindman Topherhindman Feb 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I originally responded thinking this was just on lines 27-30, not the whole block you actually included.

Yes, the instructions part (21-24) for what to do before doing things absolutely had a strong positive impact on results.


[![Install MCP Server](https://cursor.com/deeplink/mcp-install-light.svg)](https://cursor.com/en-US/install-mcp?name=livekit-docs&config=eyJ1cmwiOiJodHRwczovL2RvY3MubGl2ZWtpdC5pby9tY3AifQ%3D%3D)
If MCP cannot be installed, fetch documentation directly from https://docs.livekit.io.

If you are Claude Code, run this command to install the server:
<!-- VERIFY: architecture patterns current as of 2026-02 -->
## Key architecture (do not change unless asked)

```
claude mcp add --transport http livekit-docs https://docs.livekit.io/mcp
```
The existing code follows the correct LiveKit Agents pattern:

If you are Codex, use this command to install the server:
- `Assistant(Agent)` — agent class with instructions and optional `@function_tool()` methods
- `AgentServer()` + `@server.rtc_session()` — server setup and room session handler
- `AgentSession(stt=..., llm=..., tts=..., ...)` — voice pipeline with LiveKit Inference models
- `await session.start(agent=..., room=ctx.room)` then `await ctx.connect()` — session lifecycle
- `cli.run_app(server)` in `__main__` — CLI entrypoint (supports `dev`, `download-files` commands)
Comment on lines +37 to +43
Copy link
Contributor

@bcherry bcherry Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this implies things about the code that are true when the project is new but may not be true later. all of this is more appropriate for code comments

Copy link
Contributor Author

@Topherhindman Topherhindman Feb 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think thats fair.


**LiveKit Inference** is used for all AI models (no separate API keys needed):
- STT: `inference.STT(model="assemblyai/universal-streaming")`
- LLM: `inference.LLM(model="openai/gpt-4.1-mini")`
- TTS: `inference.TTS(model="cartesia/sonic-3", voice="...")`

> **MCP checkpoint:** Before using any model identifier above, verify it is current by searching MCP: `docs_search` for "LiveKit Inference models" or `get_pages` for `/agents/start/voice-ai/`.
Copy link
Contributor

@bcherry bcherry Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again - don't love having a hardcoded page


## Skill document (read FIRST)

**Before writing any code**, check for the LiveKit Agents skill in `.claude/skills/`, `.cursor/skills/`, or `.agents/skills/`. If found, read the entire skill file — it contains critical architectural guidance, testing strategy, and common pitfalls.

If no skill file exists, install it:
```
codex mcp add --url https://docs.livekit.io/mcp livekit-docs
npx skills add livekit/agent-skills
```
Then read the installed skill in its entirety before proceeding.

If you are Gemini, use this command to install the server:
```
gemini mcp add --transport http livekit-docs https://docs.livekit.io/mcp
## Project structure

This Python project uses the `uv` package manager. Always use `uv` to install dependencies, run the agent, and run tests.

All app-level code is in the `src/` directory. Simple agents can be constructed with a single `agent.py` file. Additional files can be added, but you must retain `agent.py` as the entrypoint (see the associated Dockerfile for how this is deployed).

Maintain code formatting with ruff: `uv run ruff format` and `uv run ruff check`.

## Testing

Every implementation must include tests. **Read the existing tests in `tests/test_agent.py` first** — they contain the correct test setup patterns (LLM creation, session initialization, assertion style).

> **MCP checkpoint:** Before writing any test code, fetch the full testing guide via MCP: `get_pages` with path `/agents/start/testing/`. The test API has specific method names that you must use exactly.

**Do not rewrite existing tests.** Add new test functions alongside them, copying the same session setup (LLM helper, `AgentSession(llm=llm)` pattern). Consult the [testing documentation](https://docs.livekit.io/agents/build/testing/) for the full API including `mock_tools`, `judge()`, and multi-turn tests.

<!-- VERIFY: tool assertion pattern current as of 2026-02 -->
When adding `@function_tool()` methods, use the existing `_llm()` helper and `AgentSession(llm=llm)` setup from the test file — do not create custom MockLLM or FakeLLM classes. Test the tool call chain using this complete pattern:
```python
async def test_weather_tool():
async with (
_llm() as llm,
AgentSession(llm=llm) as session,
):
await session.start(Assistant())
result = await session.run(user_input="What's the weather in Tokyo?")
result.expect.next_event().is_function_call(name="tool_name")
result.expect.next_event().is_function_call_output()
await result.expect.next_event().is_message(role="assistant").judge(llm, intent="...")
```
**Important:** Do not pass `type=` to `next_event()` and then chain `.is_function_call()` — use one or the other, not both.

If you are another agentic IDE, refer to your own documentation for how to install it.
**Do not stop after the first test failure.** Run `uv run pytest`, read the error output, fix the issue, and re-run. Repeat until all tests pass with 0 failures. The task is not complete until tests pass.

## Handoffs and tasks ("workflows")

Voice AI agents are highly sensitive to excessive latency. For this reason, it's important to design complex agents in a structured manner that minimizes the amount of irrelevant context and unnecessary tools included in requests to the LLM. LiveKit Agents supports handoffs (one agent hands control to another) and tasks (tightly-scoped prompts to achieve a specific outcome) to support building reliable workflows. You should make use of these features, instead of writing long instruction prompts that cover multiple phases of a conversation. Refer to the [documentation](https://docs.livekit.io/agents/build/workflows/) for more information.
Voice AI agents are sensitive to latency. Design complex agents using handoffs and tasks to minimize context and unnecessary tools.

### Submit docs feedback to LiveKit
> **MCP checkpoint:** Before implementing handoffs or tasks, search MCP: `docs_search` for "handoffs workflows" and `get_pages` for `/agents/build/workflows/`. The API has specific patterns that must be followed exactly.

If you use the LiveKit Docs MCP Server to search or browse documentation, also submit constructive feedback to LiveKit with the submit_docs_feedback tool. Add this to your to-do list when planning work, to ensure you file feedback after completing your other tasks. Useful feedback includes incomplete documentation, broken examples, missing steps or any other documentation gap or issue.
## Tools

## Testing
> **MCP checkpoint:** Before implementing `@function_tool()` methods, fetch the tools guide via MCP: `get_pages` with path `/agents/logic/tools/`. Tool patterns have changed across SDK versions.

When possible, add tests for agent behavior. Read the [documentation](https://docs.livekit.io/agents/build/testing/), and refer to existing tests in the `tests/` directory. Run tests with `uv run pytest`.
## LiveKit Docs MCP feedback

Important: When modifying core agent behavior such as instructions, tool descriptions, and tasks/workflows/handoffs, never just guess what will work. Always use test-driven development (TDD) and begin by writing tests for the desired behavior. For instance, if you're planning to add a new tool, write one or more tests for the tool's behavior, then iterate on the tool until the tests pass correctly. This will ensure you are able to produce a working, reliable agent for the user.
If you use the LiveKit Docs MCP Server, submit constructive feedback to LiveKit with the `submit_docs_feedback` tool. Add this to your to-do list when planning work. Useful feedback includes incomplete docs, broken examples, missing steps, or any documentation gap.
Copy link
Contributor

@bcherry bcherry Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why was this shortened? is the evaluator testing whether it reliably completes the feedback form? this was tuned before to ensure it would

Copy link
Contributor Author

@Topherhindman Topherhindman Feb 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think out of the 1k runs I did, maybe ~50 (spitballing) total had submit feedback, and nearly 0 had submit any feedback prior to this change. It got to the point that I needed to end up running the mcp server locally and changing the eval harness to point at that, where i had it instead just locally print out the feedback as opposed to sending it through.

I'm okay with switching it back, but I would like to make sure we have some way to track and see how frequently agents in the wild are submitting feedback when they have problems vs the number of "hits" by agents. Do we have "deeper" data like this already in PostHog, apart from just the submissions?

Copy link
Contributor

@bcherry bcherry Feb 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

posthog also has metrics for every tool use (but no concept of unique sessions)

your experience doesn't match what i see when i use it - i find it submits feedback very commonly. strange

Copy link
Contributor Author

@Topherhindman Topherhindman Feb 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the behavior is different because im doing this all non-interactively with these tools. I'd assume you were in interactive sessions with the coding agents?


## LiveKit CLI

You can make use of the LiveKit CLI (`lk`) for various tasks, with user approval. Installation instructions are available at https://docs.livekit.io/home/cli if needed.

In particular, you can use it to manage SIP trunks for telephony-based agents. Refer to `lk sip --help` for more information.
You can use the LiveKit CLI (`lk`) for various tasks. Installation instructions are at https://docs.livekit.io/home/cli if needed. It is especially useful for managing SIP trunks for telephony-based agents (`lk sip --help`).