Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
[submodule "tests_integration/lib/_definitions/zizmor/feedstock"]
path = tests_integration/lib/_definitions/zizmor/resources/feedstock
url = https://github.com/conda-forge/zizmor-feedstock.git
[submodule "tests_integration/lib/_definitions/dominodatalab/resources/feedstock"]
path = tests_integration/lib/_definitions/dominodatalab/resources/feedstock
url = https://github.com/conda-forge/dominodatalab-feedstock.git
[submodule "tests_integration/lib/_definitions/witr/resources/feedstock"]
path = tests_integration/lib/_definitions/witr/resources/feedstock
url = https://github.com/conda-forge/witr-feedstock.git
226 changes: 226 additions & 0 deletions CLAUDE.md
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you name this file AGENTS.md and symlink from CLAUDE.md to it? This way, other agents (such as Copilot and Cursor) also take this file into account.

(You might also need to remove all references to Claude Code in particular)

Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

This is **autotick-bot** (conda-forge-tick), the automated maintenance bot for the conda-forge ecosystem. It creates PRs to update packages, run migrations, and maintain the conda-forge dependency graph across thousands of feedstocks.

## Common Commands

### Development Setup
```bash
# Using environment.yml (recommended for development)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this true? I think we should also recommend the lockfile for development

conda env create -f environment.yml

# Or using the production lockfile
wget https://raw.githubusercontent.com/regro/cf-scripts/main/conda-lock.yml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should already be there locally, no need to wget

conda-lock install conda-lock.yml

# Install in editable mode
pip install -e .
```

### Running Tests
```bash
# Run all tests (requires docker for container tests)
pytest -v

# Run tests in parallel
pytest -v -n 3

# Run a single test
pytest -v tests/test_file.py::test_function

# Skip MongoDB tests
pytest -v -m "not mongodb"

# To enable container-based tests, first build the test image:
docker build -t conda-forge-tick:test .
```

### CLI Usage
```bash
# General help
conda-forge-tick --help

# Debug mode (enables debug logging, disables multiprocessing)
conda-forge-tick --debug <command>

# Online mode (fetches graph data from GitHub, useful for local testing)
conda-forge-tick --online <command>

# Disable containers (for debugging, but note security implications)
conda-forge-tick --no-containers <command>

# Example: update upstream versions for a single package
conda-forge-tick --debug --online update-upstream-versions numpy
```

### Linting
```bash
# Pre-commit handles linting (ruff, mypy, typos)
pre-commit run --all-files
```

## Architecture

### Core Components

**CLI Entry Points** (`conda_forge_tick/cli.py`, `conda_forge_tick/container_cli.py`):
- `conda-forge-tick`: Main CLI for bot operations
- `conda-forge-tick-container`: CLI for containerized operations

**Key Modules**:
- `auto_tick.py`: Main bot job - creates PRs for migrations and version updates
- `make_graph.py`: Builds the conda-forge dependency graph
- `make_migrators.py`: Initializes migration objects
- `update_upstream_versions.py`: Fetches latest versions from upstream sources
- `update_prs.py`: Updates PR statuses from GitHub
- `feedstock_parser.py`: Parses feedstock metadata

### Migrators (`conda_forge_tick/migrators/`)

Base class: `Migration` in `core.py`. Migrators handle automated changes:
- `version.py`: Version updates (special - uses `CondaMetaYAML` parser)
- `migration_yaml.py`: CFEP-09 YAML migrations from conda-forge-pinning
- `arch.py`, `cross_compile.py`: Architecture migrations
- Custom migrators for specific ecosystem changes (libboost, numpy2, etc.)

### Data Model

The bot uses `cf-graph-countyfair` repository as its database. Key structures:
- `graph.json`: NetworkX dependency graph
- `node_attrs/`: Package metadata (one JSON per package, sharded paths)
- `versions/`: Upstream version information
- `pr_json/`: PR status tracking
- `pr_info/`, `version_pr_info/`: Migration/version PR metadata

Pydantic models in `conda_forge_tick/models/` document the schema.

### LazyJson System

Data is loaded lazily via `LazyJson` class. Backends configured via `CF_TICK_GRAPH_DATA_BACKENDS`:
- `file`: Local filesystem (default, requires cf-graph-countyfair clone)
- `github`: Read-only from GitHub raw URLs (good for debugging)
- `mongodb`: MongoDB database

### Recipe Parsing

`CondaMetaYAML` in `recipe_parser/` handles Jinja2-templated YAML recipes:
- Preserves comments (important for conda selectors)
- Handles duplicate keys with different selectors via `__###conda-selector###__` tokens
- Extracts Jinja2 variables for version migration

## Environment Variables

See `conda_forge_tick/settings.py` for full list. Key ones:
- `CF_TICK_GRAPH_DATA_BACKENDS`: Colon-separated backend list
- `CF_TICK_GRAPH_DATA_USE_FILE_CACHE`: Enable/disable local caching
- `MONGODB_CONNECTION_STRING`: MongoDB connection string
- `BOT_TOKEN`: GitHub token for bot operations
- `CF_FEEDSTOCK_OPS_IN_CONTAINER`: Set to "true" when running in container

## Bot Jobs Structure

The bot runs as multiple parallel cron jobs via GitHub Actions:
- `bot-bot.yml`: Main job making PRs
- `bot-feedstocks.yml`: Updates feedstock list
- `bot-versions.yml`: Fetches upstream versions
- `bot-prs.yml`: Updates PR statuses
- `bot-make-graph.yml`: Builds dependency graph
- `bot-make-migrators.yml`: Creates migration objects
- `bot-pypi-mapping.yml`: PyPI to conda-forge mapping

## Integration Tests

Located in `tests_integration/`. Tests the full bot pipeline against real GitHub repositories using staging accounts.

### Test Environment Architecture

The integration tests require three GitHub entities that mimic production:
- **Conda-forge org** (`GITHUB_ACCOUNT_CONDA_FORGE_ORG`): Contains test feedstocks
- **Bot user** (`GITHUB_ACCOUNT_BOT_USER`): Creates forks and PRs
- **Regro org** (`GITHUB_ACCOUNT_REGRO_ORG`): Contains a test `cf-graph-countyfair` repository

Default staging accounts are `conda-forge-bot-staging`, `regro-cf-autotick-bot-staging`, and `regro-staging`. You can use your own accounts by setting environment variables.

### Setup

1. **Initialize git submodules** (test feedstock resources are stored as submodules):
```bash
git submodule update --init --recursive
```

2. **Create a `.env` file** with required environment variables:
```bash
export BOT_TOKEN='<github-classic-pat>'
export TEST_SETUP_TOKEN='<github-classic-pat>' # typically same as BOT_TOKEN
export GITHUB_ACCOUNT_CONDA_FORGE_ORG='your-conda-forge-staging-org'
export GITHUB_ACCOUNT_BOT_USER='your-bot-user'
export GITHUB_ACCOUNT_REGRO_ORG='your-regro-staging-org'
export PROXY_DEBUG_LOGGING='true' # optional, for debugging
```

GitHub token requires scopes: `repo`, `workflow`, `delete_repo`.

3. **Set up mitmproxy certificates** (required for HTTP proxy that intercepts requests):
```bash
cd tests_integration
./mitmproxy_setup_wizard.sh
```

On macOS: Add the generated certificate to Keychain Access and set "Always Trust".
On Linux: Copy to `/usr/local/share/ca-certificates/` and run `update-ca-certificates`.

4. **Build the Docker test image** (required for container-based tests):
```bash
docker build -t conda-forge-tick:test .
```

### Running Integration Tests

**Important**: Integration tests take a long time to execute (5+ minutes per test). To avoid repeated runs:
- Persist stdout/stderr to a file and grep for errors
- Run tests in the background while working on other tasks

```bash
# Source your environment variables
source .env

# Run from repository root, skipping container tests (default)
# Recommended: redirect output to file for later analysis
pytest -s -v --dist=no tests_integration -k "False" > /tmp/integration_test.log 2>&1 &
tail -f /tmp/integration_test.log # follow output in another terminal

# Or run interactively if needed
pytest -s -v --dist=no tests_integration -k "False"

# Run only container tests (requires Docker image built with test tag)
pytest -s -v --dist=no tests_integration -k "True"

# Run a specific test scenario
pytest -s -v --dist=no tests_integration -k "test_scenario[0]"
```

### Test Case Structure

Test cases are defined in `tests_integration/lib/_definitions/<feedstock>/__init__.py`. Each test case:
1. `get_router()`: Defines mock HTTP responses via FastAPI router
2. `prepare(helper)`: Sets up test state (e.g., overwrites feedstock contents)
3. `validate(helper)`: Asserts expected outcomes (e.g., PR was created with correct changes)

Current test feedstocks: `pydantic`, `polars`, `fastapi`, `zizmor`, `conda-forge-pinning`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leave out this list, it will not be maintained


### How Tests Execute

Tests run the full bot pipeline in sequence:
1. `gather-all-feedstocks`
2. `make-graph --update-nodes-and-edges`
3. `make-graph`
4. `update-upstream-versions`
5. `make-migrators`
6. `auto-tick`
7. (repeat migrators and auto-tick for state propagation)

Each step deploys to the staging `cf-graph-countyfair` repo.
9 changes: 8 additions & 1 deletion conda_forge_tick/update_deps.py
Original file line number Diff line number Diff line change
Expand Up @@ -458,7 +458,14 @@ def _make_grayskull_recipe_v1(
is_arch=not package_is_noarch,
)
_validate_grayskull_recipe_v1(recipe=recipe)
return _generate_grayskull_recipe_v1(recipe=recipe, configuration=config)
recipe_str = _generate_grayskull_recipe_v1(recipe=recipe, configuration=config)

# Grayskull generates `match(python, ...)` for noarch recipes, but v1 recipes
# use `python_min` in their variant configs (CFEP-25). Replace to make the
# recipe renderable. See: https://github.com/conda/grayskull/issues/574
recipe_str = recipe_str.replace("match(python,", "match(python_min,")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This filter is rather broad and I am concerned it might sweep up other parts of the recipe with this syntax. I suggested we write a more targeted edit that ensures the change is in a noarch: python recipe.


return recipe_str


def get_grayskull_comparison(attrs, version_key="version"):
Expand Down
53 changes: 53 additions & 0 deletions tests/test_update_deps.py
Original file line number Diff line number Diff line change
Expand Up @@ -1146,3 +1146,56 @@ def test_jsii_package_name_resolution():
resolved_name = _modify_package_name_from_github(feedstock_package_name, src)

assert resolved_name == "jsii"


def test_get_grayskull_comparison_v1_python_min_mismatch():
"""Test that get_grayskull_comparison works for v1 recipes using python_min.

This test reproduces the issue where grayskull generates a recipe with
`skip: match(python, "<3.10")` but the feedstock's variant config only has
`python_min` (not `python`). When rattler-build tries to render the recipe,
it skips all variants because the `python` variable is not set.

See: https://github.com/conda-forge/dominodatalab-feedstock/pull/21
"""
attrs = {
"meta_yaml": {
"schema_version": 1,
"package": {
"name": "dominodatalab",
"version": "1.4.7",
},
"source": {
"url": "https://pypi.org/packages/source/d/dominodatalab/dominodatalab-1.4.7.tar.gz",
},
"build": {"noarch": "python"},
},
"feedstock_name": "dominodatalab",
"version_pr_info": {"version": "2.0.0"},
"total_requirements": {
"build": set(),
"host": {"pip", "python"},
"run": {
"python",
"packaging",
"requests >=2.4.2",
"beautifulsoup4 >=4.11,<4.12",
"polling2 >=0.5.0,<0.6",
"urllib3 >=1.26.12,<1.27",
"typing-extensions >=4.5.0",
"frozendict >=2.3.4,<2.4",
"python-dateutil >=2.8.2,<2.9",
"retry ==0.9.2",
},
"test": set(),
},
}

# This should not raise an exception, but currently it does because
# grayskull generates `skip: match(python, "<3.10")` and the variant
# config only has `python_min`, causing rattler-build to skip all variants.
dep_comparison, recipe = get_grayskull_comparison(attrs=attrs)

# If we get here, the comparison should have valid results
assert "run" in dep_comparison
assert recipe != ""
11 changes: 10 additions & 1 deletion tests_integration/lib/_definitions/__init__.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,17 @@
from . import conda_forge_pinning, fastapi, polars, pydantic, witr, zizmor
from . import (
conda_forge_pinning,
dominodatalab,
fastapi,
polars,
pydantic,
witr,
zizmor,
)
from .base_classes import AbstractIntegrationTestHelper, GitHubAccount, TestCase

TEST_CASE_MAPPING: dict[str, list[TestCase]] = {
"conda-forge-pinning": conda_forge_pinning.ALL_TEST_CASES,
"dominodatalab": dominodatalab.ALL_TEST_CASES,
"fastapi": fastapi.ALL_TEST_CASES,
"polars": polars.ALL_TEST_CASES,
"pydantic": pydantic.ALL_TEST_CASES,
Expand Down
25 changes: 25 additions & 0 deletions tests_integration/lib/_definitions/base_classes.py
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,31 @@ def assert_new_run_requirements_equal_v1(
"""
pass

def assert_pr_body_not_contains(
self,
feedstock: str,
new_version: str,
not_included: list[str],
):
"""
Assert that the version update PR body does NOT contain certain strings.

Parameters
----------
feedstock
The feedstock we expect the PR for, without the -feedstock suffix.
new_version
The new version that is expected.
not_included
A list of strings that must NOT be present in the PR body.

Raises
------
AssertionError
If any of the strings are found in the PR body.
"""
pass

def assert_pr_title_starts_with(
self,
feedstock: str,
Expand Down
Loading
Loading