Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 11 additions & 12 deletions develop-docs/sdk/platform-specifics/python-sdk/ci.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,19 @@ description: Our test setup and how our CI files are generated.
sidebar_order: 20
---

The GitHub workflow files for testing specific integration groups (like AI, Web Frameworks, Databases, etc.) are generated by [this script](https://github.com/getsentry/sentry-python/blob/master/scripts/split-tox-gh-actions/split-tox-gh-actions.py). Essentially, the script scans our `tox.ini` and generates the group test CI YAML files in `.github/workflows`.
Our test suite consists of a common part that tests basic SDK functionality, and an integrations part that tests individual integrations.

There are multiple components to this:
* `tox.ini` is where the configuration (what to test and what versions) is read from
* `scripts/split-tox-gh-actions/split-tox-gh-actions.py` is the script that defines the [test groups](https://github.com/getsentry/sentry-python/blob/0f3e5db0c8aabcad0baf0e8b2d3e31e27e839b3e/scripts/split-tox-gh-actions/split-tox-gh-actions.py#L57) and generates the resulting YAML files
* `scripts/split-tox-gh-actions/templates` contains the jinja2 template snippets used to generate the YAML files
* `.github/workflows/test-integrations-*.yml` are the resulting workflow configuration YAMLs

If you update any of the components without keeping the rest in sync, for instance by making a change in one of the `test-integrations-*.yml` files that is not reflected in the templates in `scripts/split-tox-gh-actions/templates`, CI will error out as it actually checks if everything is in sync, meaning it runs the `split-tox-gh-actions.py` script and compares the result with the committed YAMLs.
Since the number of integrations to test is so large, the tests in CI are split into multiple groups (roughly by integration type) so that they can run in parallel. So there is, for example, [a GitHub workflow](https://github.com/getsentry/sentry-python/blob/master/.github/workflows/test-integrations-flags.yml) that runs all our feature flag integrations, and [another one](https://github.com/getsentry/sentry-python/blob/master/.github/workflows/test-integrations-graphql.yml) that runs the test suites of all GraphQL integrations, and so on. These workflow YAMLs are auto-generated by [this script](https://github.com/getsentry/sentry-python/blob/master/scripts/split-tox-gh-actions/split-tox-gh-actions.py).

## AWS Lambda Test Suite
The ultimate source of truth of what should be tested and on which versions (both Python versions as well as package versions) is stored in [`tox.ini`](https://github.com/getsentry/sentry-python/blob/master/tox.ini). This file is also auto-generated each week by a GitHub cron job that scans PyPI for new releases and updates the test matrix with them. The script that does this lives in [`scripts/populate_tox`](https://github.com/getsentry/sentry-python/tree/master/scripts/populate_tox) and is often referred to as "toxgen".

The AWS Lambda test suite will fail by default on new PRs. If you are an external contributor, this is not something you have to worry about, unless you've made changes actually affecting the AWS Lambda integration. It's a security precaution on our part.
## The Big Picture

Sensitive test suites (currently only AWS Lambda) require maintainer review to ensure that tests do not compromise our secrets. This review must be repeated after any code revisions.
There are multiple components to the test setup that runs in CI:
* on a weekly basis, [`scripts/populate_tox`](https://github.com/getsentry/sentry-python/tree/master/scripts/populate_tox) creates a PR that updates `tox.ini` with new releases
* `tox.ini` is where the configuration (what to test and what versions) is stored and read from
* `scripts/split-tox-gh-actions/split-tox-gh-actions.py` is the script that defines the [test groups](https://github.com/getsentry/sentry-python/blob/0f3e5db0c8aabcad0baf0e8b2d3e31e27e839b3e/scripts/split-tox-gh-actions/split-tox-gh-actions.py#L57) and generates the resulting YAML files
* `scripts/split-tox-gh-actions/templates` contains the jinja2 template snippets used to generate the YAML files
* `.github/workflows/test-integrations-*.yml` are the resulting workflow configuration YAMLs

Before running sensitive test suites, maintainers need to carefully check the PR. Then they will apply the `Trigger: tests using secrets` label. The label will be removed after any code changes to enforce our policy requiring maintainers to review all code revisions before running sensitive tests.
If you update any of the components without keeping the rest in sync, for instance by making a change in one of the `test-integrations-*.yml` files that is not reflected in the templates in `scripts/split-tox-gh-actions/templates`, they will be overwritten next time they're auto-generated.