Buildkite Test Engine Client

Buildkite Test Engine Client (bktec) is an open source tool to orchestrate your test suites. It uses your Buildkite Test Engine suite data to intelligently partition and parallelize your tests.

bktec supports multiple test runners and offers various features to enhance your testing workflow. Below is a comparison of the features supported by each test runner:

Feature	RSpec	Jest	Playwright	Cypress	pytest	pants (pytest)	Go test	Cucumber	NUnit	Custom
Split tests by file1	✅	✅	✅	✅	✅	❌	❌	✅	✅	✅
Split slow files by individual test example	✅	❌	❌	❌	✅	❌	❌	✅	❌	❌
Filter test files	✅	✅	✅	✅	✅	❌	❌	✅	✅	✅
Filter test by tags	❌	❌	❌	❌	✅	❌	❌	❌	❌	❌
Automatically retry failed test	✅	✅	✅	❌	✅	✅	✅	✅	✅	❌
Mute tests (ignore test failures)	✅	✅	✅	❌	✅	✅	✅	✅	✅	✅
Skip tests	✅	❌	❌	❌	❌	❌	❌	✅	❌	❌

Installation

The latest version of bktec can be downloaded from https://github.com/buildkite/test-engine-client/releases

Supported OS/Architecture

ARM and AMD architecture for linux, darwin, and windows

The available Go binaries

• bktec-darwin-amd64
• bktec-darwin-arm64
• bktec-linux-amd64
• bktec-linux-arm64
• bktec-windows-amd64.exe
• bktec-windows-arm64.exe

Using bktec

Buildkite Pipeline environment variables

bktec uses the following Buildkite Pipeline provided environment variables.

Environment Variable	Description
BUILDKITE_BUILD_ID	The UUID of the Buildkite build. bktec uses this UUID along with BUILDKITE_STEP_ID to uniquely identify the test plan.
BUILDKITE_JOB_ID	The UUID of the job in Buildkite build.
BUILDKITE_ORGANIZATION_SLUG	The slug of your Buildkite organization.
BUILDKITE_PARALLEL_JOB	The index number of a parallel job created from a Buildkite parallel build step. Make sure you configure parallelism in your pipeline definition. You can read more about Buildkite parallel build step on this page.
BUILDKITE_PARALLEL_JOB_COUNT	The total number of parallel jobs created from a Buildkite parallel build step. Make sure you configure parallelism in your pipeline definition. You can read more about Buildkite parallel build step on this page.
BUILDKITE_STEP_ID	The UUID of the step group in Buildkite build. bktec uses this UUID along with BUILDKITE_BUILD_ID to uniquely identify the test plan.

Important

Please make sure that the above environment variables are available in your testing environment, particularly if you use Docker or some other type of containerization to run your tests.

Create API access token

To use bktec, you need a Buildkite API access token with read_suites, read_test_plan, and write_test_plan scopes. You can generate this token from your Personal Settings in Buildkite. After creating the token, set the BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN environment variable with the token value.

export BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN=token

Configure Test Engine suite slug

To use bktec, you need to configure the BUILDKITE_TEST_ENGINE_SUITE_SLUG environment variable with your Test Engine suite slug. You can find the suite slug in the URL of your suite. For example, in the URL https://buildkite.com/organizations/my-organization/analytics/suites/my-suite, the slug is my-suite.

export BUILDKITE_TEST_ENGINE_SUITE_SLUG=my-slug

Preview: Test Selection

You can pass test selection strategy configuration and additional change context to the test plan API request. This preview is enabled only when BKTEC_PREVIEW_SELECTION is truthy (1, true, yes, or on). This functionality is under development, and these flags currently have undefined behavior.

Environment variables:

export BKTEC_PREVIEW_SELECTION=true
export BUILDKITE_TEST_ENGINE_SELECTION_STRATEGY=percent

Command-line flags:

BKTEC_PREVIEW_SELECTION=true ./bktec plan --json --selection-strategy percent \
  --selection-param percent=40

Automatic git metadata collection

When --selection-strategy is set, the plan command automatically collects git metadata from the current repository and sends it with the API request. This includes commit information (SHA, author, committer, message), diff data (files changed, numstat, full diff), and context fields (branch name, base branch, pipeline slug, build UUID).

For pipelines that use plan without --selection-strategy, you can opt in to metadata collection with the --collect-git-metadata flag (or BUILDKITE_TEST_ENGINE_COLLECT_GIT_METADATA=true). This collects the same git metadata without requiring selection to be configured:

BKTEC_PREVIEW_SELECTION=true ./bktec plan --json --collect-git-metadata

The base branch for diff computation is resolved using a fallback chain:

1. Explicit override via --metadata base_branch=<branch>
2. BUILDKITE_PULL_REQUEST_BASE_BRANCH (auto-set by Buildkite on PR builds)
3. Auto-detection via <remote>/HEAD, then <remote>/main, then <remote>/master

Most users don't need to configure anything. Override base_branch only if your repository uses a non-standard default branch (for example, develop or trunk) and <remote>/HEAD isn't configured.

The --remote flag (default origin) controls which git remote is used for base branch detection. You can also set BUILDKITE_TEST_ENGINE_REMOTE.

Auto-collected values are merged with any explicit --metadata flags you provide. Your explicit values always take precedence.

Manual metadata overrides

Use --metadata key=value to pass additional metadata or override auto-collected values. Use --selection-param key=value to pass strategy parameters. Both flags are repeatable. Values can be large and multiline.

BKTEC_PREVIEW_SELECTION=true ./bktec plan --json --selection-strategy percent \
  --selection-param percent=40 \
  --metadata base_branch=develop

--selection-param and --metadata are only supported as repeatable CLI flags.

Preview: Commit Metadata Backfill

bktec can collect historical git commit metadata from your repository and upload it to Buildkite for training test selection models. This is useful for bootstrapping models with historical changeset data so that test selection can identify which tests are relevant to your code changes.

The tools subcommands are hidden from bktec --help by default. Setting BKTEC_PREVIEW_SELECTION to a truthy value (1, true, yes, or on) makes them visible in help output. The commands can always be invoked directly regardless of this setting.

Two commands are available under bktec tools:

Collect and upload commit metadata:

bktec tools backfill-commit-metadata \
  --access-token "bkua_..." \
  --organization-slug "my-org" \
  --suite-slug "my-suite"

Generate the tarball locally for inspection before uploading:

bktec tools backfill-commit-metadata --output commit-metadata.tar.gz

# Inspect the contents
tar tzf commit-metadata.tar.gz
# commit-metadata.jsonl
# metadata.json

# Upload when ready
bktec tools backfill-commit-metadata --upload commit-metadata.tar.gz

The API access token requires read_suites and write_suites scopes.

For detailed usage, flags, and configuration options, see the Commit Metadata Backfill guide.

Configure the test runner

To configure the test runner for bktec, please refer to the detailed guides provided for each supported test runner. You can find the guides at the following links:

• Jest
• Playwright
• Cypress
• pytest
• pytest pants
• go test
• RSpec
• Cucumber
• NUnit
• Custom Test Runner

Running bktec

Please download the executable and make it available in your testing environment. To parallelize your tests in your Buildkite build, you can amend your pipeline step configuration to:

steps:
  - name: "Rspec"
    command: ./bktec run
    parallelism: 10
    env:
      BUILDKITE_TEST_ENGINE_SUITE_SLUG: my-suite
      BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN: your-secret-token
      BUILDKITE_TEST_ENGINE_TEST_RUNNER: rspec
      BUILDKITE_TEST_ENGINE_RESULT_PATH: tmp/result.json

Tip

You can find example configurations and usage instructions for each test runner in our examples repository.

Debugging

To enable debug mode, set the BUILDKITE_TEST_ENGINE_DEBUG_ENABLED environment variable to true. This will print detailed output to assist in debugging bktec.

Possible exit statuses

bktec may exit with a variety of exit statuses, outlined below:

• If there is a configuration error, bktec will exit with status 16.
• If the test runner (e.g. RSpec) exits cleanly, the exit status of the runner is returned. This will likely be 0 for successful test runs, 1 for failing test runs, but may be any other error status returned by the runner.
• If the test runner is terminated by an OS level signal, such as SIGSEGV or SIGABRT, the exit status returned will be equal to 128 plus the signal number. For example, if the runner raises a SIGSEGV, the exit status will be (128 + 11) = 139.

Development

Make sure you have Go, Ruby, and Node.js installed in your environment. You can follow the installation guides for each of these tools:

• Go Installation Guide
• Ruby Installation Guide
• Node.js Installation Guide

Once you have these dependencies installed, run bin/setup to install dependencies for the sample projects for testing purposes.

To test, run:

./bin/test

Footnotes

1. NB: Test splitting is not supported for Pants, because Pants will decide which tests to run. For go test, test splitting is by package and not by file. ↩