From 28202a64f6242232d38038f644fed362b55b96bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Sun, 30 Nov 2025 00:11:49 +0000 Subject: [PATCH 01/18] Add Documenter.jl documentation Co-authored-by: Claude --- .gitignore | 3 +- docs/Project.toml | 9 ++ docs/make.jl | 25 ++++++ docs/src/advanced.md | 200 +++++++++++++++++++++++++++++++++++++++++++ docs/src/api.md | 46 ++++++++++ docs/src/index.md | 145 +++++++++++++++++++++++++++++++ 6 files changed, 427 insertions(+), 1 deletion(-) create mode 100644 docs/Project.toml create mode 100644 docs/make.jl create mode 100644 docs/src/advanced.md create mode 100644 docs/src/api.md create mode 100644 docs/src/index.md diff --git a/.gitignore b/.gitignore index 2251642..6e24bde 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ -Manifest.toml \ No newline at end of file +Manifest.toml +/docs/build/ diff --git a/docs/Project.toml b/docs/Project.toml new file mode 100644 index 0000000..d5fab23 --- /dev/null +++ b/docs/Project.toml @@ -0,0 +1,9 @@ +[deps] +Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4" +ParallelTestRunner = "d3525ed8-44d0-4b2c-a655-542cee43accc" + +[compat] +Documenter = "1" + +[sources] +ParallelTestRunner = {path = ".."} diff --git a/docs/make.jl b/docs/make.jl new file mode 100644 index 0000000..b26593d --- /dev/null +++ b/docs/make.jl @@ -0,0 +1,25 @@ +using Documenter +using ParallelTestRunner + +makedocs(; + modules=[ParallelTestRunner], + authors="Valentin Churavy and contributors", + repo="https://github.com/JuliaTesting/ParallelTestRunner.jl/blob/{commit}{path}#{line}", + sitename="ParallelTestRunner.jl", + format=Documenter.HTML(; + prettyurls=get(ENV, "CI", "false") == "true", + canonical="https://juliatesting.github.io/ParallelTestRunner.jl", + assets=String[], + ), + pages=[ + "Home" => "index.md", + "Advanced Usage" => "advanced.md", + "API Reference" => "api.md", + ], +) + +deploydocs(; + repo="github.com/JuliaTesting/ParallelTestRunner.jl", + devbranch="main", +) + diff --git a/docs/src/advanced.md b/docs/src/advanced.md new file mode 100644 index 0000000..1160085 --- /dev/null +++ b/docs/src/advanced.md @@ -0,0 +1,200 @@ +# Advanced Usage + +```@meta +CurrentModule = ParallelTestRunner +DocTestSetup = quote + using ParallelTestRunner +end +``` + +This page covers advanced features of `ParallelTestRunner` for customizing test execution. + +## Custom Test Suites + +By default, `runtests` automatically discovers all `.jl` files in your test directory. You can provide a custom test suite dictionary to have full control over which tests run: + +```julia +using ParallelTestRunner + +# Manually define your test suite +testsuite = Dict( + "basic" => quote + include("basic.jl") + end, + "advanced" => quote + include("advanced.jl") + end +) + +runtests(MyPackage, ARGS; testsuite) +``` + +Each value in the dictionary should be an expression (use `quote...end`) that executes the test code. + +## Filtering Tests + +You can use `find_tests` to automatically discover tests and then filter or modify them: + +```julia +using ParallelTestRunner + +# Start with autodiscovered tests +testsuite = find_tests(pwd()) + +# Parse arguments manually +args = parse_args(ARGS) + +# Filter based on arguments +if filter_tests!(testsuite, args) + # Additional filtering is allowed + # For example, remove platform-specific tests + if Sys.iswindows() + delete!(testsuite, "unix_only_test") + end +end + +runtests(MyPackage, args; testsuite) +``` + +The `filter_tests!` function returns `true` if no positional arguments were provided (allowing additional filtering) and `false` if the user specified specific tests (preventing further filtering). + +## Initialization Code + +Use the `init_code` keyword argument to provide code that runs before each test file. This is useful for: +- Importing packages +- Defining constants or helper functions +- Setting up test infrastructure + +```julia +using ParallelTestRunner + +const init_code = quote + using Test + using MyPackage + + # Define a helper function available to all tests + function test_helper(x) + return x * 2 + end +end + +runtests(MyPackage, ARGS; init_code) +``` + +The `init_code` is evaluated in each test's sandbox module, so all definitions are available to your test files. + +## Custom Workers + +For tests that require specific environment variables or Julia flags, you can use the `test_worker` keyword argument to assign tests to custom workers: + +```julia +using ParallelTestRunner + +function test_worker(name) + if name == "needs_env_var" + # Create a worker with a specific environment variable + return addworker(; env = ["SPECIAL_ENV_VAR" => "42"]) + elseif name == "needs_threads" + # Create a worker with multiple threads + return addworker(; exeflags = ["--threads=4"]) + end + # Return nothing to use the default worker + return nothing +end + +testsuite = Dict( + "needs_env_var" => quote + @test ENV["SPECIAL_ENV_VAR"] == "42" + end, + "needs_threads" => quote + @test Base.Threads.nthreads() == 4 + end, + "normal_test" => quote + @test 1 + 1 == 2 + end +) + +runtests(MyPackage, ARGS; test_worker, testsuite) +``` + +The `test_worker` function receives the test name and should return either: +- A worker object (from `addworker`) for tests that need special configuration +- `nothing` to use the default worker pool + +## Custom Output Streams + +You can redirect output to custom I/O streams: + +```julia +using ParallelTestRunner + +io = IOBuffer() +runtests(MyPackage, ARGS; stdout=io, stderr=io) + +# Process the output +output = String(take!(io)) +``` + +This is useful for: +- Capturing test output for analysis +- Writing to log files +- Suppressing output in certain contexts + +## Custom Arguments + +If your package needs to accept its own command-line arguments in addition to `ParallelTestRunner`'s options, use `parse_args` with custom flags: + +```julia +using ParallelTestRunner + +# Parse arguments with custom flags +args = parse_args(ARGS; custom=["myflag", "another-flag"]) + +# Access custom flags +if args.custom["myflag"] !== nothing + println("Custom flag was set!") +end + +# Pass parsed args to runtests +runtests(MyPackage, args) +``` + +Custom flags are stored in the `custom` field of the `ParsedArgs` object, with values of `nothing` (not set) or `Some(value)` (set, with optional value). + +## Manual Worker Management + +For advanced use cases, you can manually create workers: + +```julia +using ParallelTestRunner + +# Add a single worker with custom configuration +worker = addworker( + env = ["CUSTOM_VAR" => "value"], + exeflags = ["--check-bounds=no"] +) + +# Add multiple workers +workers = addworkers(4; env = ["THREADS" => "1"]) +``` + +Workers created this way can be used with the `test_worker` function or for other distributed computing tasks. + +## Best Practices + +1. **Keep tests isolated**: Each test file runs in its own module, so avoid relying on global state between tests. + +2. **Use `init_code` for common setup**: Instead of duplicating setup code in each test file, use `init_code` to share common initialization. + +3. **Filter tests appropriately**: Use `filter_tests!` to respect user-specified test filters while allowing additional programmatic filtering. + +4. **Handle platform differences**: Use conditional logic in your test suite setup to handle platform-specific tests: + + ```julia + testsuite = find_tests(pwd()) + if Sys.iswindows() + delete!(testsuite, "unix_specific_test") + end + ``` + +5. **Use custom workers sparingly**: Custom workers add overhead. Only use them when tests genuinely require different configurations. diff --git a/docs/src/api.md b/docs/src/api.md new file mode 100644 index 0000000..9f7b8aa --- /dev/null +++ b/docs/src/api.md @@ -0,0 +1,46 @@ +# API Reference + +```@meta +CurrentModule = ParallelTestRunner +DocTestSetup = quote + using ParallelTestRunner +end +``` + +## Main Functions + +```@docs +runtests +``` + +## Test Discovery + +```@docs +find_tests +``` + +## Argument Parsing + +```@docs +parse_args +filter_tests! +``` + +## Worker Management + +```@docs +addworker +addworkers +``` + +## Configuration + +```@docs +default_njobs +``` + +## Internal Types + +```@docs +WorkerTestSet +``` diff --git a/docs/src/index.md b/docs/src/index.md new file mode 100644 index 0000000..ecd2989 --- /dev/null +++ b/docs/src/index.md @@ -0,0 +1,145 @@ +# ParallelTestRunner.jl + +```@meta +CurrentModule = ParallelTestRunner +DocTestSetup = quote + using ParallelTestRunner +end +``` + +[ParallelTestRunner.jl](https://github.com/JuliaTesting/ParallelTestRunner.jl) is a simple parallel test runner for Julia tests with automatic test discovery. +It runs each test file concurrently in isolated worker processes, providing real-time progress output and efficient resource management. + +## Quick Start + +### Basic Setup + +1. **Remove existing `include` statements** from your test files. + `ParallelTestRunner` will automatically discover and run all test files. + +2. **Update your `test/runtests.jl`**: + + ```julia + using MyPackage + using ParallelTestRunner + + runtests(MyPackage, ARGS) + ``` + +That's it! `ParallelTestRunner` will automatically: +- Discover all `.jl` files in your `test/` directory (excluding `runtests.jl`) +- Run them in parallel across multiple worker processes +- Display real-time progress with timing and memory statistics + +### Running Tests + +Run tests using the standard Julia package testing interface: + +```bash +julia --project -e 'using Pkg; Pkg.test("MyPackage")' +``` + +Or from within Julia: + +```julia +using Pkg +Pkg.test("MyPackage") +``` + +## Command Line Options + +You can pass various options to control test execution: + +```bash +julia --project test/runtests.jl [OPTIONS] [TESTS...] +``` + +### Available Options + +- `--help`: Show usage information and exit +- `--list`: List all available test files and exit +- `--verbose`: Print more detailed information during test execution (including start times for each test) +- `--quickfail`: Stop the entire test run as soon as any test fails +- `--jobs=N`: Use `N` worker processes (default: based on CPU threads and available memory) +- `TESTS...`: Filter tests by name using prefix matching + +### Examples + +```bash +# List all available tests +julia --project test/runtests.jl --list + +# Run only tests matching "integration" +julia --project test/runtests.jl integration + +# Run with verbose output and 4 workers +julia --project test/runtests.jl --verbose --jobs=4 + +# Run with quick-fail enabled +julia --project test/runtests.jl --quickfail +``` + +### Using with Pkg.test + +You can also pass arguments through `Pkg.test`: + +```julia +using Pkg +Pkg.test("MyPackage"; test_args=`--verbose --jobs=4 integration`) +``` + +## Features + +### Automatic Test Discovery + +`ParallelTestRunner` automatically discovers all `.jl` files in your `test/` directory and subdirectories, excluding `runtests.jl`. +Tests are sorted by file size (largest first) for optimal load balancing. + +### Parallel Execution + +Tests run concurrently in isolated worker processes, each with: +- Its own sandbox module +- Independent memory space +- Automatic resource cleanup + +### Real-time Progress + +The test runner provides real-time output showing: +- Test name and worker assignment +- Execution time +- GC time and percentage +- Memory allocation +- RSS (Resident Set Size) memory usage + +Example output: + +``` +Test | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) | +basic (1) | 0.12 | 0.01 | 8.3 | 5.23 | 125.45 | +integration (2) | 2.45 | 0.15 | 6.1 | 45.67 | 234.12 | +``` + +### Graceful Interruption + +Press `Ctrl+C` to interrupt the test run. The framework will: +- Clean up running tests +- Display a summary of completed tests +- Exit gracefully + +You can also press `?` during execution to see which tests are currently running. + +## Test File Structure + +Your test files should be standard Julia test files using the `Test` standard library: + +```julia +using Test +using MyPackage + +@testset "MyPackage tests" begin + @test 1 + 1 == 2 + @test MyPackage.my_function(42) == 84 +end +``` + +Each test file runs in its own isolated module, so you don't need to worry about test pollution between files. From c1ed9b938dfbf21fc8a30b3b7f84488f942243f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 18:11:37 +0000 Subject: [PATCH 02/18] Add workflow for deploying the docs --- .github/workflows/Documentation.yml | 117 ++++++++++++++++++++++++++++ docs/deploy.jl | 9 +++ docs/make.jl | 6 -- 3 files changed, 126 insertions(+), 6 deletions(-) create mode 100644 .github/workflows/Documentation.yml create mode 100644 docs/deploy.jl diff --git a/.github/workflows/Documentation.yml b/.github/workflows/Documentation.yml new file mode 100644 index 0000000..133b53c --- /dev/null +++ b/.github/workflows/Documentation.yml @@ -0,0 +1,117 @@ +name: Documentation + +on: + push: + branches: + - main + tags: "*" + paths: + - ".github/workflows/Documentation.yml" + - "docs/**" + - "src/**" + - "Project.toml" + pull_request: + paths: + - ".github/workflows/Documentation.yml" + - "docs/**" + - "src/**" + - "Project.toml" + release: + +concurrency: + # Skip intermediate builds: always. + # Cancel intermediate builds: always. + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build-docs: + permissions: + actions: write + contents: write + pull-requests: read + statuses: write + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v6 + - uses: julia-actions/setup-julia@v2 + with: + version: "1.12" + - uses: julia-actions/cache@v2 + id: julia-cache + - name: Instantiate docs environment + shell: julia --color=yes --project=docs {0} + run: | + using Pkg + Pkg.instantiate() + - name: Build documentation + run: + julia --color=yes --project=docs docs/make.jl + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + - name: Save Julia depot cache on cancel or failure + id: julia-cache-save + if: cancelled() || failure() + uses: actions/cache/save@v4 + with: + path: | + ${{ steps.julia-cache.outputs.cache-paths }} + key: ${{ steps.julia-cache.outputs.cache-key }} + - uses: actions/upload-artifact@v5 + with: + name: documentation-build + path: docs/build + retention-days: 10 + + deploy-docs: + # Deploy docs only if triggers is not a PR, or a PR not from a fork. + if: ${{ (github.event_name != 'pull_request') || (! github.event.pull_request.head.repo.fork) }} + needs: build-docs + permissions: + actions: write + contents: write + pull-requests: read + statuses: write + runs-on: ubuntu-latest + timeout-minutes: 30 + concurrency: + # Have only one job pushing the docs at a time. + group: docs-pushing + steps: + - uses: actions/checkout@v6 + - uses: julia-actions/setup-julia@v2 + with: + version: "1.12" + - uses: julia-actions/cache@v2 + id: julia-cache + - uses: actions/download-artifact@v6 + with: + name: documentation-build + path: docs/build + - name: Remove extra files + # There may be JLD2 files generated during execution of Literate + # examples, we don't need them on the website. + run: + rm -fv docs/build/literated/*.jld2 + - name: Instantiate docs environment + shell: julia --color=yes {0} + run: | + # We only need `Documenter` for publishing the docs, let's not + # reinstall the world all over again. + using Pkg + Pkg.add(; name="Documenter", version="1") + - name: Deploy documentation + run: + julia --color=yes docs/deploy.jl + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }} + - name: Save Julia depot cache on cancel or failure + id: julia-cache-save + if: cancelled() || failure() + uses: actions/cache/save@v4 + with: + path: | + ${{ steps.julia-cache.outputs.cache-paths }} + key: ${{ steps.julia-cache.outputs.cache-key }} diff --git a/docs/deploy.jl b/docs/deploy.jl new file mode 100644 index 0000000..86c2031 --- /dev/null +++ b/docs/deploy.jl @@ -0,0 +1,9 @@ +using Documenter + +deploydocs(; + repo="github.com/JuliaTesting/ParallelTestRunner.jl", + devbranch="main", + # Only push previews if all the relevant environment variables are non-empty. This is an + # attempt to work around https://github.com/JuliaDocs/Documenter.jl/issues/2048. + push_preview = all(!isempty, (get(ENV, "GITHUB_TOKEN", ""), get(ENV, "DOCUMENTER_KEY", ""))), +) diff --git a/docs/make.jl b/docs/make.jl index b26593d..e91cc2a 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -17,9 +17,3 @@ makedocs(; "API Reference" => "api.md", ], ) - -deploydocs(; - repo="github.com/JuliaTesting/ParallelTestRunner.jl", - devbranch="main", -) - From 02ba9460b036e26b9097cd5c8394d95a28444902 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 18:54:45 +0000 Subject: [PATCH 03/18] Some fixes and move information from README to the docs --- README.md | 100 +++---------------------------------------- docs/src/advanced.md | 74 ++++++++++++++++++++++---------- docs/src/index.md | 23 ++++++++-- 3 files changed, 76 insertions(+), 121 deletions(-) diff --git a/README.md b/README.md index 0e923a8..dccb64f 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,8 @@ # ParallelTestRunner.jl +[![Stable Documentation](https://img.shields.io/badge/docs-stable-blue.svg)](https://juliatesting.github.io/ParallelTestRunner.jl/) +[![Latest Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://juliatesting.github.io/ParallelTestRunner.jl/dev) + Simple parallel test runner for Julia tests with autodiscovery. ## Usage @@ -40,102 +43,9 @@ using ParallelTestRunner runtests(MyModule, ARGS) ``` -### Customizing the test suite - -By default, `runtests` automatically discovers all `.jl` files in your `test/` directory (excluding `runtests.jl` itself) using the `find_tests` function. You can customize which tests to run by providing a custom `testsuite` dictionary: - -```julia -# Manually define your test suite -testsuite = Dict( - "basic" => quote - include("basic.jl") - end, - "advanced" => quote - include("advanced.jl") - end -) - -runtests(MyModule, ARGS; testsuite) -``` - -You can also use `find_tests` to automatically discover tests and then filter or modify them. This requires manually parsing arguments so that filtering is only applied when the user did not request specific tests to run: - -```julia -# Start with autodiscovered tests -testsuite = find_tests(pwd()) - -# Parse arguments -args = parse_args(ARGS) - -if filter_tests!(testsuite, args) - # Remove tests that shouldn't run on Windows - if Sys.iswindows() - delete!(testsuite, "ext/specialfunctions") - end -end - -runtests(MyModule, args; testsuite) -``` - -### Provide defaults - -`runtests` takes a keyword argument that one can use to provide default definitions to be loaded before each testfile. -As an example one could always load `Test` and the package under test. - -```julia -const init_code = quote - using Test - using MyPackage -end - -runtests(MyModule, ARGS; init_code) -``` - -### Interactive use - -Arguments can also be passed via the standard `Pkg.test` interface for interactive control. For example, here is how we could run the subset of tests that start with the testset name "MyTestsetA" in i) verbose mode, and ii) with default threading enabled: - -```julia-repl -# In an environment where `MyPackage.jl` is available -julia --proj - -julia> using Pkg - -# No need to start a fresh session to change threading -julia> Pkg.test("MyModule"; test_args=`--verbose MyTestsetA`, julia_args=`--threads=auto`); -``` -Alternatively, arguments can be passed directly from the command line with a shell alias like the one below: - -```julia-repl -jltest --threads=auto -- --verbose MyTestsetA -``` - -
Shell alias - -```shell -function jltest { - julia=(julia) - - # certain arguments (like those beginnning with a +) need to come first - if [[ $# -gt 0 && "$1" = +* ]]; then - julia+=("$1") - shift - fi - - "${julia[@]}" --startup-file=no --project -e "using Pkg; Pkg.API.test(; test_args=ARGS)" "$@" -} -``` - -
- -## Packages using ParallelTestRunner.jl - -There are a few packages already using `ParallelTestRunner.jl` to parallelize their tests, you can look at their setups if you need inspiration to move your packages as well: +## Documentation -* [`Enzyme.jl`](https://github.com/EnzymeAD/Enzyme.jl/blob/main/test/runtests.jl) -* [`GPUArrays.jl`](https://github.com/JuliaGPU/GPUArrays.jl/blob/master/test/runtests.jl) -* [`GPUCompiler.jl`](https://github.com/JuliaGPU/GPUCompiler.jl/blob/master/test/runtests.jl) -* [`Metal.jl`](https://github.com/JuliaGPU/Metal.jl/blob/main/test/runtests.jl) +For more details about the use of this package, read the [documentation](https://juliatesting.github.io/ParallelTestRunner.jl/). ## Inspiration Based on [@maleadt](https://github.com/maleadt) test infrastructure for [CUDA.jl](https://github.com/JuliaGPU/CUDA.jl). diff --git a/docs/src/advanced.md b/docs/src/advanced.md index 1160085..9970b56 100644 --- a/docs/src/advanced.md +++ b/docs/src/advanced.md @@ -9,13 +9,12 @@ end This page covers advanced features of `ParallelTestRunner` for customizing test execution. -## Custom Test Suites +## Customizing the test suite -By default, `runtests` automatically discovers all `.jl` files in your test directory. You can provide a custom test suite dictionary to have full control over which tests run: +By default, [`runtests`](@ref) automatically discovers all `.jl` files in your `test/` directory (excluding `runtests.jl` itself) using the `find_tests` function. +You can customize which tests to run by providing a custom `testsuite` dictionary: ```julia -using ParallelTestRunner - # Manually define your test suite testsuite = Dict( "basic" => quote @@ -26,43 +25,39 @@ testsuite = Dict( end ) -runtests(MyPackage, ARGS; testsuite) +runtests(MyModule, ARGS; testsuite) ``` -Each value in the dictionary should be an expression (use `quote...end`) that executes the test code. - -## Filtering Tests +## Filtering Test Files -You can use `find_tests` to automatically discover tests and then filter or modify them: +You can also use [`find_tests`](@ref) to automatically discover test files and then filter or modify them. +This requires manually parsing arguments so that filtering is only applied when the user did not request specific tests to run: ```julia -using ParallelTestRunner - # Start with autodiscovered tests testsuite = find_tests(pwd()) -# Parse arguments manually +# Parse arguments args = parse_args(ARGS) -# Filter based on arguments if filter_tests!(testsuite, args) - # Additional filtering is allowed - # For example, remove platform-specific tests + # Remove tests that shouldn't run on Windows if Sys.iswindows() - delete!(testsuite, "unix_only_test") + delete!(testsuite, "ext/specialfunctions") end end -runtests(MyPackage, args; testsuite) +runtests(MyModule, args; testsuite) ``` -The `filter_tests!` function returns `true` if no positional arguments were provided (allowing additional filtering) and `false` if the user specified specific tests (preventing further filtering). +The [`filter_tests!`](@ref) function returns `true` if no positional arguments were provided (allowing additional filtering) and `false` if the user specified specific tests (preventing further filtering). ## Initialization Code -Use the `init_code` keyword argument to provide code that runs before each test file. This is useful for: +Use the `init_code` keyword argument to [`runtests`](@ref) to provide code that runs before each test file. +This is useful for: - Importing packages -- Defining constants or helper functions +- Defining constants, defaults or helper functions - Setting up test infrastructure ```julia @@ -118,7 +113,7 @@ runtests(MyPackage, ARGS; test_worker, testsuite) ``` The `test_worker` function receives the test name and should return either: -- A worker object (from `addworker`) for tests that need special configuration +- A worker object (from [`addworker`](@ref) for tests that need special configuration - `nothing` to use the default worker pool ## Custom Output Streams @@ -142,7 +137,7 @@ This is useful for: ## Custom Arguments -If your package needs to accept its own command-line arguments in addition to `ParallelTestRunner`'s options, use `parse_args` with custom flags: +If your package needs to accept its own command-line arguments in addition to `ParallelTestRunner`'s options, use [`parse_args`](@ref) with custom flags: ```julia using ParallelTestRunner @@ -180,6 +175,41 @@ workers = addworkers(4; env = ["THREADS" => "1"]) Workers created this way can be used with the `test_worker` function or for other distributed computing tasks. +### Interactive use + +Arguments can also be passed via the standard `Pkg.test` interface for interactive control. For example, here is how we could run the subset of tests that start with the testset name "MyTestsetA" in i) verbose mode, and ii) with default threading enabled: + +```julia-repl +# In an environment where `MyPackage.jl` is available +julia --proj + +julia> using Pkg + +# No need to start a fresh session to change threading +julia> Pkg.test("MyModule"; test_args=`--verbose MyTestsetA`, julia_args=`--threads=auto`); +``` +Alternatively, arguments can be passed directly from the command line with a shell alias like the one below: + +```julia-repl +jltest --threads=auto -- --verbose MyTestsetA +``` + +Shell alias: + +```shell +function jltest { + julia=(julia) + + # certain arguments (like those beginnning with a +) need to come first + if [[ $# -gt 0 && "$1" = +* ]]; then + julia+=("$1") + shift + fi + + "${julia[@]}" --startup-file=no --project -e "using Pkg; Pkg.API.test(; test_args=ARGS)" "$@" +} +``` + ## Best Practices 1. **Keep tests isolated**: Each test file runs in its own module, so avoid relying on global state between tests. diff --git a/docs/src/index.md b/docs/src/index.md index ecd2989..d85d91e 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -115,8 +115,8 @@ Example output: ``` Test | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) | -basic (1) | 0.12 | 0.01 | 8.3 | 5.23 | 125.45 | -integration (2) | 2.45 | 0.15 | 6.1 | 45.67 | 234.12 | +basic (1) | 0.12 | 0.01 | 8.3 | 5.23 | 125.45 | +integration (2) | 2.45 | 0.15 | 6.1 | 45.67 | 234.12 | ``` ### Graceful Interruption @@ -126,8 +126,6 @@ Press `Ctrl+C` to interrupt the test run. The framework will: - Display a summary of completed tests - Exit gracefully -You can also press `?` during execution to see which tests are currently running. - ## Test File Structure Your test files should be standard Julia test files using the `Test` standard library: @@ -143,3 +141,20 @@ end ``` Each test file runs in its own isolated module, so you don't need to worry about test pollution between files. + +## Packages using ParallelTestRunner.jl + +There are a few packages already [using `ParallelTestRunner.jl`](https://github.com/search?q=%22using+ParallelTestRunner%22+language%3AJulia++NOT+is%3Aarchived+NOT+is%3Afork&type=code) to parallelize their tests, you can look at their setups if you need inspiration to move your packages as well: + +* [`ApproxFun.jl`](https://github.com/JuliaApproximation/ApproxFun.jl/blob/master/test/runtests.jl) +* [`BlockArrays.jl`](https://github.com/JuliaArrays/BlockArrays.jl/blob/master/test/runtests.jl) +* [`CuNESSie.jl`](https://github.com/tkemmer/CuNESSie.jl/blob/master/test/runtests.jl) +* [`Enzyme.jl`](https://github.com/EnzymeAD/Enzyme.jl/blob/main/test/runtests.jl) +* [`GPUArrays.jl`](https://github.com/JuliaGPU/GPUArrays.jl/blob/master/test/runtests.jl) +* [`GPUCompiler.jl`](https://github.com/JuliaGPU/GPUCompiler.jl/blob/master/test/runtests.jl) +* [`HyperHessians.jl`](https://github.com/KristofferC/HyperHessians.jl/blob/master/test/runtests.jl) +* [`Metal.jl`](https://github.com/JuliaGPU/Metal.jl/blob/main/test/runtests.jl) +* [`WCS.jl`](https://github.com/JuliaAstro/WCS.jl/blob/master/test/runtests.jl) + +## Inspiration +Based on [@maleadt](https://github.com/maleadt) test infrastructure for [CUDA.jl](https://github.com/JuliaGPU/CUDA.jl). From dbfe8d2a157f57dc059c2bc2d6b51b05e6f0cf59 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 19:21:01 +0000 Subject: [PATCH 04/18] More tips --- docs/src/advanced.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/src/advanced.md b/docs/src/advanced.md index 9970b56..bbe749c 100644 --- a/docs/src/advanced.md +++ b/docs/src/advanced.md @@ -214,11 +214,11 @@ function jltest { 1. **Keep tests isolated**: Each test file runs in its own module, so avoid relying on global state between tests. -2. **Use `init_code` for common setup**: Instead of duplicating setup code in each test file, use `init_code` to share common initialization. +1. **Use `init_code` for common setup**: Instead of duplicating setup code in each test file, use `init_code` to share common initialization. -3. **Filter tests appropriately**: Use `filter_tests!` to respect user-specified test filters while allowing additional programmatic filtering. +1. **Filter tests appropriately**: Use [`filter_tests!`](@ref) to respect user-specified test filters while allowing additional programmatic filtering. -4. **Handle platform differences**: Use conditional logic in your test suite setup to handle platform-specific tests: +1. **Handle platform differences**: Use conditional logic in your test suite setup to handle platform-specific tests: ```julia testsuite = find_tests(pwd()) @@ -227,4 +227,7 @@ function jltest { end ``` -5. **Use custom workers sparingly**: Custom workers add overhead. Only use them when tests genuinely require different configurations. +1. **Load balance the test files**: `ParallelTestRunner` runs the tests files in parallel, ideally all test files should run for _roughly_ the same time for better performance. + Having few long-running test files and other short-running ones hinders scalability. + +1. **Use custom workers sparingly**: Custom workers add overhead. Only use them when tests genuinely require different configurations. From 553c6c6f497d7f0dfdce047965c8d5564daadf73 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 19:35:48 +0000 Subject: [PATCH 05/18] Remove non-sense --- docs/src/index.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/src/index.md b/docs/src/index.md index d85d91e..75a4d4d 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -93,14 +93,11 @@ Pkg.test("MyPackage"; test_args=`--verbose --jobs=4 integration`) ### Automatic Test Discovery `ParallelTestRunner` automatically discovers all `.jl` files in your `test/` directory and subdirectories, excluding `runtests.jl`. -Tests are sorted by file size (largest first) for optimal load balancing. ### Parallel Execution -Tests run concurrently in isolated worker processes, each with: -- Its own sandbox module -- Independent memory space -- Automatic resource cleanup +Tests run concurrently in isolated worker processes, each inside own module. +`ParallelTestRunner` records historical tests duration for each package, so that in subsequent runs long-running tests are executed first, to improve load balancing. ### Real-time Progress From 731044ca8a0e5c21884bd0953e2e9d0d21017d9e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 19:43:00 +0000 Subject: [PATCH 06/18] Remove custom output streams section --- docs/src/advanced.md | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/docs/src/advanced.md b/docs/src/advanced.md index bbe749c..2e85ba7 100644 --- a/docs/src/advanced.md +++ b/docs/src/advanced.md @@ -116,25 +116,6 @@ The `test_worker` function receives the test name and should return either: - A worker object (from [`addworker`](@ref) for tests that need special configuration - `nothing` to use the default worker pool -## Custom Output Streams - -You can redirect output to custom I/O streams: - -```julia -using ParallelTestRunner - -io = IOBuffer() -runtests(MyPackage, ARGS; stdout=io, stderr=io) - -# Process the output -output = String(take!(io)) -``` - -This is useful for: -- Capturing test output for analysis -- Writing to log files -- Suppressing output in certain contexts - ## Custom Arguments If your package needs to accept its own command-line arguments in addition to `ParallelTestRunner`'s options, use [`parse_args`](@ref) with custom flags: From 721f1ad9d78c4e8cf28eaf7b4a7ffaa4d6384dd1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 19:49:47 +0000 Subject: [PATCH 07/18] Remove redundant section about custom workers --- docs/src/advanced.md | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-) diff --git a/docs/src/advanced.md b/docs/src/advanced.md index 2e85ba7..f5c1b44 100644 --- a/docs/src/advanced.md +++ b/docs/src/advanced.md @@ -80,7 +80,7 @@ The `init_code` is evaluated in each test's sandbox module, so all definitions a ## Custom Workers -For tests that require specific environment variables or Julia flags, you can use the `test_worker` keyword argument to assign tests to custom workers: +For tests that require specific environment variables or Julia flags, you can use the `test_worker` keyword argument to [`runtests`](@ref) to assign tests to custom workers: ```julia using ParallelTestRunner @@ -137,25 +137,6 @@ runtests(MyPackage, args) Custom flags are stored in the `custom` field of the `ParsedArgs` object, with values of `nothing` (not set) or `Some(value)` (set, with optional value). -## Manual Worker Management - -For advanced use cases, you can manually create workers: - -```julia -using ParallelTestRunner - -# Add a single worker with custom configuration -worker = addworker( - env = ["CUSTOM_VAR" => "value"], - exeflags = ["--check-bounds=no"] -) - -# Add multiple workers -workers = addworkers(4; env = ["THREADS" => "1"]) -``` - -Workers created this way can be used with the `test_worker` function or for other distributed computing tasks. - ### Interactive use Arguments can also be passed via the standard `Pkg.test` interface for interactive control. For example, here is how we could run the subset of tests that start with the testset name "MyTestsetA" in i) verbose mode, and ii) with default threading enabled: From 241b9c204628e9349590669e39c5abe4f7c289c4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 19:53:52 +0000 Subject: [PATCH 08/18] Min fixes --- docs/src/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/src/index.md b/docs/src/index.md index 75a4d4d..3d04cf3 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -48,7 +48,7 @@ Pkg.test("MyPackage") ## Command Line Options -You can pass various options to control test execution: +You can pass various options to the `runtests.jl` script to control test execution: ```bash julia --project test/runtests.jl [OPTIONS] [TESTS...] @@ -90,7 +90,7 @@ Pkg.test("MyPackage"; test_args=`--verbose --jobs=4 integration`) ## Features -### Automatic Test Discovery +### Automatic Test Files Discovery `ParallelTestRunner` automatically discovers all `.jl` files in your `test/` directory and subdirectories, excluding `runtests.jl`. From 5871997b238811ac834f9207ccf4ec33b75a5d5a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 19:59:14 +0000 Subject: [PATCH 09/18] Fix sectioning level --- docs/src/advanced.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/advanced.md b/docs/src/advanced.md index f5c1b44..fa0130a 100644 --- a/docs/src/advanced.md +++ b/docs/src/advanced.md @@ -137,7 +137,7 @@ runtests(MyPackage, args) Custom flags are stored in the `custom` field of the `ParsedArgs` object, with values of `nothing` (not set) or `Some(value)` (set, with optional value). -### Interactive use +## Interactive use Arguments can also be passed via the standard `Pkg.test` interface for interactive control. For example, here is how we could run the subset of tests that start with the testset name "MyTestsetA" in i) verbose mode, and ii) with default threading enabled: From cd38a324b7100442a8cc7d43dee43de4140e4ce0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 19:57:52 +0000 Subject: [PATCH 10/18] Simplify docs deploy job --- .github/workflows/Documentation.yml | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/.github/workflows/Documentation.yml b/.github/workflows/Documentation.yml index 133b53c..a12d6cf 100644 --- a/.github/workflows/Documentation.yml +++ b/.github/workflows/Documentation.yml @@ -37,9 +37,11 @@ jobs: - uses: actions/checkout@v6 - uses: julia-actions/setup-julia@v2 with: - version: "1.12" + version: "1" - uses: julia-actions/cache@v2 id: julia-cache + with: + cache-name: Documentation - name: Instantiate docs environment shell: julia --color=yes --project=docs {0} run: | @@ -82,18 +84,15 @@ jobs: - uses: actions/checkout@v6 - uses: julia-actions/setup-julia@v2 with: - version: "1.12" + version: "1" - uses: julia-actions/cache@v2 - id: julia-cache + with: + # Steal cache from the build job + cache-name: Documentation - uses: actions/download-artifact@v6 with: name: documentation-build path: docs/build - - name: Remove extra files - # There may be JLD2 files generated during execution of Literate - # examples, we don't need them on the website. - run: - rm -fv docs/build/literated/*.jld2 - name: Instantiate docs environment shell: julia --color=yes {0} run: | @@ -107,11 +106,3 @@ jobs: env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }} - - name: Save Julia depot cache on cancel or failure - id: julia-cache-save - if: cancelled() || failure() - uses: actions/cache/save@v4 - with: - path: | - ${{ steps.julia-cache.outputs.cache-paths }} - key: ${{ steps.julia-cache.outputs.cache-key }} From 3a72bad7802cb46ae4a0eb54f6cb6f8003bee4af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 20:10:09 +0000 Subject: [PATCH 11/18] Fix syntax highlighting? --- docs/src/advanced.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/advanced.md b/docs/src/advanced.md index fa0130a..a2d6bb6 100644 --- a/docs/src/advanced.md +++ b/docs/src/advanced.md @@ -158,7 +158,7 @@ jltest --threads=auto -- --verbose MyTestsetA Shell alias: -```shell +```bash function jltest { julia=(julia) From d9874652f3d4557a0ec40361eef47aa2e592d5a6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Thu, 4 Dec 2025 20:19:05 +0000 Subject: [PATCH 12/18] Add workflow for cleaning up previews on schedule --- .github/workflows/DocPreviewsCleanup.yml | 32 ++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 .github/workflows/DocPreviewsCleanup.yml diff --git a/.github/workflows/DocPreviewsCleanup.yml b/.github/workflows/DocPreviewsCleanup.yml new file mode 100644 index 0000000..7037f1c --- /dev/null +++ b/.github/workflows/DocPreviewsCleanup.yml @@ -0,0 +1,32 @@ +name: Doc Previews Cleanup + +on: + schedule: + - cron: "25 22 * * 2" + workflow_dispatch: + +concurrency: + # Same group concurrency as the `Documentation.yml` workflow, because they both + # git-push to the same branch, so we want to avoid clashes. + group: docs-pushing + +jobs: + doc-preview-cleanup: + runs-on: ubuntu-latest + permissions: + contents: write + steps: + - name: Checkout gh-pages branch + uses: actions/checkout@v6 + with: + ref: gh-pages + - name: Delete preview and history + push changes + run: | + if [[ -d previews ]]; then + git config user.name "${{github.actor}}" + git config user.email "${{github.actor_id}}+${{github.actor}}@users.noreply.github.com" + git rm -rf previews/ + git commit -m 'Delete previews directory' + git branch gh-pages-new $(echo "Delete history" | git commit-tree HEAD^{tree}) + git push --force-with-lease origin gh-pages-new:gh-pages + fi From 9e729762e7050170537a3dee58930a1fb641fe80 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= <765740+giordano@users.noreply.github.com> Date: Fri, 5 Dec 2025 00:21:40 +0000 Subject: [PATCH 13/18] Tweak query of packages using PTR --- docs/src/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/index.md b/docs/src/index.md index 3d04cf3..4859c75 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -141,7 +141,7 @@ Each test file runs in its own isolated module, so you don't need to worry about ## Packages using ParallelTestRunner.jl -There are a few packages already [using `ParallelTestRunner.jl`](https://github.com/search?q=%22using+ParallelTestRunner%22+language%3AJulia++NOT+is%3Aarchived+NOT+is%3Afork&type=code) to parallelize their tests, you can look at their setups if you need inspiration to move your packages as well: +There are a few packages already [using `ParallelTestRunner.jl`](https://github.com/search?q=%22using+ParallelTestRunner%22+language%3AJulia++NOT+is%3Aarchived+NOT+is%3Afork+path%3A%2F%5Etest%5C%2Fruntests.jl%2F&type=code) to parallelize their tests, you can look at their setups if you need inspiration to move your packages as well: * [`ApproxFun.jl`](https://github.com/JuliaApproximation/ApproxFun.jl/blob/master/test/runtests.jl) * [`BlockArrays.jl`](https://github.com/JuliaArrays/BlockArrays.jl/blob/master/test/runtests.jl) From 864d445a9f4775da572fc2988427ebe40815c167 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Fri, 5 Dec 2025 00:33:35 +0000 Subject: [PATCH 14/18] Clarify filtering is on test files, not testsets --- docs/src/advanced.md | 15 +++++++++------ docs/src/index.md | 2 +- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/src/advanced.md b/docs/src/advanced.md index a2d6bb6..28d0e2b 100644 --- a/docs/src/advanced.md +++ b/docs/src/advanced.md @@ -139,21 +139,24 @@ Custom flags are stored in the `custom` field of the `ParsedArgs` object, with v ## Interactive use -Arguments can also be passed via the standard `Pkg.test` interface for interactive control. For example, here is how we could run the subset of tests that start with the testset name "MyTestsetA" in i) verbose mode, and ii) with default threading enabled: +Arguments can also be passed via the standard [`Pkg.test`](https://pkgdocs.julialang.org/v1/api/#Pkg.test) interface for interactive control. +For example, here is how we could run the subset of test files that start with the name `test_cool_feature` in i) verbose mode, and ii) with a specific number of Julia threads enabled: +```bash +# Start julia in an environment where `MyPackage.jl` is available +julia --project +``` ```julia-repl -# In an environment where `MyPackage.jl` is available -julia --proj - julia> using Pkg # No need to start a fresh session to change threading -julia> Pkg.test("MyModule"; test_args=`--verbose MyTestsetA`, julia_args=`--threads=auto`); +julia> Pkg.test("MyModule"; test_args=`--verbose test_cool_feature`, julia_args=`--threads=auto`); ``` + Alternatively, arguments can be passed directly from the command line with a shell alias like the one below: ```julia-repl -jltest --threads=auto -- --verbose MyTestsetA +jltest --threads=auto -- --verbose test_cool_feature ``` Shell alias: diff --git a/docs/src/index.md b/docs/src/index.md index 4859c75..ea35b40 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -61,7 +61,7 @@ julia --project test/runtests.jl [OPTIONS] [TESTS...] - `--verbose`: Print more detailed information during test execution (including start times for each test) - `--quickfail`: Stop the entire test run as soon as any test fails - `--jobs=N`: Use `N` worker processes (default: based on CPU threads and available memory) -- `TESTS...`: Filter tests by name using prefix matching +- `TESTS...`: Filter test files by name, matched using `startswith` ### Examples From dccdfee99d5d0ffc79c08bcd5e7fe079213d69f9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Fri, 5 Dec 2025 00:40:23 +0000 Subject: [PATCH 15/18] Load balance parentheses --- docs/src/advanced.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/advanced.md b/docs/src/advanced.md index 28d0e2b..eef7ae8 100644 --- a/docs/src/advanced.md +++ b/docs/src/advanced.md @@ -113,7 +113,7 @@ runtests(MyPackage, ARGS; test_worker, testsuite) ``` The `test_worker` function receives the test name and should return either: -- A worker object (from [`addworker`](@ref) for tests that need special configuration +- A worker object (from [`addworker`](@ref)) for tests that need special configuration - `nothing` to use the default worker pool ## Custom Arguments From 663da638b76a84fd8a2a4da8430ff30446b729dc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Fri, 5 Dec 2025 00:49:14 +0000 Subject: [PATCH 16/18] Add more cross-references in docstrings --- src/ParallelTestRunner.jl | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/src/ParallelTestRunner.jl b/src/ParallelTestRunner.jl index 9b590f5..92032b1 100644 --- a/src/ParallelTestRunner.jl +++ b/src/ParallelTestRunner.jl @@ -410,9 +410,15 @@ const WORKER_IDS = Dict{Int32, Int32}() worker_id(wrkr) = WORKER_IDS[wrkr.proc_pid] """ - addworkers(X; kwargs...) + addworkers(; env=Vector{Pair{String, String}}(), exename=nothing, exeflags=nothing) Add `X` worker processes. +To add a single worker, use [`addworker`](@ref). + +## Arguments +- `env`: Vector of environment variable pairs to set for the worker process. +- `exename`: Custom executable to use for the worker process. +- `exeflags`: Custom flags to pass to the worker process. """ addworkers(X; kwargs...) = [addworker(; kwargs...) for _ in 1:X] @@ -420,6 +426,7 @@ addworkers(X; kwargs...) = [addworker(; kwargs...) for _ in 1:X] addworker(; env=Vector{Pair{String, String}}(), exename=nothing, exeflags=nothing) Add a single worker process. +To add multiple workers, use [`addworkers`](@ref). ## Arguments - `env`: Vector of environment variable pairs to set for the worker process. @@ -623,11 +630,11 @@ Run Julia tests in parallel across multiple worker processes. - `mod`: The module calling runtests - `ARGS`: Command line arguments array, typically from `Base.ARGS`. When you run the tests with `Pkg.test`, this can be changed with the `test_args` keyword argument. If the caller - needs to accept args too, consider using `parse_args` to parse the arguments first. + needs to accept args too, consider using [`parse_args`](@ref) to parse the arguments first. Several keyword arguments are also supported: -- `testsuite`: Dictionary mapping test names to expressions to execute (default: `find_tests(pwd())`). +- `testsuite`: Dictionary mapping test names to expressions to execute (default: [`find_tests(pwd())`](@ref)). By default, automatically discovers all `.jl` files in the test directory. - `init_code`: Code use to initialize each test's sandbox module (e.g., import auxiliary packages, define constants, etc). From 35fad33aaf81e2827226494a09ef314ae351931c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Fri, 5 Dec 2025 00:49:27 +0000 Subject: [PATCH 17/18] Clarify once again filtering is on test files, not testsets --- src/ParallelTestRunner.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ParallelTestRunner.jl b/src/ParallelTestRunner.jl index 92032b1..7ef9631 100644 --- a/src/ParallelTestRunner.jl +++ b/src/ParallelTestRunner.jl @@ -649,7 +649,7 @@ Several keyword arguments are also supported: - `--verbose`: Print more detailed information during test execution - `--quickfail`: Stop the entire test run as soon as any test fails - `--jobs=N`: Use N worker processes (default: based on CPU threads and available memory) -- `TESTS...`: Filter tests by name, matched using `startswith` +- `TESTS...`: Filter test files by name, matched using `startswith` ## Behavior From 2e888b10f1768c031a21918555fb65d6f5fcf4ab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mos=C3=A8=20Giordano?= Date: Fri, 5 Dec 2025 00:51:35 +0000 Subject: [PATCH 18/18] Update load balancing strategy in docstring --- src/ParallelTestRunner.jl | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/ParallelTestRunner.jl b/src/ParallelTestRunner.jl index 7ef9631..9f945d8 100644 --- a/src/ParallelTestRunner.jl +++ b/src/ParallelTestRunner.jl @@ -654,12 +654,12 @@ Several keyword arguments are also supported: ## Behavior - Automatically discovers all `.jl` files in the test directory (excluding `runtests.jl`) -- Sorts tests by file size (largest first) for load balancing +- Sorts test files by runtime (longest-running are started first) for load balancing - Launches worker processes with appropriate Julia flags for testing - Monitors memory usage and recycles workers that exceed memory limits - Provides real-time progress output with timing and memory statistics - Handles interruptions gracefully (Ctrl+C) -- Returns nothing, but throws `Test.FallbackTestSetException` if any tests fail +- Returns `nothing`, but throws `Test.FallbackTestSetException` if any tests fail ## Examples