From 38fcc0030d48ce9f964f1bd13f5ed5600b7eaa9a Mon Sep 17 00:00:00 2001 From: nathanrboyer Date: Tue, 26 Aug 2025 16:47:06 -0400 Subject: [PATCH 1/5] First try --- optimizing/index.md | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/optimizing/index.md b/optimizing/index.md index e68b72d..5f6b540 100644 --- a/optimizing/index.md +++ b/optimizing/index.md @@ -36,7 +36,7 @@ With this in mind, after you're done with the current page, you should read the ## Measurements -\tldr{Use BenchmarkTools.jl's `@benchmark` with a setup phase to get the most accurate idea of your code's performance. Use Chairmarks.jl as a faster alternative.} +\tldr{Use Chairmarks.jl's `@be` with a setup phase to get the most accurate idea of your code's performance.} The simplest way to measure how fast a piece of code runs is to use the `@time` macro, which returns the result of the code and prints the measured runtime and allocations. Because code needs to be compiled before it can be run, you should first run a function without timing it so it can be compiled, and then time it: @@ -54,20 +54,20 @@ Using `@time` is quick but it has flaws, because your function is only measured That measurement might have been influenced by other things going on in your computer at the same time. In general, running the same block of code multiple times is a safer measurement method, because it diminishes the probability of only observing an outlier. -### BenchmarkTools +### Chairmarks -[BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) is the most popular package for repeated measurements on function executions. -Similarly to `@time`, BenchmarkTools offers `@btime` which can be used in exactly the same way but will run the code multiple times and provide an average. -Additionally, by using `$` to [interpolate external values](https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Interpolating-values-into-benchmark-expressions), you remove the overhead caused by global variables. +[Chairmarks.jl](https://github.com/LilithHafner/Chairmarks.jl) is the latest and greatest benchmarking suite used to make fast and accurate timing measurements. +Chairmarks offers `@b` (for "benchmark") which can be used in exactly the same way as `@time` but will run the code multiple times and provide a minimum execution time. +Alternatively, Chairmarks also provides `@be` to run the benchmark and output all of its statistics. ```>$-example -using BenchmarkTools -@btime sum_abs(v); -@btime sum_abs($v); +using Chairmarks +@b sum_abs(v) +@be sum_abs(v) ``` -In more complex settings, you might need to construct variables in a [setup phase](https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Setup-and-teardown-phases) that is run before each sample. -This can be useful to generate a new random input every time, instead of always using the same input. +Chairmarks supports a pipeline syntax with optional `init`, `setup`, `teardown`, and `keywords` arguments for more extensive control over the benchmarking process. +For example, you could write the following to benchmark a matrix multiplication function for one second excluding time spent creating the arrays. ```>setup-example my_matmul(A, b) = A * b; @@ -75,17 +75,16 @@ my_matmul(A, b) = A * b; A = rand(1000, 1000); # use semi-colons between setup lines b = rand(1000) ); +# Needs fixed ``` -For better visualization, the `@benchmark` macro shows performance histograms: +For better visualization, [PrettyChairmarks.jl](https://github.com/astrozot/PrettyChairmarks.jl) shows performance histograms. \advanced{ Certain computations may be [optimized away by the compiler]((https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Understanding-compiler-optimizations)) before the benchmark takes place. If you observe suspiciously fast performance, especially below the nanosecond scale, this is very likely to have happened. } -[Chairmarks.jl](https://github.com/LilithHafner/Chairmarks.jl) offers an alternative to BenchmarkTools.jl, promising faster benchmarking while attempting to maintain high accuracy and using an alternative syntax based on pipelines. - ### Benchmark suites While we previously discussed the importance of documenting breaking changes in packages using [semantic versioning](/sharing/index.md#versions-and-registration), regressions in performance can also be vital to track. @@ -97,10 +96,13 @@ Several packages exist for this purpose: ### Other tools -BenchmarkTools.jl works fine for relatively short and simple blocks of code (microbenchmarking). +Chairmarks.jl works fine for relatively short and simple blocks of code (microbenchmarking). To find bottlenecks in a larger program, you should rather use a [profiler](#profiling) or the package [TimerOutputs.jl](https://github.com/KristofferC/TimerOutputs.jl). It allows you to label different sections of your code, then time them and display a table of grouped by label. +[BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) was the previous standard for benchmarking in Julia. It is still widely used today. +However, it is slower than Chairmarks and requires interpolating variables into the benchmarked expressions with `$`. + Finally, if you know a loop is slow and you'll need to wait for it to be done, you can use [ProgressMeter.jl](https://github.com/timholy/ProgressMeter.jl) or [ProgressLogging.jl](https://github.com/JuliaLogging/ProgressLogging.jl) to track its progress. ## Profiling @@ -141,7 +143,7 @@ To integrate profile visualisations into environments like Jupyter and Pluto, us No matter which tool you use, if your code is too fast to collect samples, you may need to run it multiple times in a loop. \advanced{ - To visualize memory allocation profiles, use PProf.jl or VSCode's `@profview_allocs`. + To visualize memory allocation profiles, use PProf.jl or VSCode's `@profview_allocs`. A known issue with the allocation profiler is that it is not able to determine the type of every object allocated, instead `Profile.Allocs.UnknownType` is shown instead. Inspecting the call graph can help identify which types are responsible for the allocations. } @@ -386,7 +388,7 @@ However, in order for all workers to know about a function or module, we have to using Distributed # Add additional workers then load code on the workers -addprocs(3) +addprocs(3) @everywhere using SharedArrays @everywhere f(x) = 3x^2 From 48ae0fe35e8d75a21cca78405d3373903165a7c3 Mon Sep 17 00:00:00 2001 From: nathanrboyer Date: Wed, 27 Aug 2025 11:14:01 -0400 Subject: [PATCH 2/5] Ready for PR --- optimizing/index.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/optimizing/index.md b/optimizing/index.md index 5f6b540..a774378 100644 --- a/optimizing/index.md +++ b/optimizing/index.md @@ -53,6 +53,7 @@ using BenchmarkTools Using `@time` is quick but it has flaws, because your function is only measured once. That measurement might have been influenced by other things going on in your computer at the same time. In general, running the same block of code multiple times is a safer measurement method, because it diminishes the probability of only observing an outlier. +The Chairmarks.jl package provides convenient syntax to do just that. ### Chairmarks @@ -60,25 +61,28 @@ In general, running the same block of code multiple times is a safer measurement Chairmarks offers `@b` (for "benchmark") which can be used in exactly the same way as `@time` but will run the code multiple times and provide a minimum execution time. Alternatively, Chairmarks also provides `@be` to run the benchmark and output all of its statistics. -```>$-example +```>chairmarks-example using Chairmarks @b sum_abs(v) @be sum_abs(v) ``` Chairmarks supports a pipeline syntax with optional `init`, `setup`, `teardown`, and `keywords` arguments for more extensive control over the benchmarking process. -For example, you could write the following to benchmark a matrix multiplication function for one second excluding time spent creating the arrays. +The `sum_abs` function could also be benchmarked using pipeline syntax as below. -```>setup-example +```>pipeline-example-simple +@be v sum_abs +``` + +For a more complicated example, you could write the following to benchmark a matrix multiplication function for one second, excluding the time spent to *setup* the arrays. + +```>pipeline-example-complex my_matmul(A, b) = A * b; -@btime my_matmul(A, b) setup=( - A = rand(1000, 1000); # use semi-colons between setup lines - b = rand(1000) -); -# Needs fixed +@be (A=rand(1000,1000), b=rand(1000)) my_matmul(_.A, _.b) seconds=1 ``` -For better visualization, [PrettyChairmarks.jl](https://github.com/astrozot/PrettyChairmarks.jl) shows performance histograms. +See the [Chairmarks documentation](https://chairmarks.lilithhafner.com/) for more details on benchmarking options. +For better visualization, [PrettyChairmarks.jl](https://github.com/astrozot/PrettyChairmarks.jl) shows performance histograms alongside the numerical results. \advanced{ Certain computations may be [optimized away by the compiler]((https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Understanding-compiler-optimizations)) before the benchmark takes place. From 478a79be5be50592397432d6c74077115c0f8987 Mon Sep 17 00:00:00 2001 From: nathanrboyer Date: Wed, 27 Aug 2025 11:59:21 -0400 Subject: [PATCH 3/5] Remove "exactly" --- optimizing/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimizing/index.md b/optimizing/index.md index a774378..d4402ce 100644 --- a/optimizing/index.md +++ b/optimizing/index.md @@ -58,7 +58,7 @@ The Chairmarks.jl package provides convenient syntax to do just that. ### Chairmarks [Chairmarks.jl](https://github.com/LilithHafner/Chairmarks.jl) is the latest and greatest benchmarking suite used to make fast and accurate timing measurements. -Chairmarks offers `@b` (for "benchmark") which can be used in exactly the same way as `@time` but will run the code multiple times and provide a minimum execution time. +Chairmarks offers `@b` (for "benchmark") which can be used in the same way as `@time` but will run the code multiple times and provide a minimum execution time. Alternatively, Chairmarks also provides `@be` to run the benchmark and output all of its statistics. ```>chairmarks-example From 5ae3504188d0cbcead90571c2b9f2b98800efa2e Mon Sep 17 00:00:00 2001 From: Nathan Boyer <65452054+nathanrboyer@users.noreply.github.com> Date: Mon, 1 Sep 2025 13:07:53 -0400 Subject: [PATCH 4/5] Apply suggestions from code review Co-authored-by: Guillaume Dalle <22795598+gdalle@users.noreply.github.com> --- optimizing/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/optimizing/index.md b/optimizing/index.md index d4402ce..f6c3e73 100644 --- a/optimizing/index.md +++ b/optimizing/index.md @@ -57,9 +57,9 @@ The Chairmarks.jl package provides convenient syntax to do just that. ### Chairmarks -[Chairmarks.jl](https://github.com/LilithHafner/Chairmarks.jl) is the latest and greatest benchmarking suite used to make fast and accurate timing measurements. +[Chairmarks.jl](https://github.com/LilithHafner/Chairmarks.jl) is the latest benchmarking toolkit, designed to make fast and accurate timing measurements. Chairmarks offers `@b` (for "benchmark") which can be used in the same way as `@time` but will run the code multiple times and provide a minimum execution time. -Alternatively, Chairmarks also provides `@be` to run the benchmark and output all of its statistics. +Alternatively, Chairmarks also provides `@be` to run the same benchmark and output all of its statistics. ```>chairmarks-example using Chairmarks @@ -104,8 +104,8 @@ Chairmarks.jl works fine for relatively short and simple blocks of code (microbe To find bottlenecks in a larger program, you should rather use a [profiler](#profiling) or the package [TimerOutputs.jl](https://github.com/KristofferC/TimerOutputs.jl). It allows you to label different sections of your code, then time them and display a table of grouped by label. -[BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) was the previous standard for benchmarking in Julia. It is still widely used today. -However, it is slower than Chairmarks and requires interpolating variables into the benchmarked expressions with `$`. +[BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) is the older standard for benchmarking in Julia. It is still widely used today. +However, its default parameters run benchmarks for longer than Chairmarks, and it requires interpolating variables into the benchmarked expressions with `$`. Finally, if you know a loop is slow and you'll need to wait for it to be done, you can use [ProgressMeter.jl](https://github.com/timholy/ProgressMeter.jl) or [ProgressLogging.jl](https://github.com/JuliaLogging/ProgressLogging.jl) to track its progress. From 531d2641f93513ea75dfa7e6fa97d27c2727eee0 Mon Sep 17 00:00:00 2001 From: Martin Smit <57063134+jacobusmmsmit@users.noreply.github.com> Date: Mon, 8 Sep 2025 14:00:06 +0200 Subject: [PATCH 5/5] Clarify computation elision --- optimizing/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimizing/index.md b/optimizing/index.md index f6c3e73..ff58fa1 100644 --- a/optimizing/index.md +++ b/optimizing/index.md @@ -85,7 +85,7 @@ See the [Chairmarks documentation](https://chairmarks.lilithhafner.com/) for mor For better visualization, [PrettyChairmarks.jl](https://github.com/astrozot/PrettyChairmarks.jl) shows performance histograms alongside the numerical results. \advanced{ -Certain computations may be [optimized away by the compiler]((https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Understanding-compiler-optimizations)) before the benchmark takes place. +No matter the benchmarking tool used, certain computations may be [optimized away by the compiler]((https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Understanding-compiler-optimizations)) before the benchmark takes place. If you observe suspiciously fast performance, especially below the nanosecond scale, this is very likely to have happened. }