Skip to content

Commit efb20bf

Browse files
economoneconomon
authored
Merge pull request #30 from su2code/feature_unit_testing
Add documentation for unit testing
2 parents 828b0c1 + 87a8182 commit efb20bf

File tree

3 files changed

+304
-0
lines changed

3 files changed

+304
-0
lines changed

‎_data/docs_v7.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
docs_v7:
2525
- Build-SU2-Linux-MacOS
2626
- Build-SU2-Windows
27+
- Running-Unit-Tests
2728

2829
- title: Theory
2930
docs_v7:
@@ -53,6 +54,7 @@
5354
- Style-Guide
5455
- Advanced-AD-Techniques
5556
- Container-Development
57+
- Writing-Unit-Tests
5658

5759
- title: FAQ
5860
docs_v7:

‎_docs_v7/Running-Unit-Tests.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
---
2+
title: Running Unit Tests
3+
permalink: /docs/Running-Unit-Tests/
4+
---
5+
6+
Unit tests are small tests of individual "units" of code. They allow
7+
developers and users to check that specific parts of the code are
8+
behaving as expected. These tests differ from integration or validation
9+
tests, which test how different units of the code interact.
10+
11+
If you are building from source, you can run the unit tests yourself to
12+
verify the the specific behaviors in the tests. Unit tests can never be
13+
comprehensive; the unit tests may all pass even if there is a bug in the
14+
code.
15+
16+
## Compiling Unit Tests
17+
18+
Unit tests are only supported with meson builds. You must build from
19+
source to build the unit tests. They are not part of the pre-compiled
20+
executables. For more information, see [Installation](/docs_v7/Installation.md),
21+
and [Build SU2 on Linux/MacOS](/docs_v7/Build-SU2-Linux-MacOS/).
22+
23+
In order to compile the unit tests, add the flag `-Denable-tests=true`
24+
to your meson configure call. Then, you can build and run the tests by
25+
calling `ninja test`.
26+
27+
## Running Tests
28+
29+
There are three ways to run the main unit tests:
30+
31+
1. `meson test -C builddir`, where `builddir` is the build directory.
32+
2. `ninja test -C builddir`, where `builddir` is the build directory.
33+
3. `./UnitTests/test_driver` from the SU2 build directory.
34+
35+
If you have run `ninja install`, then the `test_driver` executable will
36+
also be located in the `bin` directory where you have installed SU2. The
37+
first option will call ninja, which will then run the `test_driver`
38+
executable. The second option will call the `test_driver` executable.
39+
The last option, manually running the test driver, gives the most flexibility.
40+
This help page will focus on the command-line options for that last option.
41+
42+
By default, Catch2 will only show the output from failing tests. To also
43+
see the output for failing tests, add the command line argument `-s` when
44+
running the `test_driver` manually.
45+
46+
The above discussion over-simplifies the test driver setup. There
47+
are actually three test drivers:
48+
`test_driver`, `test_driver_AD`, and `test_driver_DD`. These test drivers
49+
are built or run depending on the type of installation (e.g. direcdiff,
50+
autodiff). For the most common use-case, you will not have a directdiff
51+
or autodiff build and will only use `test_driver`. If you call
52+
`meson_test` or `ninja test`, the correct
53+
drivers will run automatically. For more on tests using algorithmic
54+
differentiation or direct differentiation, see the section "AD and
55+
Direct-differentiation tests" below.
56+
57+
### Selecting subsets of the tests
58+
59+
You can also filter tests in two ways. The most basic, top-level
60+
grouping is by test cases. You can see all the test cases by running:
61+
```
62+
./UnitTests/test_driver --list-tests
63+
```
64+
You can then select a test case by name. For example, if I want to run
65+
the test case "Volume Computation", I would run:
66+
```
67+
./UnitTests/test_driver "Volume Computation"
68+
```
69+
Within each test case, the test sections form trees of arbitrary depth.
70+
Each branch can be selected with the `-c` or the `--section` command line
71+
argument. For example, if I want to run the test section "2D Edge" from
72+
the test case "Volume Computation", I would run:
73+
```
74+
./UnitTests/test_driver "Volume Computation" -c "2D Edge"
75+
```
76+
77+
Further sub-sections can be selected by chaining together multiple "-c"
78+
selections, e.g.
79+
```
80+
./UnitTests/test_driver "Volume Computation" -c "Section Name" -c "Subsection Name"
81+
```
82+
83+
You can also filter tests by tags. To see all the available tags,
84+
run:
85+
86+
```
87+
./UnitTests/test_driver --list-tags
88+
```
89+
90+
To run tests matching a specific tag, write the tag name in square braces
91+
as an argument for the test driver. For example, if I want to run the
92+
tests with the tag "Dual Grid", I would run:
93+
94+
```
95+
./UnitTests/test_driver "[Dual Grid]"
96+
```
97+
98+
If you want to run tests matching any of multiple tags, then include them all
99+
in the quotations, but keep the square brackets separate. For example:
100+
101+
```
102+
./UnitTests/test_driver "[Primal Grid], [Dual Grid]"
103+
```
104+
105+
There is also a fundamental difference between
106+
`"[Primal Grid], [Dual Grid]"` and `"[Primal Grid][Dual Grid]"`. The first
107+
selects all test matching *either* "Primal Grid" or "Dual Grid".
108+
The second selects all tests matching *both* "Primal Grid" and "Dual Grid".
109+
If that's confusing to you or you would like to know more, then please read
110+
Catch2's documentation.
111+
112+
## AD and Direct-differentiation tests
113+
114+
While the above discussion focused on the `test_driver`, there are also
115+
two additional test drivers for algorithmic differentiation (AD) and
116+
direct-differentiation (DD). Because these executables need to be linked
117+
to different libraries (normal vs AD vs DD libraries), each exists as a
118+
standalone executable. As an example, `test_driver_AD` will test the
119+
SU2 libraries compiled with AD support.
120+
121+
Depending on your installation, one or two of these test drivers may be
122+
missing. This is normal. SU2 will only install the test drivers that
123+
match the compilation type you requested. For example, neither
124+
`test_driver_AD` nor `test_driver_DD` will be compiled if
125+
you build SU2 with the default options `-Denable-autodiff=false` and
126+
`-Denable-directdiff=false`.
127+
128+
To run the test drivers manually, you can run any of the three options
129+
from the build directory. If you installed the program, these test
130+
drivers will also be found in the `bin` folder of your install directory:
131+
```
132+
./UnitTests/test_driver
133+
./UnitTests/test_driver_AD
134+
./UnitTests/test_driver_DD
135+
```
136+
137+
## FAQ
138+
139+
#### What's the difference between a test case, a test section, and a tag?
140+
141+
Test cases are the highest level of organization. Each test must belong
142+
to exactly one test case. Within a test case, test sections can be used
143+
to organize related tests. These test sections form a tree of arbitrary
144+
depth. Test sections allow multiple tests to share setup, teardown, or
145+
data. However, note that the tests are always run separately from start
146+
to finish. Even if two tests use the same data, the two tests cannot
147+
interact.
148+
149+
Tags allow you to identify related tests. One important detail is that each
150+
test can have multiple tags. The tags can span multiple test sections. Tags
151+
allow complicated groupings of tests. Tags do not facilitate sharing of
152+
setup, teardown, or data.

‎_docs_v7/Writing-Unit-Tests.md‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
---
2+
title: Writing Unit Tests
3+
permalink: /docs/Writing-Unit-Tests/
4+
---
5+
6+
This document covers the question "How do I write unit tests for SU2?"
7+
Before reading this page, make sure you are familiar with the pages
8+
[Running Unit Tests](/docs_v7/Running-Unit-Tests/) and
9+
[Build SU2 on Linux/MacOS](/docs_v7/Build-SU2-Linux-MacOS/).
10+
11+
This document is intended to be an introduction only. There are plenty of
12+
details that are *not* covered in this document. This omission is
13+
intentional. The Catch2 documentation is the best place to learn about
14+
using Catch2, so this guide does not duplicate any discussion of the
15+
complex features of Catch2. These more complex features include:
16+
17+
+ Grouping tests into sections with similar setup or teardown
18+
+ Parameterized tests
19+
+ Logging context to report alongside failures
20+
+ Tests that are expected to throw exceptions
21+
+ Hiding tests from the default list
22+
+ Using Catch2 with a debugger
23+
+ Custom matchers
24+
25+
Please refer to the Catch2 documentation for more information on these
26+
capabilities.
27+
28+
## Catch2 Unit-Testing Framework
29+
30+
A unit-testing framework takes most of the boilerplate code out of unit
31+
testing. There's no need to manually create a `main()` for each test,
32+
nor do you need to write your own floating-point matcher. A good unit-testing
33+
framework also provides many command-line tools, such as filtering the tests
34+
and controlling the output.
35+
Catch2 was chosen for the unit testing library for the following reasons:
36+
37+
+ It can be included as a header-only library.
38+
+ It has a very clean, easy-to-use syntax
39+
+ It has widespread use, including the FEniCS library.
40+
41+
## Design Overview
42+
43+
+ Catch2 is included as a header file in the `externals` folder, and is
44+
distributed with the SU2 source code.
45+
+ All tests are placed in a top-level directory named `UnitTests`. Inside this directory is a structure just like the existing folder structure (e.g. `SU2_CFD/numerics`), except that there are not separate `src` and `include` folders.
46+
+ Tests are grouped by class, and put in files such as `CNumerics_tests.cpp`.
47+
+ A single test executable is compiled and run, as opposed to a separate test executable for each group of tests. With the single test executable, you can always filter down to groups of tests or individual tests.
48+
+ When `ninja test` or `meson test` is run, the test executable is only run once, sweeping through all the unit tests. The result only shows a single failure or a single success. If more detail is desired, then the test executable can be run manually.
49+
+ The relevant SU2 code is compiled as a library (e.g. `libSU2core`) and then linked into both the main executable (`SU2_CFD`) and the test program (`test_driver`).
50+
51+
## Writing a Basic Unit Test
52+
53+
There are many working examples of unit tests within the SU2 codebase.
54+
To see them, browse the folder `UnitTests/`. For an introduction, a simple
55+
example is given here. In order to add this test, the developer would create
56+
a `.cpp` file with something like the following lines:
57+
58+
```
59+
#include "catch.hpp"
60+
61+
TEST_CASE("Addition", "[arithmetic]") {
62+
63+
int a = 2, b = 2;
64+
65+
REQUIRE(a + b == 4);
66+
67+
}
68+
```
69+
70+
+ The header `catch.hpp` contains the macros used for unit tests. These
71+
macros replace a lot of the boilerplate code needed for testing.
72+
+ `TEST_CASE` is used to define a test case. Each unit test *must* start with
73+
this macro.
74+
+ Here, `Addition` is the name of the test case. The name is a proper,
75+
string, and can have spaces, numbers, or puncutation. This stands in
76+
contrast to many unit-testing frameworks, such as Boost test or Google
77+
test, where the test names must be valid C++ variable names.
78+
+ The second argument, `[arithmetic]` is a tag. Tags are used to group
79+
related tests together, even if they occur in different test cases or
80+
files.
81+
+ `REQUIRE()` is similar to `assert()`. It checks that the contained logical
82+
statement is true. If not, it will display relevant error messages.
83+
+ You can also use the macro `CHECK()`. This also checks if the contained
84+
logical statment is true, but it does not stop on the first failure. It
85+
will record the results and continue execution, whether or not the check
86+
is true.
87+
88+
For more detail on writing tests, please visit Catch2's documentation.
89+
90+
## Adding the Unit Test to the Test Driver
91+
92+
In order to run the unit test, it must be added to the meson build scripts.
93+
Add the new `.cpp` file to `UnitTests/meson.build`. This will tell meson
94+
to build your unit test, then run it using the test driver provided by Catch2.
95+
96+
## Floating Point Unit Tests
97+
98+
An important part of unit testing in scientific computing is floating
99+
point comparisons. For tests with floating point numbers, it is not proper
100+
to require a strong equality. Instead, Catch2 provides the wrapper
101+
class `Approx`. `Approx` can be placed on either side of an equality
102+
to indicate that two floating-point numbers are only *approximately* equal.
103+
104+
For example:
105+
```
106+
REQUIRE(volume == Approx(0.85478577));
107+
```
108+
109+
The type of approximation and the precision can be customized. For more
110+
detail on these customizations, please see Catch2's documentation.
111+
Alternatively, Catch2 also supplies custom matchers for use with
112+
floating-point numbers. These include `WithinAbs`, `WithinULP`,
113+
and `WithinRel`.
114+
115+
## Directdiff and AD Tests
116+
117+
You can also write tests that involve algorithmic differentiation (AD)
118+
and direct differentiation (DD). There are a few things that must be done
119+
differently when writing these tests:
120+
121+
- The test must be linked with the correct AD or DD libraries. To do
122+
so, add the test to the meson lists `su2_cfd_tests_ad` or
123+
`su2_cfd_tests_dd`, respectively, in `UnitTests/meson.build`.
124+
- When using the `su2double` datatype in and AD or DD build, the value
125+
stored in su2double is accessed using `SU2_TYPE::GetValue()`. A simple
126+
assert such as `CHECK(y == Approx(64.0)` will fail to compile
127+
if `y` is an su2double. This is because you're trying to compare an
128+
su2double object, with both values and derivatives, with a float. The
129+
correct form to use is `CHECK(SU2_TYPE::GetValue(y) == Approx(64.0))`
130+
(and you can also use `REQUIRE`).
131+
132+
## Real Examples
133+
134+
If you find the above examples a bit simplistic, then you can refer to the
135+
following files to find simple, real-world unit-tests:
136+
137+
- `UnitTests/SU2_CFD/numerics/CNumerics_tests.cpp`
138+
- `UnitTests/Common/geometry/primal_grid/CPrimalGrid_tests.cpp`
139+
- `UnitTests/Common/geometry/dual_grid/CDualGrid_tests.cpp`
140+
- `UnitTests/Common/simple_ad_test.cpp`
141+
- `UnitTests/Common/simple_directdiff_test.cpp`
142+
143+
## FAQ
144+
145+
### Where can I learn more about unit tests?
146+
147+
The following two books are great introductions to software testing:
148+
149+
+ "Working Effectively with Legacy Code," by Michael Feathers
150+
+ "Modern C++ Programming with Test-Driven Development," by Jeff Langr

0 commit comments

Comments
 (0)