Overhaul CI lesson

Author: LiamPattinson

episodes/10-CI.Rmd

@@ -1,7 +1,7 @@
 ---
 title: "Continuous Integration with GitHub Actions"
-teaching: 10
-exercises: 2
+teaching: 20
+exercises: 25
 ---
 
 :::::::::::::::::::::::::::::::::::::: questions 
@@ -20,39 +20,61 @@ exercises: 2
 
 ## Continuous Integration
 
-Continuous Integration (CI) is the practice of automating the merging of code changes into a project.
-In the context of software testing, CI is the practice of running tests on every code change to ensure that the code is working as expected.
-GitHub provides a feature called GitHub Actions that allows you to integrate this into your projects.
+Continuous Integration (CI) is the practice of automating the merging of code
+changes into a project. In the context of software testing, CI is the practice
+of running tests on every code change to ensure that the code is working as
+expected. GitHub provides a feature called GitHub Actions that allows you to
+integrate this into your projects.
 
-In this lesson we will go over the very basics of how to set up a GitHub Action to run tests on your code.
+In this lesson we will go over the very basics of how to set up a GitHub Action
+to run tests on your code.
+
+:::::: prereq
+
+This lesson assumes a working knowledge of Git and GitHub. If you get stuck,
+you may find it helpful to review the Research Coding Course's
+[material on version control](https://researchcodingclub.github.io/course/#version-control-introduction-to-git-and-github)
+
+:::::::::::::
 
 ## Setting up your project repository
 
-- Create a new repository on GitHub for this lesson called "python-testing-course" (whatever you like really)
-- Clone the repository into your local machine using `git clone <repository-url>` or GitKraken if you use that.
+- Create a new repository on GitHub for this lesson called
+  "python-testing-course" (whatever you like really). We
+  recommended making it public for now.
+- Clone the repository into your local machine using `git clone
+  <repository-url>` or via Github Desktop.
 - Move over all your code from the previous lessons into this repository.
 - Commit the changes using `git add .` and `git commit -m "Add all the project code"`
-- Create a new file called `requirements.txt` in the root of your repository and add the following contents:
+- Create a new file called `requirements.txt` in the root of your repository
+  and add the following contents:
 
 ```
 pytest
 numpy
 pandas
-pytest-mpl
-pytest-regtest
-matplotlib
 ```
 
-This is just a list of all the packages that your project uses and will be needed later.
-Recall that each of these are used in various lessons in this course.
+This is just a list of all the packages that your project uses and will be
+needed later. Recall that each of these are used in various lessons in this
+course.
+
+:::::: callout
 
+Nowadays it is usually preferable to list dependencies in a file called
+`pyproject.toml`, which also allows Python packages to be installed and
+published. Look out for our upcoming course on reproducible environments to
+learn more!
+
+::::::::::::::
 
 Now we have a repository with all our code in it online on GitHub.
 
 ## Creating a GitHub Action
 
-GitHub Actions are defined in `yaml` files (these are just simple text files that contain a list of instructions). They are stored
-in the `.github/workflows` directory in your repository.
+GitHub Actions are defined in `yaml` files -- a structured text file which is
+commonly used to pass settings to programs. They are stored in the
+`.github/workflows` directory in your repository.
 
 - Create a new directory in your repository called `.github`
 - Inside the `.github` directory, create a new directory called `workflows`
@@ -66,93 +88,243 @@ Let's add some instructions to the `tests.yaml` file:
 # This is just the name of the action, you can call it whatever you like.
 name: Tests (pytest)
 
-# This is the event that triggers the action. In this case, we are telling GitHub to run the tests whenever a pull request is made to the main branch.
+# This sets the events that trigger the action. In this case, we are telling
+# GitHub to run the tests whenever a push is made to the repository.
+# The trailing colon is intentional!
 on:
-    pull_request:
-        branches:
-            - main
+  push:
 
-# This is a list of jobs that the action will run. In this case, we have only one job called build.
+# This is a list of jobs that the action will run. In this case, we have only
+# one job called build.
 jobs:
-    build:
-        # This is the environment that the job will run on. In this case, we are using the latest version of Ubuntu, however you can ues other operating systems like Windows or MacOS if you like!
-        runs-on: ubuntu-latest
-
-        # This is a list of steps that the job will run. Each step is a command that will be executed on the environment.
-        steps:
-            # This command tells GitHub to use a pre-built action. In this case, we are using the actions/checkout action to check out the repository. This just means that GitHub will use this repository's code to run the tests.
-            - uses: actions/checkout@v3 # Check out the repository on github
-            # This is the name of the step. This is just a label that will be displayed in the GitHub UI.
-            - name: Set up Python 3.10
-              # This command tells GitHub to use a pre-built action. In this case, we are using the actions/setup-python action to set up Python 3.10.
-              uses: actions/setup-python@v3
-              with:
-                  python-version: "3.10"
-            
-            # This step installs the dependencies for the project such as pytest, numpy, pandas, etc using the requirements.txt file we created earlier.
-            - name: Install dependencies
-              run: |
-                python -m pip install --upgrade pip
-                pip install -r requirements.txt
-                    
-            # This step runs the tests using the pytest command. Remember to use the --mpl and --regtest flags to run the tests that use matplotlib and pytest-regtest.
-            - name: Run tests
-              run: |
-                pytest --mpl --regtest
+
+  build:
+
+    # This is the environment that the job will run on. In this case, we are
+    # using the latest version of Ubuntu, however you can use other operating
+    # systems like Windows or MacOS if you like!
+    runs-on: ubuntu-latest
+
+    # This is a list of steps that the job will run. Each step is a command
+    # that will be executed on the environment.
+    steps:
+
+      # This command tells GitHub to use a pre-built action. In this case, we
+      # are using the actions/checkout action to check out the repository. This
+      # just means that GitHub will clone this repository to the current
+      # working directory.
+      - uses: actions/checkout@v6
+
+      # This is the name of the step. This is just a label that will be
+      # displayed in the GitHub UI.
+      - name: Set up Python 3.12
+        # This command tells GitHub to use a pre-built action. In this case, we
+        # are using the actions/setup-python action to set up Python 3.12.
+        uses: actions/setup-python@v6
+        with:
+            python-version: "3.12"
+
+      # This step installs the dependencies for the project such as pytest,
+      # numpy, pandas, etc using the requirements.txt file we created earlier.
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      # This step runs the tests using the pytest command. 
+      - name: Run tests
+        run: |
+          pytest
 ```
 
-This is a simple GitHub Action that runs the tests for your code whenever a pull request is made to the main branch.
+This is a simple GitHub Action that runs the tests for your code whenever code
+is pushed to the repository, regardless of what was changed in the repository
+or which branch you push too. We'll see later how to run tests only when
+certain criteria are fulfilled.
 
 ## Upload the workflow to GitHub
 
 Now that you have created the `tests.yaml` file, you need to upload it to GitHub.
 
 - Commit the changes using `git add .` and `git commit -m "Add GitHub Action to run tests"`
-- Push the changes to GitHub using `git push origin main`
+- Push the changes to GitHub using `git push`
+
+This should trigger a workflow on the repository. While it's running, you'll see an orange
+circle next to your profile name at the top of the repo. When it's done, it'll change to
+a green tick if it finished successfully, or a red cross if it didn't.
+
+![GitHub repository view with a green tick indicating a successful workflow run](fig/github_repo_view.png){alt="GitHub repository view with a green tick indicating a successful workflow run"}
+
+You can view all previous workflow runs by clicking the 'Actions' button on the
+top bar of your repository.
+
+![GitHub Actions button](fig/github_actions_button.png){alt="GitHub Actions Button"}
+
+If you click on the orange circle/green tick/red cross, you can also view the
+individual stages of the workflow and inspect the terminal output.
+
+![Detailed view of a GitHub workflow run](fig/github_action.png){alt="Detailed view of a GitHub workflow run"}
+
 
 ## Enable running the tests on a Pull Request
 
-The typical use-case for a CI system is to run the tests whenever a pull request is made to the main branch to add a feature.
+The typical use-case for a CI system is to run the tests when a pull request is
+made to the `main` branch to add a feature or fix a bug. However, there is a
+subtlety to consider: what if the tests pass on your feature branch, but would
+fail after merging into `main` due to a divergence between the two branches?
 
-<!-- figure
-![branch protection visual instruction](episodes/fig/github-branch-protection.png) -->
-- Go to your GitHub repository
-- Click on the "Settings" tab
-- Scroll down to "Branches"
-- Under "Branch protection rules" / "Branch name pattern" type "main"
-- Select the checkbox for "Require status checks to pass before merging"
-- Select the checkbox for "Require branches to be up to date before merging"
+To verify this, we can set our workflow to run on both `push` and
+`pull_request` by modifying the `tests.yaml` file so that the `on:` sections
+reads:
 
-This makes it so when a Pull Request is made, trying to merge code into main, it will need to have all of its tests passing
-before the code can be merged.
+```yaml
+on:
+  push:
+  pull_request:
+```
 
 Let's test it out.
 
-- Create a new branch in your repository called `subtract` using `git checkout -b subtract`
-- Add a new function in your `calculator.py` file that subtracts two numbers, but make it wrong on purpose:
+- From within your `main` branch, add a new file `sandwich.py` containing the
+  following function:
 
 ```python
-def subtract(a, b):
-    return a + b
+def sandwich():
+    bread = "======"
+    filling = "jam"
+    return "\n".join([bread, filling, bread])
 ```
 
-- Then add a test for this function in your `test_calculator.py` file:
+- Create a new branch in your repository called `sanwich_test` using `git switch -c sandwich_test`
+- Add a new file `test_sandwich.py` with the function:
 
 ```python
-def test_subtract():
-    assert subtract(5, 3) == 2
+from sandwich import sandwich
+
+def test_sandwich():
+    assert sandwich() == "======\njam\n======"
+```
+
+- Push this to your repo with `git push -u origin sandwich_test`, but don't
+  raise a pull request just yet.
+- Imagine that somebody else modifies the `main` branch before your branch can
+  be merged -- something that happens all too often in open source projets!
+  Return to the `main` branch using `git switch main`, and modify
+  `sandwich.py`:
+
+```python
+def sandwich():
+    bread = "======"
+    filling = "cheese"
+    return "\n".join([bread, filling, bread])
+```
+
+- Push this change to `main` with `git push`.
+- Now raise a pull request on your respository to merge `sandwich_test` into
+  `main`.
+
+After raising the pull request, you should see that the GitHub Action runs twice:
+once for a push, which will pass, and once for a pull request, which will fail:
+
+![Example of tests failing on pull requests.](fig/pull_request_test_failed.png){alt="Example of tests failing on pull requests."}
+
+To fix this:
+
+- Switch to the `sandwich_test` branch, and call `git merge main`.
+- Fix the test on the `sandwich_test` branch, ensure the tests pass locally,
+  and commit/push the changes.
+
+The tests should now pass for both `push` and `pull_request`.
+
+
+## Testing across multiple platforms
+
+A very useful feature of GitHub Actions is the ability to test over a wider
+range of platforms than just your own machine:
+
+- Operating systems
+- Python versions
+- Compiler versions (for those writing C/C++/Fortran/etc)
+
+We can achieve this by setting `jobs.<job_id>.strategy.matrix` in our workflow:
+
+```yaml
+jobs:
+  build:
+    strategy:
+      matrix:
+        python_version: ["3.12", "3.13", "3.14"]
+        os: ["ubuntu-latest", "windows-latest"]
+    runs-on: ${{ matrix.os }}
+    steps:
+      ...
+```
+
+Later in the file, the `setup-python` step should be changed to:
+
+```yaml
+      - name: Set up Python ${{ matrix.python_version }}
+        uses: actions/setup-python@v6
+        with:
+            python-version: ${{ matrix.python_version }}
 ```
 
-- Commit the changes using `git add .` and `git commit -m "Add subtract function"`
-- Push the changes to GitHub using `git push origin subtract`
+By default, all combinations in the matrix will be run in separate jobs. The
+syntax `${{ matrix.x }}` inserts the text from the `x` list for the given matrix job.
+
+- Switch to a new branch `matrix_test` using `git switch -c matrix_test`
+- Make the changes above to `.github/workflows/tests.yaml`
+- Commit, push, and raise a pull request to `main`.
+
+You should now see that _12_ separate jobs run!
+
+![Completed matrix jobs.](fig/matrix_tests.png){alt="Completed matrix tests."}
+
+This ensures that code that runs on your machine should, in theory, run on many
+other peoples' machines too. However, it's best to restrict the matrix to the
+minimum number of necessary platforms to ensure you don't waste resources. You
+can do so with a list of exclusions:
+
+```yaml
+    strategy:
+      matrix:
+        python_version: ["3.12", "3.13", "3.14"]
+        os: ["ubuntu-latest", "windows-latest"]
+        # Only run windows on latest Python version
+        exclude:
+          - os: "windows-latest"
+            python_version: "3.12"
+          - os: "windows-latest"
+            python_version: "3.13"
+````
 
-- Now go to your GitHub repository and create a new Pull Request to merge the `subtract` branch into `main`
 
-You should see that the GitHub Action runs the tests and fails because the test for the `subtract` function is failing.
+## Other tips
+
+You may have wondered why there is a trailing colon when we specify `push:` or `pull_request:`.
+The reason is that we can optionally set additional conditions on when they'll run. For example:
+
+```yaml
+on:
+  push:
+    # Only check when Python files are changed.
+    # Don't need to check when the README is updated!
+    paths:
+      - '**.py'
+      - 'pyproject.toml'
+  pull_request:
+    paths:
+      - '**.py'
+      - 'pyproject.toml'
+    # Only check when somebody raises a pull_request to main.
+    branches: [main]
+  # This allows you to run the tests manually if you choose
+  workflow_dispatch:
+```
+
+Doing this can prevent pointless CI jobs from running and save resources.
 
-- Let's now fix the test and commit the changes: `git add .` and `git commit -m "Fix subtract function"`
-- Push the changes to GitHub using `git push origin subtract` again
-- Go back to the Pull Request on GitHub and you should see that the tests are now passing and you can merge the code into the main branch.
+## Keypoints
 
 So now, when you or your team want to make a feature or just update the code, the workflow is as follows:
 
@@ -171,7 +343,7 @@ This will greatly improve the quality of your code and make it easier to collabo
 - Continuous Integration (CI) is the practice of automating the merging of code changes into a project.
 - GitHub Actions is a feature of GitHub that allows you to automate the testing of your code.
 - GitHub Actions are defined in `yaml` files and are stored in the `.github/workflows` directory in your repository.
-- You can use GitHub Actions to only allow code to be merged into the main branch if the tests pass.
+- You can use GitHub Actions to ensure your tests pass before merging new code into your `main` branch.
 
 ::::::::::::::::::::::::::::::::::::::::::::::::