RFC: Publish Tagged Test Implementations for Benchmarking

[msmith-techempower]

Summary

• Implement a version key in test implementation config.tomls
• Remove Toolset implementation that builds Docker images from test implementation config.toml files on debug, verify, and benchmark modes
• Formalize implementation that fetches Docker images from Dockerhub based on config.toml version key
• Formalize CI/CD strategy for automatically publishing a new image/tag for a test implementation prior to merging changes from a pull request for a test implementation

Motivation

False-negatives often occur in verification of test implementations because of ephemeral failures (network issues, removed dependency versions, etc). This classification of failure can be avoided during a benchmark by capturing the success of the build of a Docker image by publishing it to Dockerhub, and relying on its existence at the time a benchmark requires it.

Drawbacks

• Requires publishing a Docker image to Docker hub as an extra step to merging a pull request
• Requires manual intervention in pull requests to ensure that a version has been updated when changes occur (for example, if a PR is opened where the version has not been updated, manual intervention from the reviewer is required to alert the contributor to it and request that update be made - read: more manual administration)
• Requires the Docker image and tag are available at the time of a benchmark (read: for local benchmarking, a manual build and tag of the image is required beforehand; and, for benchmarking generally Dockerhub must be available)

Alternatives

• Leave current implementation which builds a new image, possibly from local cache, on every benchmark