Improve SeBS documentation

.circleci/config.yml

@@ -35,6 +35,11 @@ jobs:
             . python-venv/bin/activate
             mypy sebs --config-file=.mypy.ini
           name: Python static code verification with mypy
+      - run:
+          command: |
+            . python-venv/bin/activate
+            interrogate -v --fail-under 100 sebs
+          name: Check for Python documentation coverage 
       - store_artifacts:
           path: flake-reports
           destination: flake-reports

.readthedocs.yaml

@@ -0,0 +1,28 @@
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the OS, Python version, and other tools you might need
+build:
+  os: ubuntu-24.04
+  tools:
+    python: "3.13"
+
+# Build documentation in the "docs/" directory with Sphinx
+sphinx:
+  configuration: docs/source/conf.py
+
+# Optionally, but recommended,
+# declare the Python requirements required to build your documentation
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+python: 
+  install:
+    - requirements: requirements.docs.txt
+    - requirements: requirements.txt
+    - requirements: requirements.aws.txt
+    - requirements: requirements.azure.txt
+    - requirements: requirements.gcp.txt
+    - requirements: requirements.local.txt
+

CHANGELOG.md

@@ -0,0 +1,126 @@
+
+## WiP [1.2.0](https://github.com/spcl/serverless-benchmarks/compare/v1.1...v1.2) (XXXX)
+
+### Features
+
+#### Container & Multi-Architecture Support
+
+* Container deployment support for AWS Lambda (#205)
+* Multi-architecture support (x86_64, arm64) for benchmarks (#227)
+
+#### Language Support
+
+* **C++ benchmarks**: Full support for C++ on AWS Lambda with dependency management system (#99, #251)
+  - Docker-based build system with dependency caching
+  - Dynamic dependency resolution with CMake generation
+  - Support for Boost, OpenCV, igraph, PyTorch, hiredis libraries
+  - C++ implementations: 010.sleep, 210.thumbnailer, 501.graph-pagerank, 503.graph-bfs, 411.image-recognition
+* **Python**: Updated support for Python 3.8, 3.9, 3.10, 3.11, 3.12
+* **Node.js**: Updated support for Node.js 14, 16, 18, 20
+
+#### NoSQL Database Support
+
+* Complete NoSQL storage integration across platforms (#214)
+  - AWS: DynamoDB support with query interface
+  - Azure: CosmosDB integration
+  - GCP: Cloud Datastore support
+  - Local/OpenWhisk: ScyllaDB for local testing
+* New CRUD API benchmark (130.crud-api) demonstrating NoSQL operations (#214)
+* Multi-tier storage system with both object and NoSQL storage
+
+#### Platform-Specific Enhancements
+
+* **AWS**:
+  - Container deployment with ECR integration
+  - Support for ARM64 Lambda functions
+  - DynamoDB table management
+  - Updated Lambda runtime support (#166)
+* **Azure**:
+  - CosmosDB database management
+  - Improved HTTP trigger handling
+* **GCP**:
+  - Datastore database management
+  - Updated dependency versions for Python 3.10+
+  - Custom deployment waiters
+* **Local**:
+  - ScyllaDB wrapper for NoSQL database 
+  - Improved container lifecycle management
+  - Memory measurement improvements
+* **OpenWhisk**:
+  - ScyllaDB wrapper for NoSQL database 
+  - Adapted to new container build API
+  - Downloading metrics 
+
+### Bug Fixes
+
+* Fix init.sh to quote variables and add curl fallback (#287)
+* Fix bug in type serialization (#264)
+* Add SeBS user agent to 120.uploader (#255)
+* Fix GCP and local deployment issues (#252)
+* Fix local deployment invocation issues (#231, #249)
+* Fix invocation overhead experiment (#240)
+* Fix storage connection timeout on non-Linux platforms (#197)
+* Fix PyTorch benchmark support for Python 3.8 and 3.9 (#165)
+* Fix memory measurements on local deployment (#136)
+* Fix incorrect igraph version (#113)
+* Fix cache handling for code packages and containers
+
+### Improvements
+
+* Comprehensive docstrings across codebase (#244)
+* Sphinx-based HTML documentation with API reference (#244)
+* Update linting process (#241)
+* Dynamic port mapping for function containers in local deployment (#199)
+* Single-bucket design for cloud storage (#186)
+* Improved handling of cloud credentials (#181)
+* Invocation statistics reporting for benchmark and experiment results
+* Improved logging throughout the system
+* Improved storage and benchmarks documentation
+* Versioning for Docker build images
+* Support for multiple Azure subscriptions
+* Enhanced regression test system with container and ARM support
+
+### Deprecations
+
+* Python 3.6 no longer supported on all platforms
+* Node.js 8, 10, 12 deprecated on various platforms
+* Older runtime versions phased out across AWS, Azure, and GCP.
+
+### Contributors
+
+This release includes contributions from:
+* @userlaurin - multi-platform robustness (#287)
+* @DJAntivenom - C++ benchmark bugfixes (#264)
+* @HoriaMercan - C++benchmarks (#251)
+* @rabbull - GCP bug fixes (#252), local deployment fixes (#249)
+* @qdelamea-aneo - Fix invocation overhead experiment (#240)
+* @ojninja16 - Versioning and resource IDs (#232)
+* @MahadMuhammad - Fixes in local deployment (#231)
+* @aidenh6307 - Local documentation updates (#210)
+* @prajinkhadka - Container support for AWS (#205)
+* @octonawish-akcodes - Local deployment and benchmark version improvements (#198)
+* @Kaleab-git - Dynamic port mapping (#199), storage timeout fix (#197)
+* @nurSaadat - Documentation improvements (#175)
+* @lawrence910426 - Colored CLI output (#141)
+* @alevy - Documentation improvements (#139)
+* @skehrli - Local memory measurements (#101)
+* And many others who contributed bug reports, testing, and feedback!
+
+## [1.1.0](https://github.com/spcl/serverless-benchmarks/compare/v1.0...v1.1) (2022-05-30)
+
+### Features
+
+* Support for the open-source FaaS platform OpenWhisk.
+* New system of handling non-root containers that do not require rebuilding Docker images.
+* Initial release of deploying functions as Docker containers, first released on OpenWhisk.
+* Support for function states on AWS for correct deployment and configuration updates.
+
+### Improvements
+
+* Deprecate Python 3.6 on all platforms.
+* Update documentation and tutorials.
+* Docker build images for AWS now use official AWS images.
+* AWS Lambda functions now support Python 3.9 and Node.js 14. Python 3.6 and Node.js 8 are no longer supported.
+* Azure Functions support now Python 3.8 and Python 3.9, and Node.js 12 and 14. Python 3.6, Node.js 8, and 10 are no longer supported.
+* Google Cloud Functions support now Node 12 and 14. Node 6 and 8 are no longer supported.
+* OpenWhisk supports Python 3.7 and 3.9, Node.js 10 and 12.

README.md

@@ -1,5 +1,6 @@
 
 [![CircleCI](https://circleci.com/gh/spcl/serverless-benchmarks.svg?style=shield)](https://circleci.com/gh/spcl/serverless-benchmarks)
+[![Documentation Status](https://readthedocs.org/projects/sebs/badge/?version=latest)](https://sebs.readthedocs.io/en/latest/?badge=latest)
 ![Release](https://img.shields.io/github/v/release/spcl/serverless-benchmarks)
 ![License](https://img.shields.io/github/license/spcl/serverless-benchmarks)
 ![GitHub issues](https://img.shields.io/github/issues/spcl/serverless-benchmarks)
@@ -52,12 +53,19 @@ documentation:
 * [How SeBS package is designed?](docs/design.md)
 * [How to extend SeBS with new benchmarks, experiments, and platforms?](docs/modularity.md)
 
-### Publication
+## Tutorial
 
-When using SeBS, please cite our [Middleware '21 paper](https://dl.acm.org/doi/abs/10.1145/3464298.3476133).
+We provide a tutorial on basic SeBS functionality in the [SeBS-Tutorial repository](https://github.com/spcl/sebs-tutorial.git).
+You can learn there how to install SeBS, configure it, deploy OpenWhisk on your system, and launch your first experiments.
+
+## Publications
+
+When using SeBS, please cite our published work.
+You can cite our software repository as well, using the citation button on the right.
+
+SeBS has been originally released with the [Middleware '21 paper](https://dl.acm.org/doi/abs/10.1145/3464298.3476133).
 An extended version of our paper is [available on arXiv](https://arxiv.org/abs/2012.14132), and you can
 find more details about research work [in this paper summary](https://mcopik.github.io/projects/sebs/).
-You can cite our software repository as well, using the citation button on the right.
 
 ```
 @inproceedings{copik2021sebs,
@@ -78,6 +86,50 @@ You can cite our software repository as well, using the citation button on the r
 }
 ```
 
+The SeBS-Flow paper published at [EuroSys'25](https://dl.acm.org/doi/abs/10.1145/3689031.3717465)
+extends SeBS with support for serverless workflows and NoSQL database:
+
+```
+@inproceedings{10.1145/3689031.3717465,
+  author = {Schmid, Larissa and Copik, Marcin and Calotoiu, Alexandru and Brandner, Laurin and Koziolek, Anne and Hoefler, Torsten},
+  title = {SeBS-Flow: Benchmarking Serverless Cloud Function Workflows},
+  year = {2025},
+  isbn = {9798400711961},
+  publisher = {Association for Computing Machinery},
+  address = {New York, NY, USA},
+  url = {https://doi.org/10.1145/3689031.3717465},
+  doi = {10.1145/3689031.3717465},
+  booktitle = {Proceedings of the Twentieth European Conference on Computer Systems},
+  pages = {902–920},
+  numpages = {19},
+  keywords = {benchmark, faas, function-as-a-service, orchestration, serverless, serverless DAG, workflow},
+  location = {Rotterdam, Netherlands},
+  series = {EuroSys '25}
+}
+```
+
+The SeBS 2.0 workshop paper published at [SESAME @ EuroSys'25](https://dl.acm.org/doi/abs/10.1145/3721465.3721867)
+provides an overview of new and ongoing contributions to SeBS - benchmarks, platforms, languages.
+
+```
+@inproceedings{10.1145/3721465.3721867,
+  author = {Copik, Marcin and Calotoiu, Alexandru and Hoefler, Torsten},
+  title = {SeBS 2.0: Keeping up with the Clouds},
+  year = {2025},
+  isbn = {9798400715570},
+  publisher = {Association for Computing Machinery},
+  address = {New York, NY, USA},
+  url = {https://doi.org/10.1145/3721465.3721867},
+  doi = {10.1145/3721465.3721867},
+  booktitle = {Proceedings of the 3rd Workshop on SErverless Systems, Applications and MEthodologies},
+  pages = {42–44},
+  numpages = {3},
+  keywords = {Benchmark, FaaS, Function-as-a-Service, Serverless},
+  location = {Rotterdam, Netherlands},
+  series = {SESAME' 25}
+}
+```
+
 ## Installation
 
 Requirements:

benchmarks/100.webapps/110.dynamic-html/README.md

@@ -0,0 +1,9 @@
+# 110.dynamic-html - Dynamic HTML
+
+**Type:** Webapps
+**Languages:** Python, Node.js
+**Architecture:** x64, arm64
+
+## Description
+
+The benchmark represents a dynamic generation of webpage contents through a serverless function. It generates an HTML from an existing template, with random numbers inserted to control the output. It uses the `jinja2` and `mustache` libraries on Python and Node.js, respectively.

benchmarks/100.webapps/120.uploader/README.md

@@ -0,0 +1,9 @@
+# 120.uploader - Uploader
+
+**Type:** Webapps
+**Languages:** Python, Node.js
+**Architecture:** x64, arm64
+
+## Description
+
+The benchmark implements the common workflow of uploading user-defined data to the persistent cloud storage. It accepts a URL, downloads file contents, and uploads them to the storage. Python implementation uses the standard library `requests`, while the Node.js version uses the third-party `requests` library installed with `npm`.

benchmarks/100.webapps/130.crud-api/README.md

@@ -0,0 +1,9 @@
+# 130.crud-api - CRUD API
+
+**Type:** Webapps
+**Languages:** Python
+**Architecture:** x64, arm64
+
+## Description
+
+The benchmark implements a simple CRUD application simulating a webstore cart. It offers three basic methods: add new item (`PUT`), get an item (`GET`), and query all items in a cart. It uses the NoSQL storage, with each item stored using cart id as primary key and item id as secondary key. The Python implementation uses cloud-native libraries to access the database.

benchmarks/200.multimedia/210.thumbnailer/README.md

@@ -0,0 +1,9 @@
+# 210.thumbnailer - Thumbnailer
+
+**Type:** Multimedia
+**Languages:** Python, Node.js
+**Architecture:** x64, arm64
+
+## Description
+
+This benchmark implements one of the most common serverless workloads. It downloads an image from the cloud storage, resizes it to a thumbnail size, uploads the new smaller version to the cloud storage, and returns the location to the caller, allowing them to insert the newly created thumbnail. To resize the image, it uses the `Pillow` and `sharp` libraries on Python and Node.js, respectively.

benchmarks/200.multimedia/220.video-processing/README.md

@@ -0,0 +1,9 @@
+# 220.video-processing - Video Processing
+
+**Type:** Multimedia
+**Languages:** Python
+**Architecture:** x64, arm64
+
+## Description
+
+The benchmark implements two operations on video files: adding a watermark and creating a gif. Both input and output media are passed through the cloud storage. To process the video, the benchmark uses `ffmpeg`. The benchmark installs the most recent static binary of `ffmpeg` provided by [John van Sickle](https://johnvansickle.com/ffmpeg/).

benchmarks/300.utilities/311.compression/README.md

@@ -0,0 +1,9 @@
+# 311.compression - Compression
+
+**Type:** Utilities
+**Languages:** Python
+**Architecture:** x64, arm64
+
+## Description
+
+The benchmark implements a common functionality of websites managing file operations - gather a set of files in cloud storage, compress them together, and return a single archive to the user. It implements the .zip file creation with the help of the `shutil` standard library in Python.

benchmarks/400.inference/411.image-recognition/README.md

@@ -0,0 +1,21 @@
+# 411.image-recognition - Image Recognition
+
+**Type:** Inference
+**Languages:** Python
+**Architecture:** x64
+
+## Description
+
+The benchmark is inspired by MLPerf and implements image recognition with Resnet50. It downloads the input and model from the storage and uses the CPU-only `pytorch` library in Python.
+
+## Important Notes
+
+> [!WARNING]
+> This benchmark contains PyTorch which is often too large to fit into a code package. Up to Python 3.7, we can directly ship the dependencies. For Python 3.8, we use an additional zipping step that requires additional setup during the first run, making cold invocations slower. Warm invocations are not affected.
+
+> [!WARNING]
+> This benchmark does not work on AWS with Python 3.9 due to excessive code size. While it is possible to ship the benchmark by zipping `torchvision` and `numpy` (see `benchmarks/400.inference/411.image-recognition/python/package.sh`), this significantly affects cold startup. On the lowest supported memory configuration of 512 MB, the cold startup can reach 30 seconds, making HTTP trigger unusable due to 30 second timeout of API gateway. Use Docker deployments for these configurations.
+
+> [!WARNING]
+> This benchmark does not work on GCP with Python 3.8+ due to excessive code size. To the best of our knowledge, there is no way of circumventing that limit, as Google Cloud offers neither layers nor custom Docker images.
+

benchmarks/500.scientific/501.graph-pagerank/README.md

@@ -0,0 +1,9 @@
+# 501.graph-pagerank - Graph PageRank
+
+**Type:** Scientific
+**Languages:** Python
+**Architecture:** x64, arm64
+
+## Description
+
+The benchmark represents scientific computations offloaded to serverless functions. It uses the `python-igraph` library to generate an input graph and process it with the PageRank algorithm.

benchmarks/500.scientific/502.graph-mst/README.md

@@ -0,0 +1,9 @@
+# 502.graph-mst - Graph MST
+
+**Type:** Scientific
+**Languages:** Python
+**Architecture:** x64, arm64
+
+## Description
+
+The benchmark represents scientific computations offloaded to serverless functions. It uses the `python-igraph` library to generate an input graph and process it with the Minimum Spanning Tree (MST) algorithm.

benchmarks/500.scientific/503.graph-bfs/README.md

@@ -0,0 +1,9 @@
+# 503.graph-bfs - Graph BFS
+
+**Type:** Scientific
+**Languages:** Python
+**Architecture:** x64, arm64
+
+## Description
+
+The benchmark represents scientific computations offloaded to serverless functions. It uses the `python-igraph` library to generate an input graph and process it with the Breadth-First Search (BFS) algorithm.

benchmarks/500.scientific/504.dna-visualisation/README.md

@@ -0,0 +1,9 @@
+# 504.dna-visualisation - DNA Visualization
+
+**Type:** Scientific
+**Languages:** Python
+**Architecture:** x64, arm64
+
+## Description
+
+This benchmark is inspired by the [DNAVisualization](https://github.com/Benjamin-Lee/DNAvisualization.org) project and it implements processing the `.fasta` file with the `squiggle` Python library.

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)