diff --git a/.github/workflows/check_python_package.yaml b/.github/workflows/check_python_package.yaml new file mode 100644 index 0000000..5bd82ab --- /dev/null +++ b/.github/workflows/check_python_package.yaml @@ -0,0 +1,33 @@ +name: Check + +on: + push: + pull_request: + branches: ["main"] + +jobs: + test: + strategy: + fail-fast: false + matrix: + runs-on: [ubuntu-latest, macos-latest, windows-latest] + python-version: ["3.10", "3.11", "3.12", "3.13"] + runs-on: ${{matrix.runs-on}} + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - name: Install uv + uses: astral-sh/setup-uv@v7 + with: + python-version: ${{ matrix.python-version }} + + - name: Install the project + run: uv sync --all-extras --dev + + - name: Lint with ruff + run: uv run ruff check --output-format=github . + + - name: Run tests + run: uv run pytest --cov=opltools --cov-report=term-missing diff --git a/.github/workflows/run_to_html.yml b/.github/workflows/run_to_html.yml index de599d4..2d28e87 100644 --- a/.github/workflows/run_to_html.yml +++ b/.github/workflows/run_to_html.yml @@ -16,25 +16,16 @@ jobs: uses: actions/checkout@v5 with: ref: ${{ github.event.pull_request.head.ref }} - # Value already defaults to true, but `persist-credentials` is required to push new commits to the repository. persist-credentials: true - - - name: Set up Python - uses: actions/setup-python@v4 + - name: Install uv + uses: astral-sh/setup-uv@v7 with: - python-version: '3.x' - - - name: Install dependencies - run: | - python -m pip install --upgrade pip - pip install -r utils/requirements.txt - + python-version: "3.10" + - name: Install the project + run: uv sync --all-extras --dev - name: Generate html - run: | - python yaml_to_html.py - + run: uv run yaml_to_html.py - name: Commit changes uses: stefanzweifel/git-auto-commit-action@v7 with: - commit_message: "Auto-update HTML files" - + commit_message: "chore: Auto-update HTML files" diff --git a/SCHEMA.md b/SCHEMA.md new file mode 100644 index 0000000..50de382 --- /dev/null +++ b/SCHEMA.md @@ -0,0 +1,307 @@ +# OPL Schema + +The OPL schema catalogs optimization **problems**, **suites**, **generators**, and their **implementations** in a single, machine-readable format. + +Three design choices shape everything below: + +1. **One flat library, keyed by ID.** + Every entity lives in a `Library` dict. + Suites reference problems, problems reference implementations using their respective ID. + There is no embedding of problems or implementations within suites to facilitate reuse. + F.e. an implementation might be referenced by multiple problems or suites. +2. **Numeric fields accept a scalar, a set, or a range.** + A problem may have exactly `2` objectives, one of `{2, 3, 5}`, or any value in `{min: 2, max: 50}`. + The same union type is used for variable dimensions and constraint counts. +3. **Three-valued logic for yes/no fields.** + Many boolean fields (f.e. `hard`, `allows_partial_evaluation`, ...) [`YesNoSome`](#yesnosome) as their value. + We lose some expressive power but simplify the data entry. + If we force authors to decide on yes or no, then we would need more complex structures for variables, constraints etc. and that would make the usual case unnecessarily complex. + +## Contents + +- [Library](#library) +- [Thing types](#thing-types) + - [Implementation](#implementation) + - [ProblemLike](#problemlike) (shared fields) + - [Problem](#problem) + - [Suite](#suite) + - [Generator](#generator) +- [Shared building blocks](#shared-building-blocks) + - [Variable](#variable) / [VariableType](#variabletype) + - [Constraint](#constraint) / [ConstraintType](#constrainttype) + - [Reference](#reference) / [Link](#link) + - [ValueRange](#valuerange) + - [YesNoSome](#yesnosome) + +--- + +## Notation / Conventions + +- When an attribute is followed by a `?`, it is optional and can be left out. +- When we refer to a list of unique items, we call them a set. + Technically they are a set in Python, but in the YAML representation they are a list. + However, they _must_ be unique (i.e. obey the set property) + +## Library + +A `Library` is a dict from ID to a [Thing](#thing-types). +IDs are free-form but must be unique and the convention is to add a prefix marking the type to avoid collisions: + +| Prefix | Type | +|---------|------------------| +| `impl_` | Implementation | +| `fn_` | Problem | +| `suite_`| Suite | +| `gen_` | Generator | + +On load the library validates that every ID referenced by a suite (`problems`) or problem (`implementations`) exists and has the correct type. Suites also have their `fidelity_levels` auto-populated from their problems. + +```yaml +impl_coco: + type: implementation + name: COCO + description: Comparing Continuous Optimisers +fn_sphere: + type: problem + name: Sphere + objectives: [1] + implementations: [impl_coco] +suite_bbob: + type: suite + name: BBOB + problems: [fn_sphere] +``` + +--- + +## Thing types + +All entities inherit from `Thing`, which only carries a discriminator: + +```yaml +type: problem # or: suite | generator | implementation +``` + +We want to have as flat a structure as possible to make exploring and searching OPL as easy as possible. +That's one of the reasons the top level object is a dictionary of dissimilar things. +But we need to be able to tell them apart so we have a `type` field to discriminate between them. + +### Implementation + +A pointer to code that implements one or more problems. +Intentionally minimal so that the schema describes *what* a problem is, not how to run it. +There are separate files which contain curated usage examples for problems or suites keyed by their respective IDs. + +| Field | Type | Notes | +|-------------------|-----------------------------------|----------------------------------------------| +| `name` | str | required | +| `description` | str | required | +| `language` | str? (e.g. `Python`, `C`) | | +| `links` | list of [Link](#link)? | repo, release, docs… | +| `evaluation_time` | set of str? | free-form list ("8 minutes", "fast") | +| `requirements` | str or list of str? | URL to requirements file or list of packages | + +```yaml +impl_coco: + type: implementation + name: COCO + description: Comparing Continuous Optimisers benchmarking platform + language: c + links: + - {type: repository, url: https://github.com/numbbo/coco-experiment} +impl_py_cocoex: + type: implementation + name: Python bindings for COCO + description: The Python bindings for the experimental part of the COCO framework + language: Python + links: + - {type: source, url: https://github.com/numbbo/coco-experiment/tree/main/build/python} + - {type: package, url: https://pypi.org/project/coco-experiment/} +``` + +### ProblemLike + +Fields shared by [Problem](#problem), [Suite](#suite), and [Generator](#generator). +The schema deliberately puts most descriptive fields here so suites can be characterised without explicitly having to add all problems in the suite. + +| Field | Type | Notes | +|------------------------------------------|------------------------------------------------|----------------------------------------------------| +| `name` | str | required | +| `long_name` | str? | | +| `description` | str? (markdown) | longer prose | +| `tags` | set of str? | free-form keywords | +| `references` | set of [Reference](#reference)? | | +| `implementations` | set of IDs? | must resolve to [Implementation](#implementation)s | +| `objectives` | set of int? | e.g. `{1}`, `{2, 3}` — **not** a ValueRange | +| `variables` | set of [Variable](#variable)? | | +| `constraints` | set of [Constraint](#constraint)? | omit entirely for unconstrained | +| `dynamic_type` | set of str? | `{"no"}`, `{"time-varying"}`… | +| `noise_type` | set of str? | `{"none"}`, `{"gaussian"}`… | +| `allows_partial_evaluation` | [YesNoSome](#yesnosome)? | | +| `can_evaluate_objectives_independently` | [YesNoSome](#yesnosome)? | | +| `modality` | set of str? | `{"unimodal"}`, `{"multimodal"}` | +| `fidelity_levels` | set of int? | `{1}` = single-fidelity, `{1,2}` = multi-fidelity | +| `code_examples` | set of str? | paths to example scripts | +| `evaluation_time` | set of str? | free-form list ("8 minutes", "fast") | +| `source` | set of str? | `{"artificial"}`, `{"real-world"}` | + +> `objectives` is a set of integers because we don't assume extreme scalability in this property so explicit enumeration is fine. +> Dimensions of variables on the other hand are ranges because here problems often are scalable over wide ranges. + +When no `evaluation_time` is set, it percolates up from any referenced implementations. +The same is true for the `variables` and `constraints` properties of a suite that has references to problems. + +`fidelity_levels` is auto-unioned from member problems at validation time. + +### Problem + +One optimization problem (possibly parameterised by instances). + +Adds: + +| Field | Type | Notes | +|-------------|--------------------------------------------|--------------------------------------------| +| `instances` | [ValueRange](#valuerange) or list of str? | e.g. `{min: 1, max: 15}` or named variants | + +```yaml +fn_sphere: + type: problem + name: Sphere + objectives: [1] + variables: [{type: continuous, dim: {min: 2, max: 40}}] + modality: [unimodal] + source: [artificial] + instances: {min: 1, max: 15} + implementations: [impl_coco] +``` + +### Suite + +A curated, fixed collection of problems. + +Adds: + +| Field | Type | Notes | +|------------|--------------|-----------------------------------------------| +| `problems` | set of IDs? | must resolve to [Problem](#problem)s | + +```yaml +suite_bbob: + type: suite + name: BBOB + problems: [fn_sphere, fn_rosenbrock, fn_rastrigin] + objectives: [1] + source: [artificial] + implementations: [impl_coco] +``` + +### Generator + +A parametric family of problems — unlike a [Suite](#suite), the member problems are not enumerated. Uses the same fields as [ProblemLike](#problemlike) with no additions; the distinction from [Problem](#problem) is that a generator produces instances on demand. + +```yaml +gen_mpm2: + type: generator + name: MPM2 + description: Multiple peaks model, second instantiation + objectives: [1] + variables: [{type: continuous, dim: {min: 1}}] + modality: [multimodal] +``` + +--- + +## Shared building blocks + +### Variable + +A group of decision variables of the same type. +Multi-type problems list multiple entries. +While you can have multiple entries of the same type, this should be justified in some way like when you can evaluate the problem on only one subset of variables. + +| Field | Type | Default | +|--------|-----------------------------------------------|----------------------| +| `type` | [VariableType](#variabletype) | `unknown` | +| `dim` | int, set of int, [ValueRange](#valuerange), or null | `0` | + +```yaml +variables: + - {type: continuous, dim: 10} + - {type: integer, dim: {min: 1, max: 5}} +``` + +### VariableType + +`continuous | integer | binary | categorical | unknown`. +Use `unknown` for permutation/combinatorial problems the schema doesn't yet distinguish **and** add an appropriate tag. +We are actively watching for unknown variable types and are open to extending the above list if there is a critical mass of problems to justify it. + +### Constraint + +A group of constraints. +To indicate that the problem is unconstrained, you need an _empty_ `constraints` field. +A missing `constraints` field or if it is set to `null` means it is not known if unconstrained. + +| Field | Type | Notes | +|------------|-----------------------------------------------|------------------------------------| +| `type` | [ConstraintType](#constrainttype) | default `unknown` | +| `hard` | [YesNoSome](#yesnosome)? | hard vs. soft | +| `equality` | [YesNoSome](#yesnosome)? | equality vs. inequality | +| `number` | int, set of int, [ValueRange](#valuerange), null | | + +```yaml +constraints: + - {type: box, hard: yes, number: 10} + - {type: linear, hard: some, equality: no, number: {min: 1}} +``` + +### ConstraintType + +`box | linear | function | unknown`. `function` covers non-linear/black-box constraints. + +### Reference + +Bibliographic pointer. +Requires either a `title` or a `link` and optionally a list of `authors`. + +```yaml +references: + - title: "Evolutionsstrategie - Optimisierung technischer Systeme nach Prinzipien der biologischen Evolution" + authors: + - Ingo Rechenberg +``` + +### Link + +`{type?: str, url: str}`. +`type` is free-form (`repository`, `arxiv`, `paper`, `doi`, ...). +`url` is a URL to some resource. + +If `type` is `doi`, please use the full URL (starting with `https://doi.org/...`) instead of the raw DOI. + +### ValueRange + +An inclusive numeric range type. +At least one of `min`/`max` must be given. +If `min` is given and `max` is missing, it does not imply that there is no upper bound. +There might be one, it is just not known. +The same applies for the case where `max` is given and `min` is missing. + +```yaml +dim: {min: 2} # 2 or more +dim: {min: 2, max: 40} # between 2 and 40 +dim: {max: 100} # up to 100 +``` + +Used by `Variable.dim`, `Constraint.number`, `Problem.instances`. + +### YesNoSome + +Three-valued flag: `yes | no | some | ?` (the last serialises as the literal `'?'` string, meaning unknown). +`some` captures the common case where *part* of something has some property. +For example only some constraints might hard but we don't know the exact number of hard and soft constraints, only the total number. + +```yaml +constraints: [{type: box, hard: some}] +allows_partial_evaluation: "unknown" +``` diff --git a/docs/index.html b/docs/index.html index 78f6a70..d0e1c9b 100644 --- a/docs/index.html +++ b/docs/index.html @@ -26,1207 +26,3386 @@

OPL – Optimisation problem library

-
Visible columns
-
- - -
-
- -
+
+ Toggle Visible Columns +
+ + +
+
+ +
+
- - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + + - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - + - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - + - - - - - - - - - - - - - - + + + + + - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + - - - - - + + + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + + + - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + - - - - - - + + + + - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + - - - - - - + + + + - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + - + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + - - - - + + + + + + - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - + + - - + + + - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + -
nametextual descriptionsuite/generator/singleobjectivesdimensionalityvariable typeconstraintsdynamicnoisemulti-fidelitysource (real-world/artificial)referenceimplementationIDNameTypeVariable TypesTotal VariablesObjectivesPropertiesConstraint TypesTotal ConstraintsDynamicsNoisePartial EvaluationsIndependent ObjectivesFidelity LevelsFull NameDescriptionTagsReferencesImplementationsModalityEvaluation TimeExamplesSourceBinary VarsCategorical VarsContinuous VarsInteger VarsImplementation NamesImplementation LanguagesImplementation Evaluation TimesImplementation LinksImplementation DescriptionsImplementation RequirementsHard Box ConstraintsSoft Box ConstraintsHard Linear ConstraintsSoft Linear ConstraintsHard Function ConstraintsSoft Function Constraints
BBOBsuite1scalable
fn_atoATOProblem continuousnonononohttps://doi.org/10.1080/10556788.2020.1808977https://github.com/numbbo/coco
BBOB-biobjsuite10 22-40continuousnononono https://doi.org/10.48550/arXiv.1604.00359https://github.com/numbbo/coco
BBOB-noisy suite1scalablecontinuousnonoyesno https://hal.inria.fr/inria-00369466https://web.archive.org/web/20210416065610/https://coco.gforge.inria.fr/doku.php?id=downloads
BBOB-largescale suite120-640continuousnonono no https://doi.org/10.48550/arXiv.1903.06396https://github.com/numbbo/coco
BBOB-mixint suite15-160integer;continuous;mixednononono https://doi.org/10.1145/3321707.3321868https://github.com/numbbo/coco
BBOB-biobj-mixintParameters of the Modules of the Automatic Train Operation are optimized; two objectives: minimizing energy consumption and minimizing driving duration. suite25-160integer;continuous;mixednononono https://doi.org/10.1145/3321707.3321868https://github.com/numbbo/coco
BBOB-constrained suite12-40continuousyesnononounimodal http://numbbo.github.io/coco-doc/bbob-constrained/https://github.com/numbbo/coco
MOrepo suite2?combinatorial???noreal-world https://github.com/MCDMSociety/MOrepo
ZDT10 suite2scalablecontinuous;binarynononono https://doi.org/10.1162/106365600568202https://github.com/anyoptimization/pymoo
DTLZ suite2+scalablecontinuousnononono https://doi.org/10.1109/CEC.2002.1007032https://pymoo.org/problems/many/dtlz.html
WFG suite2+scalablecontinuousnononono https://doi.org/10.1109/TEVC.2005.861417https://pymoo.org/problems/many/wfg.html
CDMP suite2+scalablecontinuousyes??no https://doi.org/10.1145/3321707.3321878?
SDP suite2+scalablecontinuousnoyes?no https://doi.org/10.1109/TCYB.2019.2896021?
MaOP suite2+scalablecontinuousnono?no https://doi.org/10.1016/j.swevo.2019.02.003?
BP
fn_building_spatialBuilding spatial designProblembinary | continuous>=22 suite2+scalablecontinuousnono?nounknown | box>=2 https://doi.org/10.1109/CEC.2019.8790277?
GPD generator2+scalablecontinuousoptionalnooptional no https://doi.org/10.1016/j.asoc.2020.106139?
ETMOF suite2-5025-10000continuousnoyesnono https://doi.org/10.48550/arXiv.2110.08033https://github.com/songbai-liu/etmo
MMOPPOptimise the spatial layout of a building to minimise energy consumption for climate control and minimise the strain on the structure. Many hard constraints; mixed-variable (continuous+binary); expensive evaluations. suite2-7??yesnononoBuilding spatial design, https://hdl.handle.net/1887/81789impl_bso_toolbox http://www5.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1327567121&wbfileid=4764412http://www5.zzu.edu.cn/ecilab/info/1036/1251.htm
CFDexpensive evaluations 30s-15msuite1-2scalable?yesnononoreal worldhttps://doi.org/10.1007/978-3-319-99259-4_24https://bitbucket.org/arahat/cfd-test-problem-suite
GBEAexpensive evaluations 5s-35s, RW-GAN-Mario and TopTrumps are part of GBEAsuite1-2scalablecontinuousnonoyesnoreal worldhttps://doi.org/10.1145/3321707.3321805https://github.com/ttusar/coco-gbea
Car structure54 constraintssuite2144-222discreteyesnononoreal worldhttps://doi.org/10.1145/3205651.3205702http://ladse.eng.isas.jaxa.jp/benchmark/
EMO2017["1 second", "40 seconds"] suite24-24continuousnonononoreal worldhttps://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/downloads/realworld-problems-bbcomp-EMO-2017.zip
JSEC2019expensive evaluations 3s; 22 constraintssingle1-532continuousyesnononoreal worldhttp://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.htmlhttp://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html
REreal-world>=1 suite2-92-7continuous;integer;mixednonononoreal world likehttps://doi.org/10.1016/j.asoc.2020.106078https://github.com/ryojitanabe/reproblems
CRE>=1 suite2-53-7continuous;integer;mixedyesnononoreal world likehttps://doi.org/10.1016/j.asoc.2020.106078https://github.com/ryojitanabe/reproblems
Radar waveformBSO-toolboxC++{'40 seconds', '1 second'}https://github.com/TUe-excellent-buildings/BSO-toolboxBuilding Spatial Design toolbox (TU/e) single94-12integeryesnononoreal worldhttps://doi.org/10.1007/978-3-540-70928-2_53http://code.evanhughes.org/
MF2>=1 suite11-ncontinuousnononoyes https://doi.org/10.21105/joss.02049https://github.com/sjvrijn/mf2
AMVOP suite1scalablemixed continuous+ordinal+categorical+bothnononono https://doi.org/10.1109/TEVC.2013.2281531?
RWMVOP suite1scalablecontinuous;mixed continuous+ordinal+categorical+bothyesnononoreal worldhttps://doi.org/10.1109/TEVC.2013.2281531?
SBOX-COSTproblems from BBOB but allows instances with the optimum close to the boundarysuite1scalable
fn_convex_dtlz2Convex DTLZ2Problem continuousnononono>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]Variant of DTLZ2 with a convex Pareto front (instead of concave)Convex DTLZ2, https://doi.org/10.1109/TEVC.2013.2281535>=1 https://doi.org/10.48550/arXiv.2305.12221https://github.com/IOHprofiler/IOHexperimenter/
ρMNK-Landscapestunable variable and objective dimensions; tunable multimodality and correlation between objectivesgeneratorscalablescalablebinarynononono https://doi.org/10.1016/j.ejor.2012.12.019https://gitlab.com/aliefooghe/mocobench/
mUBQPtunable variable and objective dimensions; tunable density and correlation between objectivesgeneratorscalablescalablebinarynononono https://doi.org/10.1016/j.asoc.2013.11.008https://gitlab.com/aliefooghe/mocobench/
ρmTSPtunable variable and objective dimensions; tunable instance type (euclidian/random); tunable correlation between objectivesgeneratorscalablescalablepermutationsno (apart from being permutations)nonono https://doi.org/10.1007/978-3-319-45823-6_40https://gitlab.com/aliefooghe/mocobench/
CEC2015-DMOO suite2-3?continuous?yesnono Benchmark Functions for CEC 2015 Special Session and Competition on Dynamic Multi-objective Optimization
EalainReal-world-like, easily extensible to increase complexitygenerator1+scalablecontinuous,binary,integeroptionaloptionalnooptionalReal-world-likehttps://doi.org/10.1145/3638530.3654299https://github.com/qrenau/Ealain
MA-BBOBGenerator that creates affine combinations of BBOB functionsgenerator
fn_emdoElectric Motor Design OptimizationProbleminteger | continuous26 1scalablecontinuousnonononoisyunknown | box>=14noisy noartificialhttps://doi.org/10.1145/3673908https://github.com/IOHprofiler/IOHexperimenter/blob/master/example/Competitions/MA-BBOB/Example_MABBOB.ipynbElectric Motor Design Optimization# Goal\nFind a design of a synchronous electric motor for power steering systems that minimizes costs and satisfies all constraints.\n\n# Motivation\nChallenging to find good solutions in a limited time.\n\n# Key Challenges\n* Time-consuming solution evaluation\n* Highly-constrained problem\n* Constraints are multimodal\n\nThis is not an available problem, but could be interesting to show to researchers which difficulties appear in real-world problems.A Multi-Step Evaluation Process in Electric Motor Design, Tea Tušar; Peter Korošec; Bogdan Filipič, https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdfimpl_emdomultimodal8 minutesreal-world1313Electric Motor Design OptimizationPython{'8 minutes'}Not publicly available>=1
MPM2nonlinear nonseparable nonsymmetric; scalable in terms of time to evaluate the objective functiongenerator
fn_fleetoptFleetOptProbleminteger{54, 13208} 1scalablecontinuousnononono https://ls11-www.cs.tu-dortmund.de/_media/techreports/tr15-01.pdfhttps://github.com/jakobbossek/smoof/blob/master/inst/mpm2.py
Convex DTLZ2Variant of DTLZ2 with a convex Pareto front (instead of concave)single2+scalablecontinuousnonononounknown>=1yesUK healthcare organisation fleet optimisation: reduce the fleet of non-emergency healthcare trip vehicles while still ensuring all trips can be covered. Bilevel: upper level 54 vars, lower level 13208 vars.FleetOpt, https://dl.acm.org/doi/abs/10.1145/3638530.3664137real-world{54, 13208} https://doi.org/10.1109/TEVC.2013.2281535?
Inverted DTLZ1Variant of DTLZ1 with an inverted Pareto frontsingle2+scalablecontinuousnononono https://doi.org/10.1109/TEVC.2013.2281534?
Minus DTLZVariant of DTLZ that minimises the inverse of the base DTLZ functionssuite2+scalablecontinuousnononono https://doi.org/10.1109/TEVC.2016.2587749?
Minus WFGVariant of WFG that minimises the inverse of the base WFG functionssuite2+scalablecontinuousnononono https://doi.org/10.1109/TEVC.2016.2587749?
L1-ZDTVariant of ZDT with linkages between variables within one of two groups but not between variables in a different group; Linear recombination operators can potentially take advantage of the problem structuresuite2scalablecontinuous;binarynononono https://doi.org/10.1145/1143997.1144179?
L2-ZDTVariant of ZDT with linkages between all variables; Linear recombination operators can potentially take advantage of the problem structuresuite
fn_gasolineGasoline direct injection engine designProbleminteger | continuous14 2scalablecontinuous;binarynonononomulti-fidelityunknown5[1, 2]Multi-objective optimization to minimize fuel consumption and NOx emissions over a two-minute dynamic duty cycle, subject to five constraints (turbine inlet temperature, knock occurrences, peak cylinder pressure, peak cylinder pressure rise, total work). Seven decision variables cover hardware choices and engine control parameters.Gasoline direct injection engine design, https://doi.org/10.1016/j.ejor.2022.08.032impl_gasoline[]real-world77Gasoline direct injection engine designMatlab Simulink / Wave RThttps://doi.org/10.1016/j.ejor.2022.08.032Proprietary Matlab Simulink + Wave RT co-simulation https://doi.org/10.1145/1143997.1144179?
L3-ZDTVariant of L2-ZDT using a mapping to prevent linear recombination operators from potentially taking advantage of the problem structuresuite
fn_invdeceptive_deceptive_rotellInverseDeceptiveTrap+RotatedEllipsoid / DeceptiveTrap+RotatedEllipsoidProblembinary | continuous>=2 2scalablecontinuous;binarynononono https://doi.org/10.1145/1143997.1144179?
L2-DTLZVariant of DTLZ2 and DTLZ3 with linkages between all variables; Linear recombination operators can potentially take advantage of the problem structuresuite2+scalablecontinuousnononono https://doi.org/10.1145/1143997.1144179?
L3-DTLZVariant of L2-DTLZ using a mapping to prevent linear recombination operators from potentially taking advantage of the problem structuresuite2+scalablecontinuousnononono https://doi.org/10.1145/1143997.1144179?
CEC2018 DT - CEC2018 Competition on Dynamic Multiobjective Optimisation14 problems. Time-dependent: Pareto front/Pareto set geometry; irregular Pareto front shapes; variable-linkage; number of disconnected Pareto front segments; etc.suite2 or 3scalable??noyesnonoMixed-variable multi-objective test problems, https://doi.org/10.1145/3449726.3459521 artificialhttps://www.academia.edu/download/94499025/TR-CEC2018-DMOP-Competition.pdfhttps://pymoo.org/problems/dynamic/df.html
MODAct - multiobjective design of actuatorsRealistic Constrained Multi-Objective Optimization Benchmark Problems from Design. Need the https://github.com/epfl-lamd/modact package installed; evaluation times around 20mssuite2 3 4 or 520mixed; integer and continuousyesnononoreal-worldhttps://doi.org/10.1109/TEVC.2020.3020046https://pymoo.org/problems/constrained/modact.html
IOHClusteringSet of benchmark problems from clustering: optimization task is selecting cluster centers for a given set of data, with the number of clusters defining problem dimensionality. Includes both a suite and a generator. Based on ML clustering datasetssuite; generator1scalablecontinuousnonononoartificial, but based on real datahttps://arxiv.org/pdf/2505.09233https://github.com/IOHprofiler/IOHClustering>=1>=1
GNBG-IIGeneralized Numerical Benchmark Generator (version 2). Also in IOH https://github.com/IOHprofiler/IOHGNBGsuite; generator1scalable
fn_inverted_dtlz1Inverted DTLZ1Problem continuousnonononoartificialhttps://dl.acm.org/doi/pdf/10.1145/3712255.3734271https://github.com/rohitsalgotra/GNBG-II>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]Variant of DTLZ1 with an inverted Pareto frontInverted DTLZ1, https://doi.org/10.1109/TEVC.2013.2281534>=1
GNBGGeneralized Numerical Benchmark Generatorsuite; generator1scalable
fn_jsec2019JSEC2019Problem continuousnononono32[1, 2, 3, 4, 5]unknown22expensive evaluations 3s; 22 constraintsJPNSEC EC-Symposium 2019 competition, http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.htmlimpl_jsec20193sreal-world32JSEC 2019 competition{'3s'}http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.htmlJPNSEC EC-Symposium 2019 competition problem
fn_onemax_sphere_deceptive_rotellOnemax+Sphere / DeceptiveTrap+RotatedEllipsoidProblembinary | continuous>=22Mixed-variable multi-objective test problems, https://doi.org/10.1145/3449726.3459521 artificialhttps://arxiv.org/abs/2312.07083https://github.com/Danial-Yazdani/GNBG-Generator>=1>=1
DynamicBinValFour versions of the dynamic binary value problemsuite1scalablebinarynoyesnono
fn_onemax_sphere_zeromax_sphereOnemax+Sphere / Zeromax+SphereProblembinary | continuous>=22Onemax+Sphere / Zeromax+Sphere, https://doi.org/10.1145/3449726.3459521 artificialhttps://arxiv.org/pdf/2404.15837https://github.com/IOHprofiler/IOHexperimenter>=1>=1
PBOSuite of 25 binary optimization problemssuite1scalablebinarynonono
fn_radar_waveformRadar waveformProbleminteger4-129unknown>=1Radar waveform design, https://doi.org/10.1007/978-3-540-70928-2_53impl_radar_waveform[]real-world4-12Evan Hughes radar waveform codehttp://code.evanhughes.org/Radar waveform design reference implementation
gen_beaconBEACONGeneratorcontinuous>=12box0 noContinuous Bi-objective Benchmark problems with Explicit Adjustable COrrelatioN controlGenerator for bi-objective benchmark problems with explicitly controlled correlations in continuous spaces. Multimodal with random structure.BEACON, https://dl.acm.org/doi/10.1145/3712255.3734303impl_beaconmultimodalnegligible artificialhttps://dl.acm.org/doi/pdf/10.1145/3319619.3326810https://github.com/IOHprofiler/IOHexperimenter>=1BEACONPython{'negligible'}https://github.com/Stebbet/BEACON/Continuous Bi-objective Benchmark with Explicit Adjustable COrrelatioN control0
W-modelTunable generator for binary optimization based on several difficulty featuresgenerator1scalablebinarynonono
gen_bono_benchBONO-BenchGeneratorcontinuous>=12box>=1 noBi-objective Numerical Optimization BenchmarkBi-objective problem generator and suite with scalable continuous decision space. Features complex problem properties and Pareto front approximations with error guarantees for the hypervolume and exact R2 indicators.impl_bonobenchmultimodal[] artificialhttps://dl.acm.org/doi/abs/10.1145/3205651.3208240?casa_token=S4U_Pi9f6MwAAAAA:U9ztNTPwmupT8K3GamWZfBL7-8fqjxPtr_kprv51vdwA-REsp0EyOFGa99BtbANb0XbqyrVg795hIwhttps://github.com/thomasWeise/BBDOB_W_Model>=1BONO-BenchPythonhttps://github.com/schaepermeier/bonobenchBi-objective Numerical Optimization Benchmark (BONO-Bench)>=1
Submodular Optimitzationset of graph-based submodular optimization problems from 4 problem typessuite1scalablebinarynononono
gen_ealainEalainGeneratorbinary | continuous | integer>=3[1, 10, 2, 3, 4, 5, 6, 7, 8, 9]dynamic | multi-fidelityunknown>=1optional[1, 2]Real-world-like, easily extensible to increase complexityEalain, https://doi.org/10.1145/3638530.3654299impl_ealain[]real-world-like>=1>=1>=1Ealainhttps://github.com/qrenau/EalainReal-world-like extensible benchmark problem generator
gen_gnbgGNBGGeneratorcontinuous>=11Generator counterpart of GNBG.GNBG, https://arxiv.org/abs/2312.07083impl_gnbg[]artificial>=1GNBG Generatorhttps://github.com/Danial-Yazdani/GNBG-GeneratorGeneralized Numerical Benchmark Generator
gen_gnbg_iiGNBG-IIGeneratorcontinuous>=11Generator counterpart of GNBG-II.GNBG-II, https://dl.acm.org/doi/pdf/10.1145/3712255.3734271["impl_gnbg_ii", "impl_iohgnbg"][]artificial>=1IOHGNBG | GNBG-IIhttps://github.com/IOHprofiler/IOHGNBG | https://github.com/rohitsalgotra/GNBG-IIIOHprofiler version of GNBG | Generalized Numerical Benchmark Generator version 2
gen_gpdGPDGeneratorcontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]noisyunknown>=1optionalGPD generator, https://doi.org/10.1016/j.asoc.2020.106139>=1
gen_iohclusteringIOHClusteringGeneratorcontinuous>=11Generator counterpart of the IOHClustering suite.IOHClustering, https://arxiv.org/pdf/2505.09233impl_iohclusteringmultimodal[]artificial-from-real-data>=1IOHClusteringhttps://github.com/IOHprofiler/IOHClusteringClustering-based optimization benchmark built on ML datasets
gen_ma_bbobMA-BBOBGeneratorcontinuous>=11Generator that creates affine combinations of BBOB functionsMA-BBOB, https://doi.org/10.1145/3673908["impl_iohexperimenter", "impl_ma_bbob"]multimodal[]artificial>=1MA-BBOB (IOHexperimenter) | IOHexperimenterC++/Pythonhttps://github.com/IOHprofiler/IOHexperimenter/blob/master/example/Competitions/MA-BBOB/Example_MABBOB.ipynb | https://github.com/IOHprofiler/IOHexperimenterExample notebook for MA-BBOB in IOHexperimenter | IOHprofiler experimenter framework
gen_mpm2MPM2Generatorcontinuous>=11nonlinear nonseparable nonsymmetric; scalable in terms of time to evaluate the objective functionMPM2 technical report TR15-01, https://ls11-www.cs.tu-dortmund.de/_media/techreports/tr15-01.pdfimpl_mpm2multimodal[]>=1MPM2 (smoof)Pythonhttps://github.com/jakobbossek/smoof/blob/master/inst/mpm2.pyPython implementation of MPM2 distributed with smoof
gen_mubqpmUBQPGeneratorbinary>=1[1, 10, 2, 3, 4, 5, 6, 7, 8, 9]tunable variable and objective dimensions; tunable density and correlation between objectivesmUBQP benchmark, https://doi.org/10.1016/j.asoc.2013.11.008impl_mocobench["multimodal", "quadratic"][]>=1mocobenchC++https://gitlab.com/aliefooghe/mocobench/Multi-objective combinatorial optimization benchmark
gen_puboiPUBOiGeneratorbinary>=11noPolynomial Unconstrained Binary Optimization with tunable importanceA benchmark in which variable importance is tunable, based on the Walsh function.PUBOi, https://link.springer.com/chapter/10.1007/978-3-031-04148-8_12impl_puboi[]artificial>=1PUBO Importance BenchmarkPython / C++https://gitlab.com/verel/pubo-importance-benchmarkA benchmark in which variable importance is tunable, based on the Walsh function
gen_randoptgenRandOptGenGeneratorbinary | continuous | integer>=3[1, 10, 2, 3, 4, 5, 6, 7, 8, 9]noRandOptGenA Unified Random Problem Generator for Single- and Multi-Objective Optimization Problems with Mixed-Variable Input Spaces.impl_randoptgenmultimodalmillisecondsartificial>=1>=1>=1RandOptGenPython{'milliseconds'}https://github.com/MALEO-research-group/RandOptGen | https://doi.org/10.1145/3712256.3726478Unified Random Problem Generator for Single- and Multi-Objective Optimization with Mixed-Variable Input Spaces
gen_rho_mnk_landscapesρMNK-LandscapesGeneratorbinary>=1[1, 10, 2, 3, 4, 5, 6, 7, 8, 9]tunable variable and objective dimensions; tunable multimodality and correlation between objectivesOn the design of multi-objective evolutionary algorithms based on NK-landscapes, https://doi.org/10.1016/j.ejor.2012.12.019impl_mocobenchmultimodal[]>=1mocobenchC++https://gitlab.com/aliefooghe/mocobench/Multi-objective combinatorial optimization benchmark
gen_rho_mtspρmTSPGeneratorunknown>=1[1, 10, 2, 3, 4, 5, 6, 7, 8, 9]tunable variable and objective dimensions; tunable instance type (euclidean/random); tunable correlation between objectivesOn the impact of multi-objective scalability for the ρmTSP, https://doi.org/10.1007/978-3-319-45823-6_40impl_mocobench["multimodal", "quadratic"][]mocobenchC++https://gitlab.com/aliefooghe/mocobench/Multi-objective combinatorial optimization benchmark
gen_wmodelW-modelGeneratorbinary>=11Tunable generator for binary optimization based on several difficulty featuresW-model, https://dl.acm.org/doi/abs/10.1145/3205651.3208240impl_wmodel[]artificial>=1BBDOB W-Modelhttps://github.com/thomasWeise/BBDOB_W_ModelTunable generator for binary optimization
suite_amvopAMVOPSuitecategorical | continuous | integer>=31AMVOP, https://doi.org/10.1109/TEVC.2013.2281531multimodal>=1>=1>=1
suite_bbobBBOBSuitecontinuous>=11COCO: a platform for comparing continuous optimizers in a black-box setting, https://doi.org/10.1080/10556788.2020.1808977impl_cocomultimodal>=1COCO frameworkC/Pythonhttps://github.com/numbbo/cocoComparing Continuous Optimizers: black-box optimization benchmarking platform
suite_bbob_biobjBBOB-biobjSuitecontinuous2-402BBOB bi-objective test suite, https://doi.org/10.48550/arXiv.1604.00359impl_cocomultimodal2-40COCO frameworkC/Pythonhttps://github.com/numbbo/cocoComparing Continuous Optimizers: black-box optimization benchmarking platform
suite_bbob_biobj_mixintBBOB-biobj-mixintSuiteinteger | continuous10-3202BBOB bi-objective mixed-integer test suite, https://doi.org/10.1145/3321707.3321868impl_cocomultimodal5-1605-160COCO frameworkC/Pythonhttps://github.com/numbbo/cocoComparing Continuous Optimizers: black-box optimization benchmarking platform
suite_bbob_constrainedBBOB-constrainedSuitecontinuous2-401unknown>=1bbob-constrained documentation, http://numbbo.github.io/coco-doc/bbob-constrained/impl_cocomultimodal2-40COCO frameworkC/Pythonhttps://github.com/numbbo/cocoComparing Continuous Optimizers: black-box optimization benchmarking platform
suite_bbob_largescaleBBOB-largescaleSuitecontinuous20-6401BBOB large-scale test suite, https://doi.org/10.48550/arXiv.1903.06396impl_cocomultimodal20-640COCO frameworkC/Pythonhttps://github.com/numbbo/cocoComparing Continuous Optimizers: black-box optimization benchmarking platform
suite_bbob_mixintBBOB-mixintSuiteinteger | continuous10-3201BBOB mixed-integer test suite, https://doi.org/10.1145/3321707.3321868impl_cocomultimodal5-1605-160COCO frameworkC/Pythonhttps://github.com/numbbo/cocoComparing Continuous Optimizers: black-box optimization benchmarking platform
suite_bbob_noisyBBOB-noisySuitecontinuous>=11noisynoisyReal-parameter black-box optimization benchmarking: noisy functions definitions, https://hal.inria.fr/inria-00369466impl_coco_legacymultimodal>=1COCO legacy (bbob-noisy)C/Pythonhttps://web.archive.org/web/20210416065610/https://coco.gforge.inria.fr/doku.php?id=downloadsArchived COCO download page that hosted the bbob-noisy suite
suite_bpBPSuitecontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]noisyunknownBP benchmark, https://doi.org/10.1109/CEC.2019.8790277>=1
suite_brachytherapyBrachytherapy treatment planningSuitecontinuous100-500[2, 3]multi-fidelityunknown>=1yes[1, 2]Brachytherapy treatment planningTreatment planning for internal radiation therapy. Multi-objective with aggregated objectives; no public source code.Brachytherapy treatment planning, https://www.sciencedirect.com/science/article/pii/S1538472123016781multimodalreal-world100-500
suite_car_structureCar structureSuiteinteger144-2222unknown5454 constraintsCar structure design benchmark, https://doi.org/10.1145/3205651.3205702impl_car_structurereal-world144-222Car-structure benchmarkhttp://ladse.eng.isas.jaxa.jp/benchmark/JAXA LADSE benchmark problems
suite_cdmpCDMPSuitecontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]dynamic | noisyunknown>=1unknownunknownCDMP benchmark, https://doi.org/10.1145/3321707.3321878>=1
suite_cec2013CEC2013Suitecontinuous>=11suite used for cec2013 competition. Also in IOH.CEC2013 definitions, https://peerj.com/articles/cs-2671/CEC2013.pdf["impl_cec2013", "impl_iohexperimenter"]artificial>=1IOHexperimenter | CEC2013 reference codeC++/Pythonhttps://github.com/IOHprofiler/IOHexperimenter | https://github.com/P-N-Suganthan/CEC2013IOHprofiler experimenter framework | Suganthan's reference implementation
suite_cec2015_dmooCEC2015-DMOOSuitecontinuous0[2, 3]dynamicunknown>=1dynamicBenchmark Functions for CEC 2015 Special Session and Competition on Dynamic Multi-objective Optimization0
suite_cec2018_dtCEC2018 DTSuiteunknown>=1[2, 3]dynamicdynamicCEC2018 Competition on Dynamic Multiobjective Optimisation14 problems. Time-dependent: Pareto front/Pareto set geometry; irregular Pareto front shapes; variable-linkage; number of disconnected Pareto front segments; etc.CEC2018 DMOP Competition TR, https://www.academia.edu/download/94499025/TR-CEC2018-DMOP-Competition.pdfimpl_pymooartificialpymooPythonhttps://github.com/anyoptimization/pymooMulti-objective optimization in Python
suite_cec2022CEC2022Suitecontinuous>=11suite used for cec2022 competition. Also in IOH.CEC2022 TR, https://github.com/P-N-Suganthan/2022-SO-BO/blob/main/CEC2022%20TR.pdf["impl_cec2022", "impl_iohexperimenter"]artificial>=1IOHexperimenter | CEC2022 reference codeC++/Pythonhttps://github.com/IOHprofiler/IOHexperimenter | https://github.com/P-N-Suganthan/2022-SO-BOIOHprofiler experimenter framework | Suganthan's reference implementation
suite_cfdCFDSuiteunknown>=1[1, 2]unknown>=1expensive evaluations 30s-15mCFD test problem suite, https://doi.org/10.1007/978-3-319-99259-4_24impl_cfdreal-worldCFD test problem suite{'15m', '30s'}https://bitbucket.org/arahat/cfd-test-problem-suiteExpensive real-world CFD-based test problems
suite_creCRESuiteinteger | continuous6-14[2, 3, 4, 5]unknown>=1Easy-to-evaluate real-world multi-objective optimization problems, Ryoji Tanabe; Hisao Ishibuchi, https://doi.org/10.1016/j.asoc.2020.106078impl_reproblemsreal-world-like3-73-7reproblemsPythonhttps://github.com/ryojitanabe/reproblemsReal-world inspired multi-objective optimization problem suite
suite_cuterCUTErSuitebinary | continuous | integer>=31unknown>=1noA constrained and unconstrained testing environment.CUTEr, https://dl.acm.org/doi/10.1145/962437.962439artificial>=1>=1>=1
suite_cutestCUTEstSuitebinary | continuous | integer>=31unknown | box>=2noConstrained and Unconstrained Testing Environment with safe threadsCUTEst for optimization softwareCUTEst, https://link.springer.com/article/10.1007/s10589-014-9687-3impl_pycutestmultimodalartificial>=1>=1>=1pycutestPython / C++ / Fortranhttps://github.com/jfowkes/pycutestPython interface to CUTEst>=1
suite_dtlzDTLZSuitecontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]Scalable multi-objective optimization test problems, Kalyanmoy Deb; Lothar Thiele; Marco Laumanns; Eckart Zitzler, https://doi.org/10.1109/CEC.2002.1007032impl_pymoo>=1pymooPythonhttps://github.com/anyoptimization/pymooMulti-objective optimization in Python
suite_dynamicbinvalDynamicBinValSuitebinary>=11dynamicdynamicFour versions of the dynamic binary value problemDynamicBinVal, https://arxiv.org/pdf/2404.15837impl_iohexperimenterartificial>=1IOHexperimenterC++/Pythonhttps://github.com/IOHprofiler/IOHexperimenterIOHprofiler experimenter framework
suite_emo2017EMO2017Suitecontinuous4-242BBComp EMO 2017, https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/impl_emo2017real-world4-24EMO 2017 real-world problemshttps://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/downloads/realworld-problems-bbcomp-EMO-2017.zipBBComp EMO-2017 real-world problem archive
suite_etmofETMOFSuitecontinuous25-10000[10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 2, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 3, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 4, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 5, 50, 6, 7, 8, 9]dynamicdynamicEvolutionary many-task optimization framework, https://doi.org/10.48550/arXiv.2110.08033impl_etmof25-10000ETMOFhttps://github.com/songbai-liu/etmoEvolutionary many-task optimization framework
suite_expobenchEXPObenchSuitecontinuous | categorical | integer30-4051noisyunknown | box>=2["observational", "real-life"]noEXPensive Optimization benchmark libraryWind farm layout optimization, gas filter design, pipe shape optimization, hyperparameter tuning, and hospital simulationEXPObench, https://doi.org/10.1016/j.asoc.2023.110744impl_expobenchreal-world10-13510-13510-135EXPObenchPython{'2 seconds', '80 seconds'}https://github.com/AlgTUDelft/ExpensiveOptimBenchmarkEXPensive Optimization benchmark library (wind farm layout, gas filter design, pipe shape, hyperparameter tuning, hospital simulation)>=1
suite_gbeaGBEASuitecontinuous>=1[1, 2]noisynoisyexpensive evaluations 5s-35s, RW-GAN-Mario and TopTrumps are part of GBEAGame benchmark for evolutionary algorithms, https://doi.org/10.1145/3321707.3321805impl_gbeamultimodalreal-world>=1coco-gbea{'34 seconds', '5 seconds'}https://github.com/ttusar/coco-gbeaGame-Benchmark for Evolutionary Algorithms (COCO fork)
suite_gnbgGNBGSuitecontinuous>=11Generalized Numerical Benchmark GeneratorGNBG, https://arxiv.org/abs/2312.07083impl_gnbgartificial>=1GNBG Generatorhttps://github.com/Danial-Yazdani/GNBG-GeneratorGeneralized Numerical Benchmark Generator
suite_gnbg_iiGNBG-IISuitecontinuous>=11Generalized Numerical Benchmark Generator (version 2). Also available in IOH.GNBG-II, https://dl.acm.org/doi/pdf/10.1145/3712255.3734271["impl_gnbg_ii", "impl_iohgnbg"]artificial>=1IOHGNBG | GNBG-IIhttps://github.com/IOHprofiler/IOHGNBG | https://github.com/rohitsalgotra/GNBG-IIIOHprofiler version of GNBG | Generalized Numerical Benchmark Generator version 2
suite_iohclusteringIOHClusteringSuitecontinuous>=11Set of benchmark problems from clustering: optimization task is selecting cluster centers for a given set of data.IOHClustering, https://arxiv.org/pdf/2505.09233impl_iohclusteringmultimodalartificial-from-real-data>=1IOHClusteringhttps://github.com/IOHprofiler/IOHClusteringClustering-based optimization benchmark built on ML datasets
suite_kinematics_robotarmKinematicsRobotArmSuitecontinuous211Kinematics of a robot arm, https://doi.org/10.1023/A:1013258808932impl_transfer_rf_bbob_rwunimodalreal-world21Transfer Random Forests BBOB Real-worldhttps://github.com/ShuaiqunPan/Transfer_Random_forests_BBOB_Real_worldReal-world BBOB-like problem implementations (Porkchop, KinematicsRobotArm)
suite_l1_zdtL1-ZDTSuitebinary | continuous>=22Variant of ZDT with linkages between variables within groupsLinkage ZDT/DTLZ variants, https://doi.org/10.1145/1143997.1144179>=1>=1
suite_l2_dtlzL2-DTLZSuitecontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]Variant of DTLZ2/DTLZ3 with linkages between all variablesLinkage ZDT/DTLZ variants, https://doi.org/10.1145/1143997.1144179>=1
suite_l2_zdtL2-ZDTSuitebinary | continuous>=22Variant of ZDT with linkages between all variablesLinkage ZDT/DTLZ variants, https://doi.org/10.1145/1143997.1144179>=1>=1
suite_l3_dtlzL3-DTLZSuitecontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]Variant of L2-DTLZ with anti-linkage mappingLinkage ZDT/DTLZ variants, https://doi.org/10.1145/1143997.1144179>=1
suite_l3_zdtL3-ZDTSuitebinary | continuous>=22Variant of L2-ZDT with anti-linkage mappingLinkage ZDT/DTLZ variants, https://doi.org/10.1145/1143997.1144179>=1>=1
suite_maopMaOPSuitecontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]noisyunknownMaOP benchmark, https://doi.org/10.1016/j.swevo.2019.02.003>=1
suite_mechbenchMECHBenchSuitecontinuous>=11unknown{1, 2}noMECHBenchSet of problems inspired by Structural Mechanics Design Optimization. Embeds physical simulations (plasticity only, no fracture/damage). Unstructured/non-isotropic multimodality.MECHBench, https://arxiv.org/abs/2511.10821impl_mechbenchmultimodalreal-world>=1MECHBenchPython{'1 minute', '7 minutes'}https://github.com/BayesOptApp/MECHBenchStructural mechanics design optimization benchmark
suite_mf2MF2Suitecontinuous>=11multi-fidelity[1, 2]mf2: a collection of multi-fidelity benchmark functions in Python, https://doi.org/10.21105/joss.02049impl_mf2>=1mf2Pythonhttps://github.com/sjvrijn/mf2Multi-fidelity test function collection
suite_minus_dtlzMinus DTLZSuitecontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]Variant of DTLZ that minimises the inverse of the base DTLZ functionsMinus DTLZ / Minus WFG, https://doi.org/10.1109/TEVC.2016.2587749>=1
suite_minus_wfgMinus WFGSuitecontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]Variant of WFG that minimises the inverse of the base WFG functionsMinus DTLZ / Minus WFG, https://doi.org/10.1109/TEVC.2016.2587749>=1
suite_mmoppMMOPPSuiteunknown0[2, 3, 4, 5, 6, 7]unknown>=1MMOPP technical report, http://www5.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1327567121&wbfileid=4764412impl_mmoppmultimodalMMOPPhttp://www5.zzu.edu.cn/ecilab/info/1036/1251.htmECI lab distribution page for MMOPP
suite_modactMODActSuitecontinuous | integer40[2, 3, 4, 5]unknown>=1multiobjective design of actuatorsRealistic Constrained Multi-Objective Optimization Benchmark Problems from Design.MODAct, https://doi.org/10.1109/TEVC.2020.3020046["impl_modact", "impl_pymoo"]real-world2020pymoo | modactPython{'20ms'}https://github.com/anyoptimization/pymoo | https://github.com/epfl-lamd/modactMulti-objective optimization in Python | EPFL-LAMD modact package
suite_morepoMOrepoSuiteunknown02dynamic | noisyunknown>=1unknownunknownimpl_morepoMOrepohttps://github.com/MCDMSociety/MOrepoMulti-objective optimisation problem repository
suite_pboPBOSuitebinary>=11Suite of 25 binary optimization problemsPBO benchmarks, https://dl.acm.org/doi/pdf/10.1145/3319619.3326810impl_iohexperimenter artificialhttps://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10254181>=1IOHexperimenterC++/Python https://github.com/IOHprofiler/IOHexperimenterIOHprofiler experimenter framework
CEC2013suite used for cec2013 competition. Also in IOH https://github.com/IOHprofiler/IOHexperimentersuite1scalable
suite_porkchopPorkchopPlotInterplanetaryTrajectorySuite continuousnonononoartificialhttps://peerj.com/articles/cs-2671/CEC2013.pdfhttps://github.com/P-N-Suganthan/CEC2013
CEC2022suite used for cec2022 competition. Also in IOH https://github.com/IOHprofiler/IOHexperimentersuite2 1scalablecontinuousnonononoartificialhttps://github.com/P-N-Suganthan/2022-SO-BO/blob/main/CEC2022%20TR.pdfhttps://github.com/P-N-Suganthan/2022-SO-BO
Onemax+Sphere / Zeromax+Sphere single2scalablebinary and continuous;mixed;nonononoartificialhttps://doi.org/10.1145/3449726.3459521None
Onemax+Sphere / DeceptiveTrap+RotatedEllipsoid single2scalablebinary and continuous;mixed;nonononoartificialhttps://doi.org/10.1145/3449726.3459521None
InverseDeceptiveTrap+RotatedEllipsoid / DeceptiveTrap+RotatedEllipsoid single2scalablebinary and continuous;mixed;nonononoartificialhttps://doi.org/10.1145/3449726.3459521None
PorkchopPlotInterplanetaryTrajectory suite12continuousnonononoreal-worldhttps://doi.org/10.1109/CEC65147.2025.11042973https://github.com/ShuaiqunPan/Transfer_Random_forests_BBOB_Real_world
KinematicsRobotArm suite121continuousnonononoPorkchop plot interplanetary trajectory benchmark, https://doi.org/10.1109/CEC65147.2025.11042973impl_transfer_rf_bbob_rwmultimodal real-worldhttps://doi.org/10.1023/A:10132588089322Transfer Random Forests BBOB Real-world https://github.com/ShuaiqunPan/Transfer_Random_forests_BBOB_Real_worldReal-world BBOB-like problem implementations (Porkchop, KinematicsRobotArm)
VehicleDynamics
suite_reRESuiteinteger | continuous4-14[2, 3, 4, 5, 6, 7, 8, 9]Easy-to-evaluate real-world multi-objective optimization problems, Ryoji Tanabe; Hisao Ishibuchi, https://doi.org/10.1016/j.asoc.2020.106078impl_reproblems suitereal-world-like2-72-7reproblemsPythonhttps://github.com/ryojitanabe/reproblemsReal-world inspired multi-objective optimization problem suite
suite_rwmvopRWMVOPSuitecategorical | continuous | integer>=3 12continuousnonononounknown>=1RWMVOP, https://doi.org/10.1109/TEVC.2013.2281531 real-worldhttps://www.scitepress.org/Papers/2023/121580/121580.pdfhttps://zenodo.org/records/8307853>=1>=1>=1
MECHBenchThis is a set of problems with inspiration from Structural Mechanics Design Optimization. The suite comprises three physical models, from which the user may define different kind of problems which impact the final design output.Problem Suite
suite_sbox_costSBOX-COSTSuitecontinuous>=1 1scalable'ContinuousyesnononoReal-World Applicationhttps://arxiv.org/abs/2511.10821https://github.com/BayesOptApp/MECHBenchproblems from BBOB but allows instances with the optimum close to the boundarySBOX-COST, https://doi.org/10.48550/arXiv.2305.12221impl_iohexperimentermultimodal>=1IOHexperimenterC++/Pythonhttps://github.com/IOHprofiler/IOHexperimenterIOHprofiler experimenter framework
EXPObenchWind farm layout optimization, gas filter design, pipe shape optimization, hyperparameter tuning, and hospital simulationProblem Suite110 to 135Continuous, Integer, Categorical, ConditionalyesnoyesnoReal-World Applicationhttps://doi.org/10.1016/j.asoc.2023.110744https://github.com/AlgTUDelft/ExpensiveOptimBenchmark
suite_sdpSDPSuitecontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]dynamic | noisydynamicunknownSDP dynamic multi-objective benchmark, https://doi.org/10.1109/TCYB.2019.2896021>=1
Gasoline direct injection engine designA multi-objective optimization problem seeking to minimize fuel consumption and NOx emissions over a two-minute dynamic duty cycle, subject to five constraints (turbine inlet temperature, number of knock occurrences, peak cylinder pressure, peak cylinder pressure rise, total work). Seven decision variables are defined: four define the hardware choices of cylinder compression ratio, turbo machinery and EGR cooler sizing; three relate to control variables that parameterise the engine control logic.Single Problem27Continuous, OrdinalyesnonoyesReal-World Application
suite_submodularSubmodular OptimizationSuitebinary>=11set of graph-based submodular optimization problems from 4 problem typesSubmodular optimization benchmark, https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10254181impl_iohexperimenterartificial>=1IOHexperimenterC++/Pythonhttps://github.com/IOHprofiler/IOHexperimenterIOHprofiler experimenter framework https://doi.org/10.1016/j.ejor.2022.08.032
BEACONGenerator for bi-objective benchmark problems with explicitly controlled correlations in continuous spaces.Generator2scalableContinuousnonononoArtificially Generatedhttps://dl.acm.org/doi/10.1145/3712255.3734303https://github.com/Stebbet/BEACON/
suite_tulipa_energy TulipaEnergyDetermine the optimal investment and operation decisions for different types of assets in the energy system (production, consumption, conversion, storage, and transport), while minimizing loss of load.Problem SuiteSuitecontinuous>=1 1scalableContinuousyesnoyesyesReal-World ApplicationSee https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/40-scientific-foundation/45-scientific-referenceshttps://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/
ATOParameters of the Modules of the Automatic Train Operation should be optimized. The parameters are continuous with different ranges. There are two objectives (minimizing energy consumption, minimizing driving duration.Single Problem210ContinuousnonononoReal-World Applicationnoisy | multi-fidelityunknown>=2parameter[1, 2]TulipaEnergyModel.jlDetermine the optimal investment and operation decisions for different assets in the energy system (production, consumption, conversion, storage, transport) while minimizing loss of load. Modelled as a potentially very large linear program with multiple fidelity levels.TulipaEnergyModel.jl scientific references, https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/40-scientific-foundation/45-scientific-referencesimpl_tulipaunimodalreal-world>=1TulipaEnergyModel.jlJulia / JuMP{'minutes', 'hours'}https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/ | https://github.com/TulipaEnergy/Tulipa-OBZ-CaseStudyLarge linear program for optimal investment and operation of energy systems -
Brachytherapy treatment planningTreatment planning for internal radiation therapyProblem Suite2-3100-500ContinuousyesnonoyesReal-World Applicationhttps://www.sciencedirect.com/science/article/pii/S1538472123016781
FleetOptHealthcare organisation in the UK provided data about their current fleet of vehicles to conduct non-emergency heathcare trips in the Argyll and Bute region of Scotland, UK. They also provided historical data about the trips the vehicles took and about the bases which the vehicles return to. The aim is to reduce the existing fleet of vehicles while still ensuring all trips can be covered. Moving a vehicle from one base to another to help cover trips is OK as long as the original base can still cover its trips. Link to paper with more details: https://dl.acm.org/doi/abs/10.1145/3638530.3664137Single Problem1Upper level: 54; lower level: 13208IntegeryesnononoReal-World Applicationhttps://dl.acm.org/doi/abs/10.1145/3638530.3664137Not public: was done for real client with their private data
Building spatial designOptimise the spatial layout of a building to: minimise energy consumption for climate control, and minimise the strain on the structureSingle Problem
suite_vehicle_dynamicsVehicleDynamicsSuitecontinuous 2scalable depending on problem size (e.g. 90 for)Continuous, BooleanyesnononoReal-World Applicationhttps://hdl.handle.net/1887/81789https://github.com/TUe-excellent-buildings/BSO-toolbox
Electric Motor Design OptimizationThe goal is to find a design of a synchronous electric motor for power steering systems that minimizes costs and satisfies all constraints.Single Problem 113Continuous, IntegeryesnoyesnoReal-World Applicationhttps://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf (paper in Slovene)Implementation not freely available
BONO-BenchBi-objective problem generator and suite with scalable continuous decision space. Features complex problem properties (different types of multimodality and challenges in decision and objective space) as well as Pareto front approximations with error guarantees for the hypervolume and exact R2 indicators.GeneratorVehicleDynamics benchmark, https://www.scitepress.org/Papers/2023/121580/121580.pdfimpl_vehicle_dynamicsmultimodalreal-world 2scalableContinuousnonononoArtificially Generated https://github.com/schaepermeier/bonobench
RandOptGenRandOptGen: A Unified Random Problem Generator for Single-and Multi-Objective Optimization Problems with Mixed-Variable Input SpacesGeneratorscalablescalableContinuous, Integer, BooleannonononoArtificially GeneratedVehicleDynamics (Zenodo)https://zenodo.org/records/8307853Zenodo archive for the vehicle dynamics benchmark https://github.com/MALEO-research-group/RandOptGen
CUTErA constrained and unconstrained testing environmentProblem Suite1scalableContinuous, Integer, BooleanyesnononoArtificially Generatedhttps://dl.acm.org/doi/10.1145/962437.962439Not Found
CUTEstThe Constrained and Unconstrained Testing Environment with safe threads (CUTEst) for optimization softwareProblem Suite1scalableContinuous, Integer, BooleanyesnononoArtificially Generatedhttps://link.springer.com/article/10.1007/s10589-014-9687-3https://github.com/jfowkes/pycutest
suite_wfgWFGSuitecontinuous>=1[10, 2, 3, 4, 5, 6, 7, 8, 9]A review of multiobjective test problems and a scalable test problem toolkit, Simon Huband; Philip Hingston; Luigi Barone; Lyndon While, https://doi.org/10.1109/TEVC.2005.861417impl_pymoo>=1pymooPythonhttps://github.com/anyoptimization/pymooMulti-objective optimization in Python
PUBOiA benchmark in which variable importance is tunable, based on the Walsh functionGenerator1scalableBooleannonononoArtificially Generatedhttps://link.springer.com/chapter/10.1007/978-3-031-04148-8_12https://gitlab.com/verel/pubo-importance-benchmark
suite_zdtZDTSuitebinary | continuous>=22Comparison of multiobjective evolutionary algorithms: empirical results, Eckart Zitzler; Kalyanmoy Deb; Lothar Thiele, https://doi.org/10.1162/106365600568202impl_pymoo>=1>=1pymooPythonhttps://github.com/anyoptimization/pymooMulti-objective optimization in Python
name textual description suite/generator/single objectives dimensionality variable type constraints dynamic noise multi-fidelity source (real-world/artificial) reference implementation
+ID Name Type Variable Types Total Variables Objectives Properties Constraint Types Total Constraints Dynamics Noise Partial Evaluations Independent Objectives Fidelity Levels Full Name Description Tags References Implementations Modality Evaluation Time Examples Source Binary Vars Categorical Vars Continuous Vars Integer Vars Implementation Names Implementation Languages Implementation Evaluation Times Implementation Links Implementation Descriptions Implementation Requirements Hard Box Constraints Soft Box Constraints Hard Linear Constraints Soft Linear Constraints Hard Function Constraints Soft Function Constraints
@@ -1247,10 +3426,12 @@

Snippet: call_{problem