Conform to JSON:API v1.1 #101

edan-bainglass · 2025-12-25T12:06:28Z

Based on #109

Supersedes #18

This PR implements a JSON:API adapter layer.

JSON:API package

jsonapi/
  models/
    base.py        --> JSON:API spec definition via Pydantic
    aiida.py       --> AiiDA-type-specific JSON:API document models
    errors.py      --> JSON:API error models
  adapters.py      --> JsonApiAdapter class with resource/collection transformation logic
  responses.py     --> JsonApiResponse class with status, media-type, and JsonApiDocument payload
  hooks.py         --> AiiDA-type-specific JSON:API document customization
  utils.py         --> Utilities, e.g., caching, error handling

Benchmarking

Rough GET estimates (in ms) before and after introducing JSON:API transformation

/nodes?page_size=#

#	Before	After	?include=user
1	60	60	60
10	80	80	80
100	200	270	270
1000	1800	2500	2500

Note that foreign entity inclusion is negligable here due to efficient caching (no further DB fetching + transformation if foreign entity already seen), and only having 3 users in the testing environment. It may add more time in cases where foreign entities are unique, e.g., nodes of a group (if and when this is implemented).

Implementation notes

JSON:API transformation is isolated to the JsonApiAdapter class (no instantiation needed)
The adapter can emit:
- a JSON:API resource (e.g. /users/1)
- sub-resources (e.g., /computers/1/metadata), and
- resource collections (e.g., /nodes)
A system of hooks is used to provide JSON:API document customization per-AiiDA entity/quantity (links, relationships, included foreign entities, etc.)
A cache is used for foreign entity inclusion to avoid redundant operations

`QueryBuilder` endpoint

Since the QueryBuilder can return entities, projections, or a combination, its results are collected as the attributes of a single JSON:API resource. The endpoint supports flat=True via a query parameter. It also allows the user to request the full serialization of an entity (in contrast to the default minimal serialization).

Provenance graph traversal

Links used to be returned as the actual incoming/outgoing nodes. However, these weren't exactly node objects, but rather augmented node objects, with link_label and link_type fields. Instead, they are now returned as link resources (containing the link info), with URL links to their source/target nodes, which are proper node objects. The linkage facilitates traversal, as every link contains source/target links, and every source/target contains incoming/outgoing links. So in principal, one can move forward

outgoing -> target -> outgoing -> target -> ...

or backward

incoming -> target -> incoming -> target -> ...

through the provenance graph.

Before

{
  "total": 8,
  "page": 1,
  "page_size": 10,
  "results": [
    {
      "pk": 692,
      "uuid": "ba0d9e48-f8d7-44bf-9929-bdfa753be43a",
      "node_type": "process.workflow.workchain.WorkChainNode.",
      "process_type": "aiida.workflows:quantumespresso.pw.base",
      "ctime": "2025-09-14T16:32:16.940329Z",
      "mtime": "2025-09-14T16:33:18.201701Z",
      "label": "",
      "description": "",
      "user": 1,
      "link_label": "iteration_01",
      "link_type": "call_calc"
    },
    ...
  ]
}

After

{
  "jsonapi": {
    "version": "1.1"
  },
  "links": {
    "self": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/links?direction=incoming&page=1&page_size=10",
    "last": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/links?direction=incoming&page=1&page_size=10"
  },
  "meta": {
    "total": 8,
    "page": 1,
    "page_size": 10
  },
  "data": [
    {
      "id": "9b2b343b-6840-449e-9c18-813b430b9948:ba0d9e48-f8d7-44bf-9929-bdfa753be43a",
      "type": "links",
      "attributes": {
        "link_label": "iteration_01",
        "link_type": "call_calc"
      },
      "relationships": {
        "source": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948"
          },
          "data": {
            "id": "9b2b343b-6840-449e-9c18-813b430b9948",
            "type": "nodes"
          }
        },
        "target": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/nodes/ba0d9e48-f8d7-44bf-9929-bdfa753be43a"
          },
          "data": {
            "id": "ba0d9e48-f8d7-44bf-9929-bdfa753be43a",
            "type": "nodes"
          }
        }
      }
    },
    ...
  ]
}

Examples

Resource

Before

{
  "pk": 695,
  "uuid": "9b2b343b-6840-449e-9c18-813b430b9948",
  "node_type": "process.calculation.calcjob.CalcJobNode.",
  "process_type": "aiida.calculations:quantumespresso.pw",
  "ctime": "2025-09-14T16:32:17.828434Z",
  "mtime": "2025-09-14T16:33:17.520797Z",
  "label": "",
  "description": "",
  "computer": 1,
  "user": 1
}

After

{
  "jsonapi": {
    "version": "1.1"
  },
  "links": {
    "self": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948"
  },
  "data": {
    "id": "9b2b343b-6840-449e-9c18-813b430b9948",
    "type": "nodes",
    "links": {
      "self": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948"
    },
    "attributes": {
      "pk": 695,
      "node_type": "process.calculation.calcjob.CalcJobNode.",
      "process_type": "aiida.calculations:quantumespresso.pw",
      "ctime": "2025-09-14T16:32:17.828434Z",
      "mtime": "2025-09-14T16:33:17.520797Z",
      "label": "",
      "description": ""
    },
    "relationships": {
      "collection": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes"
        }
      },
      "user": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/user"
        },
        "data": {
          "id": "1",
          "type": "users"
        }
      },
      "computer": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/computer"
        },
        "data": {
          "id": "1",
          "type": "computers"
        }
      },
      "attributes": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/attributes"
        }
      },
      "extras": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/extras"
        }
      },
      "repository_metadata": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/repo/metadata"
        }
      },
      "incoming": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/links?direction=incoming"
        }
      },
      "outgoing": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/links?direction=outgoing"
        }
      }
    }
  }
}

With foreign relationships

{
  "jsonapi": {
    "version": "1.1"
  },
  "links": {
    "self": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948?include=users,computers"
  },
  "data": {
    "id": "9b2b343b-6840-449e-9c18-813b430b9948",
    "type": "nodes",
    "links": {
      "self": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948"
    },
    "attributes": {
      "pk": 695,
      "node_type": "process.calculation.calcjob.CalcJobNode.",
      "process_type": "aiida.calculations:quantumespresso.pw",
      "ctime": "2025-09-14T16:32:17.828434Z",
      "mtime": "2025-09-14T16:33:17.520797Z",
      "label": "",
      "description": ""
    },
    "relationships": {
      "collection": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes"
        }
      },
      "user": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/user"
        },
        "data": {
          "id": "1",
          "type": "users"
        }
      },
      "computer": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/computer"
        },
        "data": {
          "id": "1",
          "type": "computers"
        }
      },
      "attributes": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/attributes"
        }
      },
      "extras": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/extras"
        }
      },
      "repository_metadata": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/repo/metadata"
        }
      },
      "incoming": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/links?direction=incoming"
        }
      },
      "outgoing": {
        "links": {
          "related": "http://127.0.0.1:8000/api/v0/nodes/9b2b343b-6840-449e-9c18-813b430b9948/links?direction=outgoing"
        }
      }
    }
  },
  "included": [
    {
      "id": "1",
      "type": "users",
      "links": {
        "self": "http://127.0.0.1:8000/api/v0/users/1"
      },
      "attributes": {
        "email": "aiida@localhost",
        "first_name": "",
        "last_name": "",
        "institution": ""
      },
      "relationships": {
        "collection": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/users"
          }
        }
      }
    },
    {
      "id": "1",
      "type": "computers",
      "links": {
        "self": "http://127.0.0.1:8000/api/v0/computers/1"
      },
      "attributes": {
        "uuid": "7ee86671-18f9-4690-bfa0-da43ddcc5ace",
        "label": "localhost",
        "description": "Localhost automatically created by `verdi presto`",
        "hostname": "localhost",
        "transport_type": "core.local",
        "scheduler_type": "core.direct"
      },
      "relationships": {
        "collection": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/computers"
          }
        },
        "metadata": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/computers/1/metadata"
          }
        }
      }
    }
  ]
}

}

Collection

Before

{
  "total": 755345,
  "page": 1,
  "page_size": 10,
  "results": [
    {
      "pk": 1,
      "uuid": "008c4bef-8622-4f87-8c09-cc9b2aca1eab",
      "node_type": "data.pseudo.upf.UpfData.",
      "ctime": "2025-06-17T08:53:40.504476Z",
      "mtime": "2025-06-17T08:53:41.128779Z",
      "label": "",
      "description": "",
      "user": 1
    },
    ...
  ]
}

After

{
  "jsonapi": {
    "version": "1.1"
  },
  "links": {
    "self": "http://127.0.0.1:8000/api/v0/nodes?page=1&page_size=10",
    "last": "http://127.0.0.1:8000/api/v0/nodes?page=75535&page_size=10",
    "next": "http://127.0.0.1:8000/api/v0/nodes?page=2&page_size=10"
  },
  "meta": {
    "total": 755345,
    "page": 1,
    "page_size": 10
  },
  "data": [
    {
      "id": "008c4bef-8622-4f87-8c09-cc9b2aca1eab",
      "type": "nodes",
      "links": {
        "self": "http://127.0.0.1:8000/api/v0/nodes/008c4bef-8622-4f87-8c09-cc9b2aca1eab"
      },
      "attributes": {
        "pk": 1,
        "node_type": "data.pseudo.upf.UpfData.",
        "ctime": "2025-06-17T08:53:40.504476Z",
        "mtime": "2025-06-17T08:53:41.128779Z",
        "label": "",
        "description": ""
      },
      "relationships": {
        "collection": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/nodes"
          }
        },
        "user": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/nodes/008c4bef-8622-4f87-8c09-cc9b2aca1eab/user"
          },
          "data": {
            "id": "1",
            "type": "users"
          }
        },
        "attributes": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/nodes/008c4bef-8622-4f87-8c09-cc9b2aca1eab/attributes"
          }
        },
        "extras": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/nodes/008c4bef-8622-4f87-8c09-cc9b2aca1eab/extras"
          }
        },
        "repository_metadata": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/nodes/008c4bef-8622-4f87-8c09-cc9b2aca1eab/repo/metadata"
          }
        },
        "incoming": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/nodes/008c4bef-8622-4f87-8c09-cc9b2aca1eab/links?direction=incoming"
          }
        },
        "outgoing": {
          "links": {
            "related": "http://127.0.0.1:8000/api/v0/nodes/008c4bef-8622-4f87-8c09-cc9b2aca1eab/links?direction=outgoing"
          }
        }
      }
    },
    ...
  ]
}

Errors

Before

{
  "details": "No result was found"
}

After

{
  "jsonapi": {
    "version": "1.1"
  },
  "links": {
    "self": "http://127.0.0.1:8000/api/v0/users/4"
  },
  "errors": [
    {
      "status": "404",
      "title": "NotExistent",
      "detail": "No result was found"
    }
  ]
}

Future work

Better OpenAPI documentation of specific types

Significant work has gone into standardizing responses (including errors). The goal is to design the API in such a way as to yield a machine-actionable OpenAPI schema. However, some parts require additional work to bridge the gap between generic JSON:API responses and resource/error-specific shapes. For example, the response of /users/1 returns in the JSON:API document's data field a general resource shape, with empty attributes and relationships objects. In practice, however, these are known a priori. Same with most if not all errors. In principle, it is possible to specify these by extending the model system further. This work is left for future discussion/implementation.

Update

Some of this is now implemented. For now (and maybe permenantly), relationships and included related resources are not typed in the documentation.

JSON:API-ish query parameters

JSON:API provides standards for:

pagination - page[number]=2&page[size]=5, page[offset]=7
sorting - sort=label,-pk
filtering - filter[node.pk][<]=42&filter[node.node_type][like]=%Job%
projections - fields[uuid,label,description,attributes.value]

There's some flexibility afforded here - see for example this JSON:API implementation, as well as the JSON:API docs themselves.

The implementation as in aiida-core.

edan-bainglass and others added 9 commits December 22, 2025 07:56

Leverage aiida-core pydantic models

9443d03

Fix test typing and fixture usage

bd5b70f

Implement read-only mode

aa3cee4

Implement server router

fe91f07

Add /nodes/statistics/user endpoint to REST API

e18f3bf

The implementation as in aiida-core.

Add with_dbenv decorator to node statistics endpoint

dadd753

Explicitly define node statistics shape

5bb56fe

Update test

a0f459d

Update docstring

0da6796

edan-bainglass force-pushed the json-api branch 2 times, most recently from ab46ef1 to 5dc29bc Compare December 25, 2025 13:04

edan-bainglass added 6 commits December 28, 2025 03:09

Implement QB endpoint

d0c1de4

Fix code creation test

bb1cf3a

Implement node links endpoint

2a8d61c

Switch from pk to uuid for entity identifier

ca534bd

Add prefixes at the router layer

5797553

Reorder routes in server endpoints endpoint

0662c07

edan-bainglass force-pushed the json-api branch 4 times, most recently from f133405 to 4eee90c Compare December 29, 2025 12:30

edan-bainglass requested a review from eimrek December 29, 2025 12:59

edan-bainglass force-pushed the json-api branch 6 times, most recently from 12ff5ab to 0cad084 Compare December 30, 2025 10:17

edan-bainglass added 2 commits December 30, 2025 11:18

Rename repository module as services

3bd7812

Discard docstrings

28938c9

edan-bainglass force-pushed the json-api branch 2 times, most recently from ceef3b6 to bef27f6 Compare January 1, 2026 10:13

edan-bainglass marked this pull request as ready for review January 1, 2026 11:13

edan-bainglass mentioned this pull request Jan 1, 2026

Cleanup: docstrings, models, endpoint params, tests, etc. #108

Open

edan-bainglass marked this pull request as draft January 1, 2026 16:14

edan-bainglass added 14 commits January 2, 2026 04:49

Centralize error handling

daacd69

Add missing docstring types

8a99432

Consolidate models

ba270ef

Rename PaginatedResults.results to data

32a747c

Peripheral cleanup

b068a1b

Fix docs

32b862f

Fix entrypoints

ab14c63

Add missing tests

1cf99e7

Organize node endpoints

3a84293

Fix tests

9e01a00

Implement related entity endpoints

484e2fa

Implement group-node endpoints

3389d01

Fix tests

468bd97

Flatten get_related_many results

6af5141

edan-bainglass force-pushed the json-api branch from bef27f6 to 4b53894 Compare January 2, 2026 16:13

edan-bainglass mentioned this pull request Jan 2, 2026

filter query parameters of old AiiDA REST API incompatible with OpenAPI/fastapi standard #16

Open

edan-bainglass force-pushed the json-api branch from 4b53894 to 98e4acd Compare January 2, 2026 16:50

edan-bainglass mentioned this pull request Jan 2, 2026

bubble up python exceptions to response? #11

Open

edan-bainglass marked this pull request as ready for review January 2, 2026 16:55

edan-bainglass force-pushed the json-api branch 3 times, most recently from 7a53d49 to 0f48e15 Compare January 3, 2026 10:39

edan-bainglass added 2 commits January 3, 2026 11:49

Change download_format query parameter to just format

28e64f6

Implement JSON:API adapter layer

6aab6f2

edan-bainglass force-pushed the json-api branch from 0f48e15 to 6aab6f2 Compare January 3, 2026 11:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Conform to JSON:API v1.1 #101

Conform to JSON:API v1.1 #101

Uh oh!

edan-bainglass commented Dec 25, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conform to JSON:API v1.1 #101

Are you sure you want to change the base?

Conform to JSON:API v1.1 #101

Uh oh!

Conversation

edan-bainglass commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

JSON:API package

Benchmarking

Implementation notes

QueryBuilder endpoint

Provenance graph traversal

Examples

Resource

Collection

Errors

Future work

Better OpenAPI documentation of specific types

Update

JSON:API-ish query parameters

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

edan-bainglass commented Dec 25, 2025 •

edited

Loading

`QueryBuilder` endpoint