The persistence-service acts as a database service and stores the runtime and evolution data generated by the span-service and the code-agent. This data is made available to the frontend.
All data is persisted in a graph database using Neo4j. The database model is shown below.
For the REST API as well as communication via gRPC and Kafka, we use the Quarkus framework. All interactions with the Neo4j database are performed through the Neo4j-OGM library by using the corresponding Quarkus integration.
The persistence-service communicates with the code-agent via gRPC and with the span-service via Apache Kafka. It provides a REST-API for the frontend.
- persistence-service
- Table of Contents
- Development Instructions
- Database Model
- REST-API
- v2
- GET /v2/landscapes/{landscapeToken}/structure
- GET /v2/landscapes/{landscapeToken}/dynamic?from={}&to={}
- GET /v2/landscapes/{landscapeToken}/timestamps?oldest={}&newest={}&commit={}
- GET /v2/code/applications/{landscapeToken}
- GET /v2/code/commit-tree/{landscapeToken}/{applicationName}
- GET /v2/code/metrics/{landscapeToken}/{applicationName}/{commitHash}
- GET /v2/code/structure/{landscapeToken}/{applicationName}/{commitHash}
- GET /v2/code/structure/{landscapeToken}/{applicationName}/{firstCommitHash}-{secondCommitHash}
- GET /v2/code/commit-comparison/{landscapeToken}/{applicationName}/{firstCommitHash}-{secondCommitHash}
- DELETE /v2/code/landscapes/{landscapeToken}/trace-data
- v3
- GET /v3/landscapes/{landscapeToken}/structure/runtime
- GET /v3/landscapes/{landscapeToken}/structure/evolution/{repositoryName}/{commitHash}
- GET /v3/landscapes/{landscapeToken}/structure/evolution/{repositoryName}/{firstCommitHash}-{secondCommitHash}
- GET /v3/landscapes/{landscapeToken}/dynamic?from={}&to={}
- GET /v3/landscapes/{landscapeToken}/timestamps?oldest={}&newest={}&commit={}
- GET /v3/landscapes/{landscapeToken}/repositories
- GET /v3/landscapes/{landscapeToken}/commit-tree/{repositoryName}
- DELETE /v3/landscapes/{landscapeToken}/trace-data
- v2
- Java: JDK 17 or higher
- Docker: Installed and running, since the persistence-service starts its own Docker container in dev mode and when running tests
We recommend using the IntelliJ IDEA IDE. Your code should follow the Google Java Style Guide. This is enforced using Spotless as part of a pre-commit hook. Before committing, run ./gradlew spotlessApply to automatically fix any formatting issues.
To integrate this format with IntelliJ's formatter (Ctrl + Alt + L shortcut), we recommend the official google-java-format plugin. Carefully follow these instructions to install it. Alternatively, you can use one of the plugins for the Spotless IDE hook. This hook also integrates with Visual Studio Code.
As part of a git pre-commit hook, your code is checked using Spotless, Checkstyle and PMD. If any of these checks fails, the commit is blocked until the issues are resolved. Additionally, all tests must pass for the commit to be successful. Please ensure that all issues are adequately resolved before commiting your changes. If absolutely required, you can skip the pre-commit hook validation using the --no-verify flag, but ensure the issues are addressed before creating a merge request.
You can run your application in dev mode that enables live reloading using:
./gradlew quarkusDevWhen starting the persistence-service in dev mode, a Neo4j Docker container is automatically started as part of Quarkus's Dev Services. If an existing Neo4j container is reachable via the default URI (http://localhost:7687), e.g. when running Neo4j as part of the Docker deployment or via the Docker compose included in this repository's .dev folder, then that instance is used instead.
An overview of the available endpoints is provided in the Quarkus Dev UI at http://localhost:8085/q/dev/. If you are only concerned with providing data to the frontend, you can use the dev-exclusive /example endpoints to populate the database with example data.
You can also inspect and manipulate the current state of the database using Neo4j Browser. By default, the browser runs on http://localhost:7474.
The application can be packaged using:
./gradlew buildIt produces the quarkus-run.jar file in the build/quarkus-app/ directory.
Be aware that it’s not an über-jar as the dependencies are copied into the build/quarkus-app/lib/ directory.
The application is now runnable using java -jar build/quarkus-app/quarkus-run.jar.
If you want to build an über-jar, execute the following command:
./gradlew build -Dquarkus.package.jar.type=uber-jarThe application, packaged as an über-jar, is now runnable using java -jar build/*-runner.jar.
You can create a native executable using:
./gradlew build -Dquarkus.native.enabled=trueOr, if you don't have GraalVM installed, you can run the native executable build in a container using:
./gradlew build -Dquarkus.native.enabled=true -Dquarkus.native.container-build=trueYou can then execute your native executable with: ./build/persistence-service-1.0.0-SNAPSHOT-runner
If you want to learn more about building native executables, please consult https://quarkus.io/guides/gradle-tooling.
To ensure the persistence-service is working properly, there are a number of tests that can be executed using the following command:
./gradlew quarkusTest
The tests are also run as part of the git pre-commit hook.
| Field | Type |
|---|---|
| id | Long |
| name | String |
| Field | Type |
|---|---|
| id | Long |
| name | String |
| Field | Type |
|---|---|
| id | Long |
| name | String |
| type | ClassType |
| modifiers | Set<String> |
| implementedInterfaces | Set<String> |
| annotations | Set<String> |
| enumValues | Set<String> |
| metrics | Map<String, Double> |
| Field | Type |
|---|---|
| id | Long |
| hash | String |
| author | String |
| authorDate | Instant |
| commitDate | Instant |
| Field | Type |
|---|---|
| id | Long |
| name | String |
| Field | Type |
|---|---|
| id | Long |
| name | String |
| type | String |
| modifiers | List<String> |
| Field | Type |
|---|---|
| id | Long |
| name | String |
| hash | String |
| hasFileData | boolean |
| language | Language |
| packageName | String |
| importNames | Set<String> |
| lastEditor | String |
| addedLines | int |
| modifiedLines | int |
| deletedLines | int |
| metrics | Map<String, Double> |
| Field | Type |
|---|---|
| id | Long |
| name | String |
| returnType | String |
| constructor | boolean |
| annotations | Set<String> |
| modifiers | Set<String> |
| outgoingMethodCalls | Set<String> |
| startLine | int |
| endLine | int |
| metrics | Map<String, Double> |
| Field | Type |
|---|---|
| tokenId | String |
| Field | Type |
|---|---|
| id | Long |
| name | String |
| type | String |
| modifiers | List<String> |
| Field | Type |
|---|---|
| id | Long |
| name | String |
| Field | Type |
|---|---|
| id | Long |
| spanId | String |
| startTime | long |
| endTime | long |
| Field | Type |
|---|---|
| id | Long |
| name | String |
| Field | Type |
|---|---|
| id | Long |
| traceId | String |
| startTime | Long |
| endTime | Long |
The database model was created using the web application arrows.app. It is free to use and is officially recommended by Neo4j.
The model can be exported in formats such as PNG and JSON.
The latest versions can be found under ./resources/.
You can import the model via JSON or create a resource link that loads the entire model.
Please note:
After every change to the model, the JSON file at ./resources/db_model.json must be updated, and the PNG file at ./resources/db_model.png should also be replaced with a new PNG export.
In addition, the link in this README.md file must be updated to the new link.
The persistence-service provides a REST-API for the frontend. There are two versions available.
Version 2 offers the standard endpoints that were used prior to the switch to the persistence-service. This can be used to work with older versions of the frontend. For developing new features, please refer to the version 3 of the API.
LandscapeDto getStructureData(String landscapeToken);Returns the landscape associated with a landscape token.
List<TraceDto> getDynamicData(String landscapeToken, Long from, Long to);Returns all traces associated with a landscape. Optionally, a time range can be specified.
Multi<TimestampDto> getTimestamps(
String landscapeToken, Long newest, Long oldest, String commit);Returns timestamps associated with a landscape. Optionally, a time range and a commit can be specified.
List<String> getStaticApplicationNamesForLandscape(String landscapeToken);Returns the names of all applications in a landscape that are backed by data from git evolution analysis.
CommitTreeDto getCommitTreeForApplicationAndLandscape(String landscapeToken, String applicationName);Returns the tree of commits associated with a landscape and an application.
ApplicationMetricsCodeDto getStaticCodeMetricsForApplicationAndCommit(
String landscapeToken, String applicationName, String commitHash);Returns the code metrics associated with a landscape, an application, and a commit.
LandscapeDto getStaticStructureForApplicationAndSingleCommit(
String landscapeToken, String applicationName, String commitHash);Returns the landscape structure associated with a landscape, an application, and a commit.
LandscapeDto getStaticStructureForApplicationAndTwoCommits(
String landscapeToken, String applicationName, String firstCommitHash, String secondCommitHash);Returns the union of the landscape structures associated with two commits of an application in a landscape.
GET /v2/code/commit-comparison/{landscapeToken}/{applicationName}/{firstCommitHash}-{secondCommitHash}
CommitComparison getCommitComparison(
String landscapeToken, String applicationName, String firstCommitHash, String secondCommitHash);Returns a comparison object of two commits associated with a landscape and an application.
void deleteTraceData(String landscapeToken);Deletes all data gathered from runtime analysis associated with a landscape from the database.
Version 3 features a redesign of the endpoints, allowing many of the calculations to be moved from the frontend to the persistence-service. The central change is the switch to a flat data model as opposed to a deeply nested one, allowing fast lookups for visualization objects.
Be sure to update this README when implementing new endpoints.
FlatLandscapeDto getRuntimeStructureData(String landscapeToken);Returns all landscape structure data gathered from runtime analysis for a landscape.
FlatLandscapeDto getStaticStructureData(String landscapeToken, String repositoryName, String commitHash);Retrieve landscape structure data gathered from static analysis for a particular repository and commit in a landscape.
GET /v3/landscapes/{landscapeToken}/structure/evolution/{repositoryName}/{firstCommitHash}-{secondCommitHash}
FlatLandscapeDto getCombinedStaticStructureData(String landscapeToken, String repositoryName, String firstCommitHash, String secondCommitHash);Retrieve the union of landscape structure data for the two provided commits within the given landscape and repository.
The value for the commitComparison attribute is set relative to the second commit, e.g. "DELETED" is written if some component is present in the first commit, but not the second.
List<TraceDto> getDynamicData(String landscapeToken, Long from, Long to);Returns all traces associated with a landscape. Optionally, a time range can be specified.
List<TimestampDto> getTimestamps(String landscapeToken, Long newest, Long oldest, String commit);Returns timestamps associated with a landscape. Optionally, a time range and a commit can be specified.
List<String> getRepositoryNames(String landscapeToken);Retrieves the names of all repositories in a landscape.
CommitTreeDto getCommitTreeForRepositoryAndLandscape(String landscapeToken, String repositoryName);Returns the tree of commits associated with a repository in a landscape.
void deleteTraceData(String landscapeToken);Deletes all data gathered from runtime analysis associated with a landscape from the database.