From d3cef1c1ce1ed46f0e8bdc3c24f5f4bbf5b52fbc Mon Sep 17 00:00:00 2001 From: Tristan Date: Mon, 12 Jan 2026 14:44:00 +0200 Subject: [PATCH 1/9] docs(guides): Add new admin guides This PR adds several adminstrator guides regarding how to install the different Elixir Cloud AAI services on a kubernetes/openshift cluster - Add new admin guides - cloud registry - CWL-WES - DRS-filer - proTES - proWES - TESK - TRS-filer - Update Makefile --- .gitignore | 1 + Makefile | 8 +- docs/guides/guide-admin/cloudregistry.md | 69 + docs/guides/guide-admin/cwlwes.md | 105 + docs/guides/guide-admin/drsfiler.md | 67 + docs/guides/guide-admin/funnel.md | 63 + .../guide-admin/images/overview_protes.svg | 1 + .../guide-admin/images/overview_prowes.svg | 6785 +++++++++++++++++ docs/guides/guide-admin/index.md | 378 +- docs/guides/guide-admin/protes.md | 131 + docs/guides/guide-admin/prowes.md | 132 + docs/guides/guide-admin/services_to_ls_aai.md | 55 +- docs/guides/guide-admin/tesk.md | 336 + docs/guides/guide-admin/trsfiler.md | 68 + docs/guides/guide-user/index.md | 52 +- mkdocs.yml | 8 + 16 files changed, 7847 insertions(+), 412 deletions(-) create mode 100644 docs/guides/guide-admin/cloudregistry.md create mode 100644 docs/guides/guide-admin/cwlwes.md create mode 100644 docs/guides/guide-admin/drsfiler.md create mode 100644 docs/guides/guide-admin/funnel.md create mode 100644 docs/guides/guide-admin/images/overview_protes.svg create mode 100644 docs/guides/guide-admin/images/overview_prowes.svg create mode 100644 docs/guides/guide-admin/protes.md create mode 100644 docs/guides/guide-admin/prowes.md create mode 100644 docs/guides/guide-admin/tesk.md create mode 100644 docs/guides/guide-admin/trsfiler.md diff --git a/.gitignore b/.gitignore index c4dd6af..8075ad7 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ site/ _site/ +.venv \ No newline at end of file diff --git a/Makefile b/Makefile index 6d39ca6..66ac366 100644 --- a/Makefile +++ b/Makefile @@ -40,11 +40,15 @@ i: install .PHONY: venv venv: @echo "\nCreating a virtual environment ++++++++++++++++++++++++++++++++++++++++++++++++\n" - @python -m venv .venv + @if command -v python >/dev/null 2>&1; then \ + python -m venv .venv; \ + else \ + python3 -m venv .venv; \ + fi @echo "\nSummary +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n" @echo "Virtual environment created successfully." @echo "To activate the environment for this shell session, run:" @echo "source .venv/bin/activate" .PHONY: v -v: venv \ No newline at end of file +v: venv diff --git a/docs/guides/guide-admin/cloudregistry.md b/docs/guides/guide-admin/cloudregistry.md new file mode 100644 index 0000000..547ed53 --- /dev/null +++ b/docs/guides/guide-admin/cloudregistry.md @@ -0,0 +1,69 @@ +## Synopsis + +GA4GH Service Registry API implementation for the ELIXIR Cloud. + +Service entries comply with the [external service schema](https://github.com/ga4gh-discovery/ga4gh-service-registry/blob/8c45be52940db92c2fa1cd821519c271c22b1c4c/service-registry.yaml#L158) defined in the [GA4GH Service Registry API][ga4gh-service-registry] + +Developers can find the API documentation [here](https://cloud-registry.readthedocs.io/en/latest/) + +## Installation + +You can find a Helm chart in the [GitHub repository](https://github.com/elixir-cloud-aai/cloud-registry/tree/dev/deployment) of Cloud-registry + +Follow these instructions + +- Install [Helm][helm-install] +- Clone the [Cloud-registry repository](https://github.com/elixir-cloud-aai/cloud-registry/) + + ```sh + git clone https://github.com/elixir-cloud-aai/cloud-registry.git + ``` + +- Browse to `deployment` to find the `Chart.yaml` and the `values.yaml` files + +## Usage + +First you must create a namespace in Kubernetes in which to deploy Cloud-registry. The +commands below assume that everything is created in the context of this +namespace. How the namespace is created depends on the cluster, so we won't +document it here. + +You need to edit the `values.yaml` file + +After this you can deploy Cloud-registry using `helm`: + +```bash +helm install cloud-registry . -f values.yaml +``` + +### Updates + +If you want to edit any of the Deployments, you can update them with +`helm` and the `values.yaml` file. Once edited, you can run this command: + +```bash +helm upgrade cloud-registry . -f values.yaml +``` + +## Technical details + +### MongoDB + +The MongoDB database is deployed using: + +- `templates/mongo-deploy.yaml` + +### Cloud-registry + +TRS-Filer is deployed using: + +- `templates/cloud-registry-deploy.yaml` + +## Destroy + +Simply run: + +```bash +helm uninstall cloud-registry +``` + diff --git a/docs/guides/guide-admin/cwlwes.md b/docs/guides/guide-admin/cwlwes.md new file mode 100644 index 0000000..eabef48 --- /dev/null +++ b/docs/guides/guide-admin/cwlwes.md @@ -0,0 +1,105 @@ +## Synopsis + +Microservice implementing the [Global Alliance for Genomics and +Health][ga4gh] (GA4GH) [Workflow Execution Service][ga4gh-wes] (WES) +API specification for the execution of workflows written in the [Common +Workflow Language](https://www.commonwl.org/) (CWL). + +cwl-WES is a core service of the [ELIXIR Cloud & AAI +project][elixir-cloud-aai-github]. + +## Description + +cwl-WES (formerly: WES-ELIXIR) is a Flask/Gunicorn +application that makes use of [Connexion](https://github.com/ga4gh/workflow-execution-service-schemas) to implement the +[GA4GH WES OpenAPI specification][ga4gh-wes]. It enables clients/users +to execute [CWL](https://www.commonwl.org) workflows in the cloud via a [GA4GH Task Execution +Service][ga4gh-tes] (TES)-compatible execution backend (e.g., +[TESK][tesk] or [Funnel][funnel]). Workflows can be sent for execution, +previous runs can be listed, and the status and run information of individual +runs can be queried. The service leverages [cwl-tes][res-cwl-tes] to +interpret [CWL](https://www.commonwl.org) workflows, break them down into individual tasks and +emit [GA4GH TES][ga4gh-tes]-compatible HTTP requests to a configured +[TES][res-ga4gh-tes] instance. Access to endpoints can be configured to require +[JSON Web Token]-based access tokens, such as those issued by +[ELIXIR AAI](https://elixir-europe.org/platforms/compute/aai). Run information is stored in a +[MongoDB]database. + +Note that development is currently in beta stage. +Further test deployments can be found at the [ELIXIR Cloud & AAI's resource +listings](https://github.com/elixir-cloud-aai/elixir-cloud-aai/blob/dev/resources/resources.md). + +cwl-WES is developed and maintained by the [ELIXIR Cloud & AAI +project][elixir-cloud], a multinational effort aimed at establishing and +implementing [FAIR][fair] research in the Life Sciences. + +## Installation + +You can find a Helm chart in the [GitHub repository](https://github.com/elixir-cloud-aai/cwl-wes/tree/dev/deployment) of CWL-WES + +Follow these instructions + +- Install [Helm][helm-install] +- Clone the [CWL-WES repository](https://github.com/elixir-cloud-aai/cwl-wes/) + + ```sh + git clone https://github.com/elixir-cloud-aai/cwl-wes.git + ``` + +- Browse to `deployment` to find the `Chart.yaml` and the `values.yaml` files + +## Usage + +First you must create a namespace in Kubernetes in which to deploy CWL-WES. The +commands below assume that everything is created in the context of this +namespace. How the namespace is created depends on the cluster, so we won't +document it here. + +You need to edit the `values.yaml` file + +After this you can deploy CWL-WES using `helm`: + +```bash +helm install CWL-WES . -f values.yaml +``` + +### Updates + +If you want to edit any of the Deployments, you can update them with +`helm` and the `values.yaml` file. Once edited, you can run this command: + +```bash +helm upgrade CWL-WES . -f values.yaml +``` + +## Technical details + +### MongoDB + +The MongoDB database is deployed using: + +- `templates/mongodb-deployment.yaml` + +### RabbitMQ + +The message broker RabbitMQ that allows the app to communicate with the +worker is deployed using: + +- `templates/rabbitmq/rabbitmq-deployment.yaml` + +### CWL-WES + +CWL-WES consists of a Flask server and a Celery worker. +There are deployed using: + +- `templates/wes-deployment.yaml` +- `templates/celery-deployment.yaml` + +## Destroy + +Simply run: + +```bash +helm uninstall cwl-wes +``` + diff --git a/docs/guides/guide-admin/drsfiler.md b/docs/guides/guide-admin/drsfiler.md new file mode 100644 index 0000000..e303e27 --- /dev/null +++ b/docs/guides/guide-admin/drsfiler.md @@ -0,0 +1,67 @@ +## Synopsis + +Microservice implementing the [Global Alliance for Genomics and +Health (GA4GH)][ga4gh] [Data Repository Service][ga4gh-drs] (DRS) +API specification. + +## Installation + +You can find a Helm chart in the [GitHub repository](https://github.com/elixir-cloud-aai/trs-filer/tree/dev/deployment) of TRS-Filer + +Follow these instructions + +- Install [Helm][helm-install] +- Clone the [TRS-Filer repository](https://github.com/elixir-cloud-aai/trs-filer/) + + ```sh + git clone https://github.com/elixir-cloud-aai/trs-filer.git + ``` + +- Browse to `deployment` to find the `Chart.yaml` and the `values.yaml` files + +## Usage + +First you must create a namespace in Kubernetes in which to deploy TRS-Filer. The +commands below assume that everything is created in the context of this +namespace. How the namespace is created depends on the cluster, so we won't +document it here. + +You need to edit the `values.yaml` file + +After this you can deploy TRS-Filer using `helm`: + +```bash +helm install trs-filer . -f values.yaml +``` + +### Updates + +If you want to edit any of the Deployments, you can update them with +`helm` and the `values.yaml` file. Once edited, you can run this command: + +```bash +helm upgrade trs-filer . -f values.yaml +``` + +## Technical details + +### MongoDB + +The MongoDB database is deployed using: + +- `templates/mongo-deploy.yaml` + +### TRS-Filer + +TRS-Filer is deployed using: + +- `templates/trs-filer-deploy.yaml` + +## Destroy + +Simply run: + +```bash +helm uninstall trs-filer +``` + diff --git a/docs/guides/guide-admin/funnel.md b/docs/guides/guide-admin/funnel.md new file mode 100644 index 0000000..dbc9669 --- /dev/null +++ b/docs/guides/guide-admin/funnel.md @@ -0,0 +1,63 @@ +Follow these instructions if you wish to deploy a TES endpoint in front of your +HPC/HTC cluster (currently tested with [Slurm][slurm] and [OpenPBS][openpbs]. + +- Make sure the build dependencies `make` and [Go 1.21+][go-install] are + installed, `GOPATH` is set and `GOPATH/bin` is added to `PATH`. + + For example, in Ubuntu this can be achieved via: + + ```sh + sudo apt update + sudo apt install make golang-go + export GOPATH=/your/desired/path + export PATH=$GOPATH/bin:$PATH + go version + ``` + +- Clone the repository: + + ```sh + git clone https://github.com/ohsu-comp-bio/funnel.git + ``` + +- Build Funnel: + + ```sh + cd funnel + make + ``` + +- Test the installation by starting the Funnel server with: + + ```sh + funnel server run + ``` + +If all works, Funnel should be ready for deployment on your HPC/HTC. + +Alternatively, you can install Funnel via Homebrew: + +```sh +brew tap ohsu-comp-bio/formula +brew install funnel@0.11 +``` + +Source: [Funnel website](https://ohsu-comp-bio.github.io/funnel/) + +### Slurm + +For the use of Funnel with Slurm, make sure the following conditions are met: + +1. The `funnel` binary must be placed in a server with access to Slurm. +2. A config file must be created and placed on the same server. [This + file][funnel-config-slurm] can be used as a starting point. +3. If we would like to deploy Funnel as a Systemd service, + [this file][funnel-config-slurm-service] can be used as a template. Set the + correct paths to the `funnel` binary and config file. + +If successfull Funnel should be listening on port `8080`. + +### OpenPBS + +!!! warning "Under construction" + More info coming soon... diff --git a/docs/guides/guide-admin/images/overview_protes.svg b/docs/guides/guide-admin/images/overview_protes.svg new file mode 100644 index 0000000..ab7ac4c --- /dev/null +++ b/docs/guides/guide-admin/images/overview_protes.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/guides/guide-admin/images/overview_prowes.svg b/docs/guides/guide-admin/images/overview_prowes.svg new file mode 100644 index 0000000..6065615 --- /dev/null +++ b/docs/guides/guide-admin/images/overview_prowes.svg @@ -0,0 +1,6785 @@ + + + + + + image/svg+xml + + TEStribute_working + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + TEStribute_working + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + WES + Workflows + GatewayWES + + + + + + + + Tasks + + Middleware + + + Tasks + Task resolution optional(implementation-dependant) + + + + + + + + + + + + + Portal + + + + + + + + + TES + + + + + + + + + + + + + + + + + + + + + + + + + GatewayTES + Middleware + + + + OtherMW + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + TES + tribute + + + + + + + + + + + + + + + TES + + + + + + OtherTES + + + + + + + + + + + + + + + + + + + + + + + + + + + OtherWES + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + OtherMW + + + + MapWES + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/guides/guide-admin/index.md b/docs/guides/guide-admin/index.md index 79fca08..4aacf61 100644 --- a/docs/guides/guide-admin/index.md +++ b/docs/guides/guide-admin/index.md @@ -70,384 +70,16 @@ optionally, available in the ELIXIR Cloud compute network. ### Deploying compute Depending on whether you have a Native Cloud cluster or an HPC/HTC, you will -need to follow the instructions for deploying [TESK][tesk] or [Funnel][funnel] -below, respectively. - -#### Deploying TESK - -[TESK][tesk] uses the Kubernetes Batch API ([Jobs][k8s-jobs]) to schedule -execution of TES tasks. This means that it should be possible to deploy TESK in -any flavor of Kubernetes, but tests are currently only performed with -[Kubernetes][k8s], [OpenShift][openshift], and [Minikube][minikube]. Follow -these instructions if you wish to deploy a TES endpoint on your Native Cloud -cluster, and please let us know if you deploy TESK in any new and interensting -platform. - -TESK currently does not use any other storage (DB) than Kubernetes itself. -[Persistent Volume Claims][k8s-pvc] are used as a temporary storage to handle -input and output files of a task and pass them over between executors of a -task. Note that PVCs are destroyed immediately after task completion! This -means your cluster will need to provide a ReadWriteMany -[StorageClass][k8s-storage-class]. Commonly used storage classes are -[NFS][nfs] and [CephFS][cephfs]. - -Here is an overview of TESK's architecture: - -
- - TESK architecture - -
- -A [Helm][helm] chart is provided for the convenient deployment of TESK. The -chart is available in the [TESK code repository][tesk-helm]. - -Follow these steps: - -1. [Install Helm][helm-install] -2. Clone the [TESK repository][tesk]: - - ```sh - git clone https://github.com/elixir-cloud-aai/TESK.git - ``` - -3. Find the Helm chart at `charts/tesk` -4. Edit file [`values.yaml`][tesk-helm-values] (see - [notes](#notes-for-editing-chart-values) below) -5. Log into the cluster and install TESK with: - - ```sh - helm install -n TESK-NAMESPACE TESK-DEPLOYMENT-NAME . \ - -f secrets.yaml \ - -f values.yaml - ``` - - * Replace `TESK-NAMESPACE` with the name of the namespace where you want to - install TESK. If the namespace is not specified, the default namespace will - be used. - * The argument provided for `TESK-DEPLOYMENT-NAME` will be used by Helm to - refer to the deployment, for example when upgrading or deleting the - deployment. You can choose whichever name you like. - -You should now have a working TESK isntance! - -##### Notes for editing chart values - -In the [TESK deployment documentation][tesk-docs-deploy] documentation there is -a [description of every value][tesk-docs-deploy-values]. Briefly, the most -important are: - -1. `host_name`: Will be used to serve the API. -2. `storageClass`: Specify the storage class. If left empty, TESK will use the - default one configred in the Kubernetes cluster. -3. `auth.mode`: Enable (`auth`) or disable (`noauth`; default) authentication. - When enabled, an OIDC client **must** be in a file `./secrets.yaml`, with - the following format: - - ```yaml - auth: - client_id: - client_secret: - ``` - -4. `ftp`: Which FTP credentials mode to use. Two options are supported: - `.classic_ftp_secret` for basic authentication (username and password) or - `.netrc_secret` for using a [`.netrc`][netrc] file. - - For the classic approach, you must write in `values.yaml`: - - ```yaml - ftp: - classic_ftp_secret: ftp-secret - ``` - - And in a file `.secrets.yaml` write down the username and password as: - - ```yaml - ftp: - username: - password: - ``` - - For the `.netrc` approach, create a `.netrc` file in the `ftp` folder with - the connections details in the correct format. - -5. `clusterType`: Type of Kubernetes flavor. Currently supported: `kubernetes` - (default) and `openshift`. - -!!! warning "Careful" - When creating a `.secrets.yaml` file, ensure that the file is never shared - or committed to a code repository! - -##### Notes for deployment with microk8s - -This section outlines how to install TESK via [microk8s][microk8s] as tested on -an Ubuntu 22.04 machine. - -First, install microk8s through the Snap Store and add yourself to the -`microk8s` group:: - -```bash -sudo snap install microk8s --classic -sudo usermod -a -G microk8s $USER -``` - -Now let's create a directory for the microk8s configuration and enable Helm: - -```bash -mkdir ~/.kube -sudo chown -R $USER ~/.kube -microk8s enable helm3 -``` - -Next, let's clone the TESK repository and move into it the Helm chart directory: - -```bash -git clone https://github.com/elixir-cloud-aai/TESK.git -cd TESK/charts/tesk -``` - -Follow the deployment instructions to create `secrets.yaml` and modify -`values.yaml` as per your requirements. - -> You **MUST** set `host_name`. To make the service available through the -> internet, see further below on how to configure the `service` section. - -Great - you are now ready to deploy TESK! - -First, let's create a namespace: - -```bash -microk8s kubectl create namespace NAMESPACE -``` - -where `NAMESPACE` is an arbitrary name for your resource group. - -Now let's use Helm to install: - -```bash -microk8s helm3 install -n NAMESPACE RELEASE_NAME . -f secrets.yaml -f values.yaml -``` - -where `RELEASE_NAME` is an arbitrary name for this particular TESK release. - -Congratulations - TESK should now be successfully deployed! - -To find out the IP address at which TESK is available, run the following -command: - -```bash -microk8s kubectl get svc -n NAMESPACE -``` - -The output could look something like this: - -```console -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -tesk-api ClusterIP 123.123.123.123 8080/TCP 8s -``` - -Use the `CLUSTER-IP` and the `PORT` with the following template to construct the -URL at which the service is available (and make sure to replace the dummy URL -when you want to try out the calls below): - -```console -http://CLUSTER-IP:PORT/ga4gh/tes/v1 -``` - -So, in this example case, we get the following URL: - -```console -http://123.123.123.123:8080/ga4gh/tes/v1 -``` - -You can now test the intallation with the following example call to get a list -of tasks: - -```bash -curl http://123.123.123.123:8080/ga4gh/tes/v1/tasks -``` - -If everything worked well, you should get an output like this: - -```json -{ - "tasks": [] -} -``` - -Let's try to send a small task to TESK: - -```console -curl \ - -H "Accept: application/json" \ - -H "Content-Type: application/json" \ - -X POST \ - --data '{"executors": [ { "command": [ "echo", "TESK says: Hello World" ], "image": "alpine" } ]}' \ - "http://123.123.123.123:8080/ga4gh/tes/v1/tasks" -``` - -That should give you a task ID: - -```json -{ - "id" : "task-123ab456" -} -``` - -You can run the task list command from before again. Now the response should not -be an empty list anymore. Rather, you should see something like this: - -```json -{ - "tasks" : [ { - "id" : "task-123ab456", - "state" : "COMPLETE" - } ] -} -``` - -To get more details on your task, use the task ID from before in a call like -this: - -```bash -curl http://123.123.123.123:8080/ga4gh/tes/v1/tasks/TASK_ID?view=FULL -``` - -We can use `jq` to parse the results. Let's say we want to see the logs of the -first (only, in this case) TES executor, we could do something like this: - -```console -$curl -s http://123.123.123.123:8080/ga4gh/tes/v1/tasks/task-123ab456?view=FULL | jq '.logs[0].logs' -``` - -Which would give us an output like this: - -```json -[ - { - "start_time": "2023-11-01T14:54:20.000Z", - "end_time": "2023-11-01T14:54:25.000Z", - "stdout": "TESK says: Hello World\n", - "exit_code": 0 - } -] -``` - -Note that in the example, the API is only accessible internally. To make it -accessible publicly, we need to properly configure the `service` section in -`values.yaml`. - -In particular, we would like to set the type to `NodePort` and then set an open -port on the host machine at which the API is exposed. For example, with - -```yaml -service: - type: NodePort - node_port: 31567 -``` - -Kubernetes will route requests coming in to port `31567` on the host machine to -port `8080` on the pod. - -Let's confirm this by upgrading the Helm chart and again inspecting the services -in our namespace with: - -```bash -microk8s helm3 upgrade -n NAMESPACE RELEASE_NAME . -f secrets.yaml -f values.yaml -microk8s kubectl get svc -n NAMESPACE -``` - -We should get an output like this: - -```console -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -tesk-api NodePort 123.123.123.111 8080:31567/TCP 5s -``` - -Indeed, the port section changed as expected. Now, note that the `CLUSTER-IP` -_also_ changed. However, this is not a problem as Kubernetes will manage the -routing, so we don't really need to know the `CLUSTER-IP`. Instead, now we can -use the hostname (or IP) of the host machine, together with the port we set to -call our TES API from anywhere: - -``` -curl http://HOST_NAME_OR_IP:31567/ga4gh/tes/v1/tasks -``` - -> Of course you need to make sure that the port you selected is opened for -> public access. This will depend on your router/firewall settings. - -If you would like to tear down the TESK service, simply run: - -```bash -microk8s helm uninstall RELEASE_NAME -n NAMESPACE -``` - -#### Deploying Funnel - -Follow these instructions if you wish to deploy a TES endpoint in front of your -HPC/HTC cluster (currently tested with [Slurm][slurm] and [OpenPBS][openpbs]. - -1. Make sure the build dependencies `make` and [Go 1.11+][go-install] are - installed, `GOPATH` is set and `GOPATH/bin` is added to `PATH`. - - For example, in Ubuntu this can be achieved via: - - ```sh - sudo apt update - sudo apt install make golang-go - export GOPATH=/your/desired/path - export PATH=$GOPATH/bin:$PATH - go version - ``` - -2. Clone the repository: - - ```sh - git clone https://github.com/ohsu-comp-bio/funnel.git - ``` - -3. Build Funnel: - - ```sh - cd funnel - make - ``` - -4. Test the installation by starting the Funnel server with: - - ```sh - funnel server run - ``` - -If all works, Funnel should be ready for deployment on your HPC/HTC. - -##### Slurm - -For the use of Funnel with Slurm, make sure the following conditions are met: - -1. The `funnel` binary must be placed in a server with access to Slurm. -2. A config file must be created and placed on the same server. [This - file][funnel-config-slurm] can be used as a starting point. -3. If we would like to deploy Funnel as a Systemd service, - [this file][funnel-config-slurm-service] can be used as a template. Set the - correct paths to the `funnel` binary and config file. - -If successfull Funnel should be listening on port `8080`. - -##### OpenPBS - -!!! warning "Under construction" - More info coming soon... +need to follow the instructions for deploying [TESK](tesk.md) or [Funnel](funnel.md) +respectively. ### Deploying storage -Follow the instructions below to connect your TES endpoint to one or more +Follow the instructions to connect your TES endpoint to one or more ELIXIR Cloud cloud storage solutions. The currently supported solutions are: - [MinIO][minio] (Amazon S3) -- [`vsftpd`][vsftpd] (FTP) +- [vsftpd][vsftpd] (FTP) !!! note "Other storage solutions" Other S3 and FTP implementations may work but have not being tested. @@ -460,6 +92,8 @@ documentation][minio-docs-k8s]. It is very simple If you are deploying Minio to OpenShift, you may find this [Minio-OpenShift][minio-deploy-openshift-template] template useful. +You can also follow the instructions of this [MinIO Helm Chart](https://github.com/CSCfi/helm-charts/tree/main/charts/minio) + #### Deploying `vsftpd` (FTP) There are a lot of guides available online to deploy [`vsftpd`][vsftpd], for diff --git a/docs/guides/guide-admin/protes.md b/docs/guides/guide-admin/protes.md new file mode 100644 index 0000000..db2359b --- /dev/null +++ b/docs/guides/guide-admin/protes.md @@ -0,0 +1,131 @@ +## Synopsis + +proTES is a robust and scalable [Global Alliance for Genomics and Health +(GA4GH)][ga4gh] [Task Execution Service (TES) API](https://github.com/ga4gh/task-execution-schemas) gateway +that may play a pivotal role in augmenting the capabilities of your GA4GH Cloud +ecosystem by offering flexible middleware injection for effectively federating +atomic, containerized workloads across on premise, hybrid and multi-cloud +environments composed of GA4GH TES nodes. + +## Description + +proTES gateway may serve as a crucial component in federated compute networks +based on the GA4GH Cloud ecosystem. Its primary purpose is to provide +centralized features to a federated network of independently operated GA4GH TES +instances. As such, it can serve, for example, as a compatibility layer, a load +balancer workload distribution layer, a public entry point to an enclave of +independent compute nodes, or a means of collecting telemetry. + +When TES requests are received, proTES applies a configured middlewares before +forwarding the requests to appropriate TES instances in the network. A plugin +system makes it easy to write and inject middlewares tailored to specific +requirements, such as for access control, request/response processing or +validation, or the selection of suitable endpoints considering data use +restrictions and client preferences. + +### Built-in middleware plugins + +Currently, there are two plugins shipped with proTES that each serve as +proof-of-concept examples for different task distribution scenarios: + +* **Load balancing**: The `pro_tes.middleware.task_distribution.random` plugin + evenly (actually: randomly!) distributes workloads across a network of TES + endpoints +* **Bringing compute to the data**: The + `pro_tes.middleware.task_distribution.distance` plugin selects TES endpoints + to relay incoming requests to in such a way that the distance the (input) data + of a task has to travel across the network of TES endpoints is minimized. + +### Implementation notes + +proTES is a [Flask][res-flask] microservice that supports +[OAuth2][res-oauth2]-based authorization out of the box (bearer authentication) +and stores information about incoming and outgoing tasks in a NoSQL database +([MongoDB][res-mongodb]). Based on our [FOCA][res-foca] microservice archetype, +it is highly configurable in a declarative (YAML-based!) manner. Forwarded tasks +are tracked asynchronously via a [RabbitMQ][res-rabbitmq] broker and +[Celery][res-celery] workers that can be easily scaled up. Both a +[Helm][res-helm] chart and a [Docker Compose][res-docker-compose] configuration +are provided for easy deployment in native cloud-based production and +development environments, respectively. + + + +## Installation + +You can find a Helm chart in the [GitHub repository](https://github.com/elixir-cloud-aai/proTES/tree/dev/deployment) of proWES + +Follow these instructions + +- Install [Helm][helm-install] +- Clone the [proTES repository](https://github.com/elixir-cloud-aai/proTES/) + + ```sh + git clone https://github.com/elixir-cloud-aai/proTES.git + ``` + +- Browse to `deployment` to find the `Chart.yaml` and the `values.yaml` files + +## Usage + +First you must create a namespace in Kubernetes in which to deploy proWES. The +commands below assume that everything is created in the context of this +namespace. How the namespace is created depends on the cluster, so we won't +document it here. + +You need to edit the `values.yaml` file to specify your `applicationDomain` and the `clusterType` + +After this you can deploy proTES using `helm`: + +```bash +helm install protes . -f values.yaml +``` + +### Updates + +If you want to edit any of the Deployments, you can update them with +`helm` and the `values.yaml` file. Once edited, you can run this command: + +```bash +helm upgrade protes . -f values.yaml +``` + +## Technical details + +### MongoDB + +The MongoDB database is deployed using: + +- `templates/mongodb/mongodb-deployment.yaml` + +### RabbitMQ + +The message broker RabbitMQ that allows the app to communicate with the worker +is deployed using: + +- `templates/rabbitmq/rabbitmq-deployment.yaml` + +### proTES + +proTES consists of five deployments: a Flask server and a Celery worker. These +are deployed using: + +- `templates/protes/protes-deployment.yaml` +- `templates/protes/celery-deployment.yaml` + +You can use `ReadWriteOnce` if you don't have `StorageClass` +that supports `RWX`. In that case, a `podAffinity` will be set to have the proTES pods +running on the same node. + +## Destroy + +Simply run: + +```bash +helm uninstall protes +``` + diff --git a/docs/guides/guide-admin/prowes.md b/docs/guides/guide-admin/prowes.md new file mode 100644 index 0000000..59dee9c --- /dev/null +++ b/docs/guides/guide-admin/prowes.md @@ -0,0 +1,132 @@ +[proWES](https://github.com/elixir-cloud-aai/proWES/) is a proxy service that wraps around [GA4GH WES](https://github.com/ga4gh/workflow-execution-service-schemas) + +When WES requests are received, proWES applies one or more configurable middlewares +before forwarding the requests to appropriate WES instances in the network. +A plugin system makes it easy to write and inject middlewares tailored to specific +requirements, such as for access control, request/response processing or validation, +or the selection of suitable endpoints considering data use restrictions and client +preferences. + +### Implementation notes + +proWES is a Flask microservice that supports OAuth2-based authorization out of the box +(bearer authentication) and stores information about incoming and outgoing tasks in a +NoSQL database (MongoDB). Based on our FOCA microservice archetype, it is highly +configurable in a declarative (YAML-based!) manner. Forwarded tasks are tracked +asynchronously via a RabbitMQ broker and Celery workers that can be easily scaled up. +Both a Helm chart and a Docker Compose configuration are provided for easy deployment in +native cloud-based production and development environments, respectively. + + + +## Installation + +You can find a Helm chart in the [GitHub repository](https://github.com/elixir-cloud-aai/proWES/tree/dev/deployment) of proWES + +Follow these instructions + +- Install [Helm][helm-install] +- Clone the [proWES repository](https://github.com/elixir-cloud-aai/proWES/) + + ```sh + git clone https://github.com/elixir-cloud-aai/proWES/ + ``` + +- Browse to `deployment` to find the `Chart.yaml` and the `values.yaml` files + +## Usage + +First you must create a namespace in Kubernetes in which to deploy proWES. The +commands below assume that everything is created in the context of this +namespace. How the namespace is created depends on the cluster, so we won't +document it here. + +There are some prerequisites to deploying proWES on Kubernetes. Namely: + +- MongoDB: + - in the same namespace reachable via 'mongodb' + - DB called `prowes-db` created + - database-user and database-password for `prowes-db` available in a Secret + called 'mongodb' +- RabbitMQ: + - in the same namespace reachable via 'rabbitmq-cluster' +- Secret called `.netrc` created (see below) + +You'll need to configure an SFTP server connection using a `.netrc` file with +the following format: + +``` +machine my-sftp-server.com +login +password +``` + +Create a Kubernetes Secret from the `.netrc` file: + +```bash +kubectl create secret generic netrc --from-file .netrc +``` + +You need to edit the `values.yaml` file to specify your `applicationDomain` and the `clusterType` + +After this you can deploy proWES using `helm`: + +```bash +helm install prowes . -f values.yaml +``` + +### Updates + +If you want to edit any of the Deployments, you can update them with +`helm` and the `values.yaml` file. Once edited, you can run this command: + +```bash +helm upgrade prowes . -f values.yaml +``` + +If you want to point to a different FTP server or change the login credentials +for the current FTP server, you can update the `.netrc` secret like so: + +```bash +kubectl create secret generic netrc --from-file .netrc --dry-run -o yaml | kubectl apply -f - +``` + +## Technical details + +### MongoDB + +The MongoDB database is deployed using: + +- `templates/mongodb/mongodb-deployment.yaml` + +### RabbitMQ + +The message broker RabbitMQ that allows the app to communicate with the worker +is deployed using: + +- `templates/rabbitmq/rabbitmq-deployment.yaml` + +### WES + +proWES consists of five deployments: a Flask server and a Celery worker. These +are deployed using: + +- `templates/prowes/prowes-deployment.yaml` +- `templates/prowes/celery-deployment.yaml` + +You can use `ReadWriteOnce` if you don't have `StorageClass` +that supports `RWX`. In that case, a `podAffinity` will be set to have the proWES pods +running on the same node. + +## Destroy + +Simply run: + +```bash +helm uninstall prowes +``` + diff --git a/docs/guides/guide-admin/services_to_ls_aai.md b/docs/guides/guide-admin/services_to_ls_aai.md index fdbcb69..e4e3125 100644 --- a/docs/guides/guide-admin/services_to_ls_aai.md +++ b/docs/guides/guide-admin/services_to_ls_aai.md @@ -37,55 +37,48 @@ cull: Before configuring Hedgedoc, you need to register your service with LS-Login. Follow the registration process at https://lifescience-ri.eu/ls-login/documentation/how-to-integrate/registration.html -Hedgedoc is configured using environment variables. This guide assumes that a Hedgedoc is already deployed, in our case we used this chart: - -https://github.com/CSCfi/helm-charts/tree/main/charts/hedgedoc +Hedgedoc is configured using environment variables. This guide assumes that a Hedgedoc is already deployed, in our case we used this [chart](https://github.com/CSCfi/helm-charts/tree/main/charts/hedgedoc). Once Hedgedoc is deployed, in order to add LS-AAI login one just needs to add these variables: +```yaml - name: CMD_OAUTH2_USER_PROFILE_URL - - value: https://login.aai.lifescience-ri.eu/oidc/userinfo + value: https://login.aai.lifescience-ri.eu/oidc/userinfo - name: CMD_OAUTH2_USER_PROFILE_USERNAME_ATTR - - value: preferred_username + value: preferred_username - name: CMD_OAUTH2_USER_PROFILE_DISPLAY_NAME_ATTR - - value: name + value: name - name: CMD_OAUTH2_USER_PROFILE_EMAIL_ATTR - - value: email + value: email - name: CMD_OAUTH2_TOKEN_URL - - value: https://login.aai.lifescience-ri.eu/oidc/token + value: https://login.aai.lifescience-ri.eu/oidc/token - name: CMD_OAUTH2_AUTHORIZATION_URL - - value: https://login.aai.lifescience-ri.eu/oidc/authorize + value: https://login.aai.lifescience-ri.eu/oidc/authorize - name: CMD_OAUTH2_CLIENT_ID - - value: _REPLACE BY CLIENT ID_ + value: _REPLACE BY CLIENT ID_ - name: CMD_OAUTH2_CLIENT_SECRET - - value: _REPLACE BY CLIENT SECRET_ + value: _REPLACE BY CLIENT SECRET_ - name: CMD_OAUTH2_PROVIDERNAME - - value: ELIXIR Cloud & AAI + value: ELIXIR Cloud & AAI - name: CMD_OAUTH2_SCOPE - - value: openid email profile - -The documentation from Hedgedoc about this is at: + value: openid email profile +``` -https://docs.hedgedoc.org/configuration/#oauth2-login +The documentation from Hedgedoc about this is available [here](https://docs.hedgedoc.org/configuration/#oauth2-login). # Using LS-Login in MinIO LS-Login can be activated in MinIO either by using the MinIO console using the OIDC configuration or by setting environmental variables ([MinIO OIDC Documentation](https://min.io/docs/minio/linux/operations/external-iam/configure-openid-external-identity-management.html)). -- Config URL (MINIO_IDENTITY_OPENID_CONFIG_URL) - - https://login.aai.lifescience-ri.eu/oidc/.well-known/openid-configuration -- Client ID (MINIO_IDENTITY_OPENID_CLIENT_ID) - - ID of the LS-Login service -- Client secret (MINIO_IDENTITY_OPENID_CLIENT_SECRET) - - Secret of the LS-Login service -- Display Name (MINIO_IDENTITY_OPENID_DISPLAY_NAME) - - A human readable label for the login button (e.g. `LS-Login`) -- Scopes (MINIO_IDENTITY_OPENID_SCOPES) - - Scopes that will be requested from LS-Login (e.g. `openid,email,profile`) -- Role policy (MINIO_IDENTITY_OPENID_ROLE_POLICY) - - Name of a policy in MinIO that will be used to manage access of LS-Login users (e.g. `readonly`). -- Claim User Info (MINIO_IDENTITY_OPENID_CLAIM_USERINFO) - - Allow MinIO to request the userinfo endpoint for additional information (`on`). +```sh +export MINIO_IDENTITY_OPENID_CONFIG_URL="https://login.aai.lifescience-ri.eu/oidc/.well-known/openid-configuration" +export MINIO_IDENTITY_OPENID_CLIENT_ID="" +export MINIO_IDENTITY_OPENID_CLIENT_SECRET="" +export MINIO_IDENTITY_OPENID_DISPLAY_NAME="" +export MINIO_IDENTITY_OPENID_SCOPES="" +export MINIO_IDENTITY_OPENID_ROLE_POLICY="" +export MINIO_IDENTITY_OPENID_CLAIM_USERINFO="" +``` MinIO supports two different mechanisms for authorization of users with OIDC ([MinIO OIDC authorization](https://min.io/docs/minio/linux/administration/identity-access-management/oidc-access-management.html#minio-external-identity-management-openid)). It is recommended to use the RolePolicy flow. Here, all LS-Login users in MinIO will be assigned to one or more policies. These policies can control access to specific buckets by group membership; e.g. require that users belong to a specific LS-AAI group (see [policy based access control](https://min.io/docs/minio/linux/administration/identity-access-management/policy-based-access-control.html#tag-based-policy-conditions)). @@ -115,4 +108,4 @@ In the example below, access to a bucket (`sensitive/`) is restricted to a list } ] } -``` \ No newline at end of file +``` diff --git a/docs/guides/guide-admin/tesk.md b/docs/guides/guide-admin/tesk.md new file mode 100644 index 0000000..b702bb2 --- /dev/null +++ b/docs/guides/guide-admin/tesk.md @@ -0,0 +1,336 @@ +## Deploying TESK + +[TESK][tesk] uses the Kubernetes Batch API ([Jobs][k8s-jobs]) to schedule +execution of TES tasks. This means that it should be possible to deploy TESK in +any flavor of Kubernetes, but tests are currently only performed with +[Kubernetes][k8s], [OpenShift][openshift], and [Minikube][minikube]. Follow +these instructions if you wish to deploy a TES endpoint on your Native Cloud +cluster, and please let us know if you deploy TESK in any new and interesting +platform. + +TESK currently does not use any other storage (DB) than Kubernetes itself. +[Persistent Volume Claims][k8s-pvc] are used as a temporary storage to handle +input and output files of a task and pass them over between executors of a +task. Note that PVCs are destroyed immediately after task completion! This +means your cluster will need to provide a ReadWriteMany +[StorageClass][k8s-storage-class]. Commonly used storage classes are +[NFS][nfs] and [CephFS][cephfs]. + +Here is an overview of TESK's architecture: + + + +A [Helm][helm] chart is provided for the convenient deployment of TESK. The +chart is available in the [TESK code repository][tesk-helm]. + +Follow these steps: + +- [Install Helm][helm-install] +- Clone the [TESK repository][tesk]: + + ```sh + git clone https://github.com/elixir-cloud-aai/TESK.git + ``` + +- Find the Helm chart at `charts/tesk` +- Edit file [`values.yaml`][tesk-helm-values] (see + [notes](#edit-chart-values) below) +- Log into the cluster and install TESK with: + + ```sh + helm install -n TESK-NAMESPACE TESK-DEPLOYMENT-NAME . \ + -f values.yaml + ``` + + * Replace `TESK-NAMESPACE` with the name of the namespace where you want to + install TESK. If the namespace is not specified, the default namespace will + be used. + * The argument provided for `TESK-DEPLOYMENT-NAME` will be used by Helm to + refer to the deployment, for example when upgrading or deleting the + deployment. You can choose whichever name you like. + +You should now have a working TESK instance! You can try to `curl` the address by +running this command: + +```sh +$ curl http:///ga4gh/tes/v1/tasks +{ + "tasks" : [] +} +``` + +### Edit Chart values + +In the [TESK deployment documentation][tesk-docs-deploy] documentation there is +a [description of every value][tesk-docs-deploy-values]. Briefly, the most +important are: + +- `host_name`: Will be used to serve the API. + +- `storage`: `none` or `s3`. If `s3` is set, you must create two files: `config` + and `credentials`. You can find templates in the `s3-config/` folder: + + `config`: + + ``` + [default] + # Non-standard entry, parsed by TESK, not boto3 + endpoint_url= + ``` + + `credentials`: + + ``` + [default] + aws_access_key_id= + aws_secret_access_key= + ``` + + These files will be retrieved during the deployment of the Helm Chart + +- `storageClass`: Specify the storage class. If left empty, TESK will use the + default one configured in the Kubernetes cluster. + +- `auth.mode`: Enable (`auth`) or disable (`noauth`; default) authentication. + When enabled, you must add those two keys: `client_id` and `client_secret` + with your values: + + ```yaml + auth: + client_id: + client_secret: + ``` + +- `ftp`: Which FTP credentials mode to use. Two options are supported: + `.classic_ftp_secret` for basic authentication (username and password) or + `.netrc_secret` for using a [`.netrc`][netrc] file. + + For the classic approach, you must write in `values.yaml` and add two values + `username` and `password`: + + ```yaml + ftp: + classic_ftp_secret: ftp-secret + netrc_secret: + username: + password: + ``` + + For the `.netrc` approach, create a `.netrc` file in the `ftp` folder with + the connections details in the correct format and set a name in `ftp.netrc_secret`: + + ```yaml + ftp: + classic_ftp_secret: + netrc_secret: netrc-secret + ``` + + You can find a template named `.netrc-TEMPLATE` in the `ftp` folder: + + ``` + machine ftp-private.ebi.ac.uk + login ftp-username + password ftp-password + ``` + +### Deploy with microk8s + +This section outlines how to install TESK via [microk8s][microk8s] as tested on +an Ubuntu 22.04 machine. + +First, install microk8s through the Snap Store and add yourself to the +`microk8s` group:: + +```bash +sudo snap install microk8s --classic +sudo usermod -a -G microk8s $USER +``` + +Next, let's clone the TESK repository and move into it the Helm chart directory: + +```bash +git clone https://github.com/elixir-cloud-aai/TESK.git +cd TESK/charts/tesk +``` + +Follow the deployment instructions to modify +`values.yaml` as per your requirements. + +!!! warning + You **MUST** set `host_name`. To make the service available through the + internet, see further below on how to configure the `service` section. + +Great - you are now ready to deploy TESK! + +First, let's create a namespace: + +```bash +microk8s kubectl create namespace NAMESPACE +``` + +where `NAMESPACE` is an arbitrary name for your resource group. + +Now let's use Helm to install: + +```bash +microk8s helm install -n NAMESPACE RELEASE_NAME . -f values.yaml +``` + +where `RELEASE_NAME` is an arbitrary name for this particular TESK release. + +Congratulations - TESK should now be successfully deployed! + +To find out the IP address at which TESK is available, run the following +command: + +```bash +microk8s kubectl get svc -n NAMESPACE +``` + +The output should look something like this: + +```console +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +tesk-api ClusterIP 123.123.123.123 8080/TCP 8s +``` + +Use the `CLUSTER-IP` and the `PORT` with the following template to construct the +URL at which the service is available (and make sure to replace the dummy URL +when you want to try out the calls below): + +```console +http://CLUSTER-IP:PORT/ga4gh/tes/v1 +``` + +So, in this example case, we get the following URL: + +```console +http://123.123.123.123:8080/ga4gh/tes/v1 +``` + +You can now test the intallation with the following example call to get a list +of tasks: + +```bash +curl http://123.123.123.123:8080/ga4gh/tes/v1/tasks +``` + +If everything worked well, you should get an output like this: + +```json +{ + "tasks": [] +} +``` + +Let's try to send a small task to TESK: + +```console +curl \ + -H "Accept: application/json" \ + -H "Content-Type: application/json" \ + -X POST \ + --data '{"executors": [ { "command": [ "echo", "TESK says: Hello World" ], "image": "alpine" } ]}' \ + "http://123.123.123.123:8080/ga4gh/tes/v1/tasks" +``` + +That should give you a task ID: + +```json +{ + "id" : "task-123ab456" +} +``` + +You can run the task list command from before again. Now the response should not +be an empty list anymore. Rather, you should see something like this: + +```json +{ + "tasks" : [ { + "id" : "task-123ab456", + "state" : "COMPLETE" + } ] +} +``` + +To get more details on your task, use the task ID from before in a call like +this: + +```bash +curl http://123.123.123.123:8080/ga4gh/tes/v1/tasks/TASK_ID?view=FULL +``` + +We can use `jq` to parse the results. Let's say we want to see the logs of the +first (only, in this case) TES executor, we could do something like this: + +```console +$curl -s http://123.123.123.123:8080/ga4gh/tes/v1/tasks/task-123ab456?view=FULL | jq '.logs[0].logs' +``` + +Which would give us an output like this: + +```json +[ + { + "start_time": "2023-11-01T14:54:20.000Z", + "end_time": "2023-11-01T14:54:25.000Z", + "stdout": "TESK says: Hello World\n", + "exit_code": 0 + } +] +``` + +Note that in the example, the API is only accessible internally. To make it +accessible publicly, we need to properly configure the `service` section in +`values.yaml`. + +In particular, we would like to set the type to `NodePort` and then set an open +port on the host machine at which the API is exposed. For example, with + +```yaml +service: + type: NodePort + node_port: 31567 +``` + +Kubernetes will route requests coming in to port `31567` on the host machine to +port `8080` on the pod. + +Let's confirm this by upgrading the Helm chart and again inspecting the services +in our namespace with: + +```bash +microk8s helm upgrade -n NAMESPACE RELEASE_NAME . -f values.yaml +microk8s kubectl get svc -n NAMESPACE +``` + +We should get an output like this: + +```console +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +tesk-api NodePort 123.123.123.111 8080:31567/TCP 5s +``` + +Indeed, the port section changed as expected. Now, note that the `CLUSTER-IP` +_also_ changed. However, this is not a problem as Kubernetes will manage the +routing, so we don't really need to know the `CLUSTER-IP`. Instead, now we can +use the hostname (or IP) of the host machine, together with the port we set to +call our TES API from anywhere: + +``` +curl http://HOST_NAME_OR_IP:31567/ga4gh/tes/v1/tasks +``` + +Of course you need to make sure that the port you selected is opened for +public access. This will depend on your router/firewall settings. + +If you would like to tear down the TESK service, simply run: + +```bash +microk8s helm uninstall RELEASE_NAME -n NAMESPACE +``` \ No newline at end of file diff --git a/docs/guides/guide-admin/trsfiler.md b/docs/guides/guide-admin/trsfiler.md new file mode 100644 index 0000000..7a82801 --- /dev/null +++ b/docs/guides/guide-admin/trsfiler.md @@ -0,0 +1,68 @@ +## Synopsis + +Microservice implementing the [Global Alliance for Genomics and +Health (GA4GH)][ga4gh] [Tool Registry Service][ga4gh-trs] (TRS) +API specification. + +## Installation + +You can find a Helm chart in the [GitHub repository](https://github.com/elixir-cloud-aai/trs-filer/tree/dev/deployment) of TRS-Filer + +Follow these instructions + +- Install [Helm][helm-install] +- Clone the [TRS-Filer repository](https://github.com/elixir-cloud-aai/trs-filer/) + + ```sh + git clone https://github.com/elixir-cloud-aai/trs-filer.git + ``` + +- Browse to `deployment` to find the `Chart.yaml` and the `values.yaml` files + +## Usage + +First you must create a namespace in Kubernetes in which to deploy TRS-Filer. The +commands below assume that everything is created in the context of this +namespace. How the namespace is created depends on the cluster, so we won't +document it here. + +You need to edit the `values.yaml` file + +After this you can deploy TRS-Filer using `helm`: + +```bash +helm install trs-filer . -f values.yaml +``` + +### Updates + +If you want to edit any of the Deployments, you can update them with +`helm` and the `values.yaml` file. Once edited, you can run this command: + +```bash +helm upgrade trs-filer . -f values.yaml +``` + +## Technical details + +### MongoDB + +The MongoDB database is deployed using: + +- `templates/mongo-deploy.yaml` + +### TRS-Filer + +TRS-Filer is deployed using: + +- `templates/trs-filer-deploy.yaml` + +## Destroy + +Simply run: + +```bash +helm uninstall trs-filer +``` + + diff --git a/docs/guides/guide-user/index.md b/docs/guides/guide-user/index.md index b70b7b5..4727612 100644 --- a/docs/guides/guide-user/index.md +++ b/docs/guides/guide-user/index.md @@ -89,11 +89,17 @@ A demo workflow is available [here][elixir-cloud-demo-cwl]. ### Nextflow -!!! warning "Under construction" - More info coming soon... +You can find an article about NextFlow with GA4GH TES [here](https://techcommunity.microsoft.com/blog/healthcareandlifesciencesblog/introducing-nextflow-with-ga4gh-tes-a-new-era-of-scalable-data-processing-on-azu/4253160) -## Workflow Execution Service (WES) +To use TES in your Nextflow config, use the plugins `nf-ga4gh`: + +``` +plugins { + id 'nf-ga4gh' +} +``` +## Workflow Execution Service (WES) The GA4GH [WES][ga4gh-wes] is a standard specification protocol for executing and monitoring bioinformatics workflows. It allows researchers to easily execute and manage complex analysis pipelines across multiple computing @@ -119,10 +125,42 @@ specification are: ## Data Repository Service (DRS) -!!! warning "Under construction" - More info coming soon... +The GA4GH [DRS][ga4gh-drs] API provides a standard set of data retrieval methods +to access genomic and related health data across different repositories. +It allows researchers to simplify and standardize data retrieval in cloud-based +environements. Some key features like Standardized data access that offers a consistent +API for retrieving datasets. Cloud-agnostic means that it works accross different +cloud infrastructures. Two use cases for the GA4GH DRS: + +- Scenario 1: A researcher wants to run an analysis pipeline on a dataset without + worrying about where the data physically resides. The researcher uses a DRS ID + to request the dataset. DRS resolves the ID to the actual storage location and + provides signed URLs or access tokens and the pipeline retrievess the data + seamlessly, regardless of the underlying cloud or storage system. + +- Scenario 2: A pharmaceutical company is collaborating with hospitals to analyze + patient genomic data. Due to privacy regulations, raw data cannot be moved outside + the hospital’s secure environment. The hospital can expose their datasets via DRS + endpointsand the pharmaceutical company's workflow engine queries DRS to get metadata. + Finally, the analysis is performed without violating data residency rules. ## Tool Registry Service (TRS) -!!! warning "Under construction" - More info coming soon... +The GA4GH [TRS][ga4gh-trs] API provides a standard mechanism to list, search and +register tools and worflows across different platforms and cloud environments. +It supports workflows written in CWL, WDL, Nextflow, Galaxy, Snakemake. +Here are examples of two use cases: + +- Scenario 1: A bioinformatics researcher develops a workflow for variant calling + using WDL and Docker containers. They want to share it with collaborators who use + different platform. TRS can help, the researcher registers the workflow in a + TRS-compliant registry like Dockstore. The collaborators can discover the workflow + via TRS API and run it on their platform. + TRS will ensure that metadata, versioning, and container are standardized and + accessible + +- Scenario 2: A hospital’s genomics lab uses an automated pipeline to analyze patient + exome data for rare disease diagnosis. The pipeline queries a TRS registry to find + the latest version of tools (like VEP or GATK), retrieves the workflow descriptor + and container images. Finally, the pipeline executes the tools in a secure, + compliant environment. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 7a3f8cf..dc9c036 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -89,7 +89,15 @@ nav: - "guides/guide-dev/index.md" - Administrators: - "guides/guide-admin/index.md" + - "Cloud-Registry": "guides/guide-admin/cloudregistry.md" + - "CWL-WES": "guides/guide-admin/cwlwes.md" + - "DRS-Filer": "guides/guide-admin/drsfiler.md" + - "Funnel": "guides/guide-admin/funnel.md" - "LS Login configuration": "guides/guide-admin/services_to_ls_aai.md" + - "proWES": "guides/guide-admin/prowes.md" + - "proTES": "guides/guide-admin/protes.md" + - "TESK" : "guides/guide-admin/tesk.md" + - "TRS-Filer": "guides/guide-admin/trsfiler.md" - Contributors: - "guides/guide-contributor/index.md" - "Workflow": "guides/guide-contributor/workflow.md" From 64ffc1539454681318c9818403cc533ee7002f03 Mon Sep 17 00:00:00 2001 From: Tristan Date: Mon, 12 Jan 2026 16:04:23 +0200 Subject: [PATCH 2/9] Fix links. Add Sourcery recommendations --- Makefile | 6 ++++-- docs/guides/guide-admin/cwlwes.md | 6 +++--- docs/guides/guide-admin/drsfiler.md | 22 +++++++++++----------- docs/guides/guide-admin/protes.md | 4 ++-- 4 files changed, 20 insertions(+), 18 deletions(-) diff --git a/Makefile b/Makefile index 66ac366..b470061 100644 --- a/Makefile +++ b/Makefile @@ -40,10 +40,12 @@ i: install .PHONY: venv venv: @echo "\nCreating a virtual environment ++++++++++++++++++++++++++++++++++++++++++++++++\n" - @if command -v python >/dev/null 2>&1; then \ + @if command -v python3 >/dev/null 2>&1; then \ + python3 -m venv .venv; \ + elif command -v python >/dev/null 2>&1; then \ python -m venv .venv; \ else \ - python3 -m venv .venv; \ + echo "Python interpreter not found" >&2; exit 1; \ fi @echo "\nSummary +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n" @echo "Virtual environment created successfully." diff --git a/docs/guides/guide-admin/cwlwes.md b/docs/guides/guide-admin/cwlwes.md index eabef48..ccb05c3 100644 --- a/docs/guides/guide-admin/cwlwes.md +++ b/docs/guides/guide-admin/cwlwes.md @@ -20,10 +20,10 @@ previous runs can be listed, and the status and run information of individual runs can be queried. The service leverages [cwl-tes][res-cwl-tes] to interpret [CWL](https://www.commonwl.org) workflows, break them down into individual tasks and emit [GA4GH TES][ga4gh-tes]-compatible HTTP requests to a configured -[TES][res-ga4gh-tes] instance. Access to endpoints can be configured to require -[JSON Web Token]-based access tokens, such as those issued by +[TES][ga4gh-tes] instance. Access to endpoints can be configured to require +JSON Web Token-based access tokens, such as those issued by [ELIXIR AAI](https://elixir-europe.org/platforms/compute/aai). Run information is stored in a -[MongoDB]database. +MongoDB database. Note that development is currently in beta stage. Further test deployments can be found at the [ELIXIR Cloud & AAI's resource diff --git a/docs/guides/guide-admin/drsfiler.md b/docs/guides/guide-admin/drsfiler.md index e303e27..74b772a 100644 --- a/docs/guides/guide-admin/drsfiler.md +++ b/docs/guides/guide-admin/drsfiler.md @@ -6,32 +6,32 @@ API specification. ## Installation -You can find a Helm chart in the [GitHub repository](https://github.com/elixir-cloud-aai/trs-filer/tree/dev/deployment) of TRS-Filer +You can find a Helm chart in the [GitHub repository](https://github.com/elixir-cloud-aai/drs-filer/tree/dev/deployment) of DRS-Filer Follow these instructions - Install [Helm][helm-install] -- Clone the [TRS-Filer repository](https://github.com/elixir-cloud-aai/trs-filer/) +- Clone the [DRS-Filer repository](https://github.com/elixir-cloud-aai/drs-filer/) ```sh - git clone https://github.com/elixir-cloud-aai/trs-filer.git + git clone https://github.com/elixir-cloud-aai/drs-filer.git ``` - Browse to `deployment` to find the `Chart.yaml` and the `values.yaml` files ## Usage -First you must create a namespace in Kubernetes in which to deploy TRS-Filer. The +First you must create a namespace in Kubernetes in which to deploy DRS-Filer. The commands below assume that everything is created in the context of this namespace. How the namespace is created depends on the cluster, so we won't document it here. You need to edit the `values.yaml` file -After this you can deploy TRS-Filer using `helm`: +After this you can deploy DRS-Filer using `helm`: ```bash -helm install trs-filer . -f values.yaml +helm install drs-filer . -f values.yaml ``` ### Updates @@ -40,7 +40,7 @@ If you want to edit any of the Deployments, you can update them with `helm` and the `values.yaml` file. Once edited, you can run this command: ```bash -helm upgrade trs-filer . -f values.yaml +helm upgrade drs-filer . -f values.yaml ``` ## Technical details @@ -51,17 +51,17 @@ The MongoDB database is deployed using: - `templates/mongo-deploy.yaml` -### TRS-Filer +### DRS-Filer -TRS-Filer is deployed using: +DRS-Filer is deployed using: -- `templates/trs-filer-deploy.yaml` +- `templates/drs-filer-deploy.yaml` ## Destroy Simply run: ```bash -helm uninstall trs-filer +helm uninstall drs-filer ``` diff --git a/docs/guides/guide-admin/protes.md b/docs/guides/guide-admin/protes.md index db2359b..5bb2ce7 100644 --- a/docs/guides/guide-admin/protes.md +++ b/docs/guides/guide-admin/protes.md @@ -57,7 +57,7 @@ development environments, respectively. ## Installation -You can find a Helm chart in the [GitHub repository](https://github.com/elixir-cloud-aai/proTES/tree/dev/deployment) of proWES +You can find a Helm chart in the [GitHub repository](https://github.com/elixir-cloud-aai/proTES/tree/dev/deployment) of proTES Follow these instructions @@ -72,7 +72,7 @@ Follow these instructions ## Usage -First you must create a namespace in Kubernetes in which to deploy proWES. The +First you must create a namespace in Kubernetes in which to deploy proTES. The commands below assume that everything is created in the context of this namespace. How the namespace is created depends on the cluster, so we won't document it here. From dc763583c53eb2a643909aa89c9494a6e7b15fbe Mon Sep 17 00:00:00 2001 From: Tristan <74349933+trispera@users.noreply.github.com> Date: Mon, 12 Jan 2026 16:05:44 +0200 Subject: [PATCH 3/9] Update docs/guides/guide-admin/index.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- docs/guides/guide-admin/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/guide-admin/index.md b/docs/guides/guide-admin/index.md index 4aacf61..977e853 100644 --- a/docs/guides/guide-admin/index.md +++ b/docs/guides/guide-admin/index.md @@ -82,7 +82,7 @@ ELIXIR Cloud cloud storage solutions. The currently supported solutions are: - [vsftpd][vsftpd] (FTP) !!! note "Other storage solutions" - Other S3 and FTP implementations may work but have not being tested. + Other S3 and FTP implementations may work but have not been tested. #### Deploying MinIO (Amazon S3) From 445025d783d47b81b04d008cf9cf61de417f3a5f Mon Sep 17 00:00:00 2001 From: Tristan <74349933+trispera@users.noreply.github.com> Date: Mon, 12 Jan 2026 16:07:28 +0200 Subject: [PATCH 4/9] Update docs/guides/guide-user/index.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- docs/guides/guide-user/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/guide-user/index.md b/docs/guides/guide-user/index.md index 4727612..0a49e31 100644 --- a/docs/guides/guide-user/index.md +++ b/docs/guides/guide-user/index.md @@ -91,7 +91,7 @@ A demo workflow is available [here][elixir-cloud-demo-cwl]. You can find an article about NextFlow with GA4GH TES [here](https://techcommunity.microsoft.com/blog/healthcareandlifesciencesblog/introducing-nextflow-with-ga4gh-tes-a-new-era-of-scalable-data-processing-on-azu/4253160) -To use TES in your Nextflow config, use the plugins `nf-ga4gh`: +To use TES in your Nextflow config, use the plugin `nf-ga4gh`: ``` plugins { From 8bb141b93bc9e93737942b0a984f6172615cb50f Mon Sep 17 00:00:00 2001 From: Tristan <74349933+trispera@users.noreply.github.com> Date: Mon, 12 Jan 2026 16:07:56 +0200 Subject: [PATCH 5/9] Update docs/guides/guide-admin/tesk.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- docs/guides/guide-admin/tesk.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/guide-admin/tesk.md b/docs/guides/guide-admin/tesk.md index b702bb2..b06b77c 100644 --- a/docs/guides/guide-admin/tesk.md +++ b/docs/guides/guide-admin/tesk.md @@ -212,7 +212,7 @@ So, in this example case, we get the following URL: http://123.123.123.123:8080/ga4gh/tes/v1 ``` -You can now test the intallation with the following example call to get a list +You can now test the installation with the following example call to get a list of tasks: ```bash From 273a09869f948a20963350cd3378bbea964aacce Mon Sep 17 00:00:00 2001 From: Tristan <74349933+trispera@users.noreply.github.com> Date: Mon, 12 Jan 2026 16:08:54 +0200 Subject: [PATCH 6/9] Update docs/guides/guide-admin/prowes.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- docs/guides/guide-admin/prowes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/guide-admin/prowes.md b/docs/guides/guide-admin/prowes.md index 59dee9c..9a48214 100644 --- a/docs/guides/guide-admin/prowes.md +++ b/docs/guides/guide-admin/prowes.md @@ -118,7 +118,7 @@ are deployed using: - `templates/prowes/prowes-deployment.yaml` - `templates/prowes/celery-deployment.yaml` -You can use `ReadWriteOnce` if you don't have `StorageClass` +You can use `ReadWriteOnce` if you don't have a `StorageClass` that supports `RWX`. In that case, a `podAffinity` will be set to have the proWES pods running on the same node. From ea236d4ac49f7a15c628afa9841d537d07a06d33 Mon Sep 17 00:00:00 2001 From: Tristan <74349933+trispera@users.noreply.github.com> Date: Mon, 12 Jan 2026 16:09:43 +0200 Subject: [PATCH 7/9] Update docs/guides/guide-admin/protes.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- docs/guides/guide-admin/protes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/guide-admin/protes.md b/docs/guides/guide-admin/protes.md index 5bb2ce7..a19b363 100644 --- a/docs/guides/guide-admin/protes.md +++ b/docs/guides/guide-admin/protes.md @@ -16,7 +16,7 @@ instances. As such, it can serve, for example, as a compatibility layer, a load balancer workload distribution layer, a public entry point to an enclave of independent compute nodes, or a means of collecting telemetry. -When TES requests are received, proTES applies a configured middlewares before +When TES requests are received, proTES applies configured middleware before forwarding the requests to appropriate TES instances in the network. A plugin system makes it easy to write and inject middlewares tailored to specific requirements, such as for access control, request/response processing or From 9964f7c70d5c51a35865769855690cb34443974d Mon Sep 17 00:00:00 2001 From: Tristan <74349933+trispera@users.noreply.github.com> Date: Mon, 12 Jan 2026 16:09:58 +0200 Subject: [PATCH 8/9] Update docs/guides/guide-admin/funnel.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- docs/guides/guide-admin/funnel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/guide-admin/funnel.md b/docs/guides/guide-admin/funnel.md index dbc9669..c46a98b 100644 --- a/docs/guides/guide-admin/funnel.md +++ b/docs/guides/guide-admin/funnel.md @@ -55,7 +55,7 @@ For the use of Funnel with Slurm, make sure the following conditions are met: [this file][funnel-config-slurm-service] can be used as a template. Set the correct paths to the `funnel` binary and config file. -If successfull Funnel should be listening on port `8080`. +If successful, Funnel should be listening on port `8080`. ### OpenPBS From 9949d363c57cdf7bc4957308b58fb6d9bf42437d Mon Sep 17 00:00:00 2001 From: Tristan Date: Mon, 12 Jan 2026 16:10:53 +0200 Subject: [PATCH 9/9] Fix typo --- docs/guides/guide-admin/services_to_ls_aai.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/guide-admin/services_to_ls_aai.md b/docs/guides/guide-admin/services_to_ls_aai.md index e4e3125..cb56d4a 100644 --- a/docs/guides/guide-admin/services_to_ls_aai.md +++ b/docs/guides/guide-admin/services_to_ls_aai.md @@ -68,7 +68,7 @@ The documentation from Hedgedoc about this is available [here](https://docs.hedg # Using LS-Login in MinIO -LS-Login can be activated in MinIO either by using the MinIO console using the OIDC configuration or by setting environmental variables ([MinIO OIDC Documentation](https://min.io/docs/minio/linux/operations/external-iam/configure-openid-external-identity-management.html)). +LS-Login can be activated in MinIO either by using the MinIO console using the OIDC configuration or by setting environment variables ([MinIO OIDC Documentation](https://min.io/docs/minio/linux/operations/external-iam/configure-openid-external-identity-management.html)). ```sh export MINIO_IDENTITY_OPENID_CONFIG_URL="https://login.aai.lifescience-ri.eu/oidc/.well-known/openid-configuration"