-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- South Australian Genomics Centre
-SAHMRI (South Australian Health and Medical Research Institute)
- Website -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+# About the service
+
+The Australian Nextflow Seqera Service provides access to a centralised web platform for managing, launching and monitoring the execution of Nextflow bioinformatics pipelines on [Seqera-compatible compute environments](https://docs.seqera.io/platform/latest/compute-envs/overview). Australian research organisations and individual researchers can add compute infrastructure they have access to and publish pipelines to private or shared workspaces to run data analyses at scale. The service provides access to all features available through the [Seqera Platform](https://seqera.io/platform/) including data exploration and analysis as well as notebook analysis.
+
+
-
-
-
-
-
-
-
-
-
-
+
-## Quick access links
-[Join the pilot project](/nextflow-seqera/main/join_us){: class="btn btn-primary col-3"}
+# Operational Partners
+
+[Australian BioCommons](https://www.biocommons.org.au/) (whose lead agent is [the University of Melbourne](https://www.unimelb.edu.au/)) operates the Australian Nextflow Seqera Service in collaboration with [Pawsey Supercomputing Research Centre](https://pawsey.org.au/), [National Computational Infrastructure (NCI)](https://nci.org.au/), and [Seqera](https://seqera.io/). The Service was established as an output of the Australian BioCommons [Bring Your Own Data Expansion Project](https://www.biocommons.org.au/byo-data-platform-expansion) and is hosted on Amazon Web Services (AWS). It is supported by [Bioplatforms Australia](https://www.bioplatforms.com/) via [NCRIS funding](https://www.education.gov.au/ncris).
-
+
+
diff --git a/main/acceptable-use.md b/main/acceptable-use.md new file mode 100755 index 0000000..55d0bbb --- /dev/null +++ b/main/acceptable-use.md @@ -0,0 +1,22 @@ +--- +title: Acceptable Use Policy +toc: false +--- + +1. You may use the Australian Nextflow Seqera Service to manipulate, analyse and interpret data for the purposes of research, or research training. +2. If you process data using the Australian Nextflow Seqera Service you shall ensure that you comply with applicable laws of the Commonwealth of Australia, as well as laws of other applicable jurisdictions relating to the sensitivity, provenance, intended use, ownership, or licensing imposed on the data being processed. +3. If you configure a compute environment within a workspace, you must abide by the terms of access and use of that compute environment. +4. You agree to interact with the Australian Nextflow Seqera Community (defined as other users of the service, technical support staff, operational staff, and those involved in the governance of the service) in a responsible and respectful manner at all times, consistent with the [Australian BioCommons Code of Conduct](https://doi.org/10.5281/zenodo.4276854). +5. The following uses of the Australian Nextflow Seqera Service are explicitly forbidden: + - The deliberate or reckless creation, transmission, storage, downloading, or display of any offensive or menacing images, data, metadata or other material, or any data capable of being resolved into such images or material + - Use which constitutes an infringement of any intellectual property rights of another person or legal entity + - Attempt to or circumvent the appropriate authentication or access control measures, security or restrictions on the use of a registered account, including the unauthorised distribution or use of tools of any kind for compromising security + - Communications which would be actionable under the law of defamation + - Use for private or business purposes other than research or research training + - Deliberate or reckless undertaking of activities which seek to or result in: + - The imposition of (or attempted imposition of) an unreasonable burden on the Operational Partners and Support teams + - Corruption of or disruption to data stored on the Service, or to the data of another user + - Disruption to other users + - Introduction or transmission of malware into the system. +6. Other uses of the Service are only permitted with the written permission of the Australian Nextflow Seqera Service Manager (contact via
-
+> **Important:** +> Please reach out to Australian BioCommons if you plan to create a new organisation:
- - - - diff --git a/main/pilot-project.md b/main/pilot-project.md new file mode 100755 index 0000000..45e6ed1 --- /dev/null +++ b/main/pilot-project.md @@ -0,0 +1,199 @@ +--- +title: The Australian Nextflow Seqera Service - Pilot project (July 2022-September 2024) +toc: false +--- + + +## Project outline + +Through a licence agreement with Seqera Labs, Australian BioCommons piloted a national Nextflow Seqera service, enabling a centralised command post for Nextflow pipelines to be offered as a fully subsidised service for Australian researchers. + +Working with pilot partners including service hosts Pawsey Supercomputing Research Centre, NCI, SIH, QCIF and Melbourne Bioinformatics, the Australian Nextflow Seqera Service will deliver a key component of BioCommon’s vision for an ecosystem of data analysis and digital asset stewardship platforms. + +{% include affiliation-tiles-selection.html type="support, pilot" %} + +## Project timeframe + +The pilot project began in **July 2022** and finished in **September 2024**. + + + +## Aims of the pilot project + +The pilot project had three key ambitions: + +1. Establish a service where BioCommons early adopters, including both science and method communities, can easily run, manage and monitor the execution of Nextflow workflows on dedicated compute infrastructure. + +2. Understand service demand and utilisation by working with BioCommons communities and assessing interest from other communities. + +3. Develop service operational models; understand costs and benefits; develop a business case for continuing the service. + +At the end of the pilot phase, Australian BioCommons will make the decision on providing this service for a long term. This depends on the feedback and outcomes of this pilot project. Successful use cases will play a major role in influencing the decision. + +## Pilot project outcomes + +Since investigations commenced in July 2022 till the end of the pilot project in September 2024: + +- Nextflow Seqera was deployed on Pawsey infrastructure + +- 29 research groups were onboarded to the service (78 users from 16 different institutions). + +- The services were successfully integrated with our partner's compute infrastructure such as NCI, Pawsey, on premises HPC, AWS and Azure + +- More than 334,000 jobs have been submitted to several compute infrastructures to perform 4,050 workflow executions. + +- Supported many groups with workflow development and deployment +- Established a user guide + +- Promoted the service at several conferences + +
+
+ + +
+ +## Participants of the pilot project + +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ South Australian Genomics Centre
+SAHMRI (South Australian Health and Medical Research Institute)
+ Website +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + diff --git a/main/privacy.md b/main/privacy.md new file mode 100755 index 0000000..b1d5d02 --- /dev/null +++ b/main/privacy.md @@ -0,0 +1,41 @@ +--- +title: Privacy Policy and Data Collection Statement +toc: false +--- + +The Australian Nextflow Seqera Service and its Operational Partners are committed to protecting your privacy. We do not collect, process, retain or share personal data other than for the purposes described below. + +# Usage data + +- In order to monitor and enhance the Service, we track and retain information about your interaction with the Service (i.e. IP addresses, cookies, pages visited, tool invocation parameters, etc). +- When you run a pipeline on a compute environment, the operators of that compute environment may track your usage in accordance with their terms of service. +- The Operational Partners may share log files of your interactions with the service. +- For promotional reasons, the Operational Partners periodically create public usage reports. When we do, we aggregate and anonymise this data to protect your privacy. + +# Registration data + +- When you register an account with the Australian Nextflow Seqera Service, we collect registration data (i.e. email address, institutional affiliation and public name). +- The Operational Partners may contact registered account holders using the email address they provide for operational reasons, including: + - Checking user account security + - Communicating service downtime + - Communicating service improvements + - Responding to your support requests + - Surveying existing and planned use in order to improve and promote the service + - Discussing breaches of these Terms of Use + - Communicating updates to the Australian Nextflow Seqera Service policies. + +- The Operational Partners may contact registered account holders using the email address they provide for promotional reasons, such as seeking consent and content for newsletters, annual reports or media releases. + +# Credentials + +- You can configure workspace credentials to store the access keys and tokens for your compute environments and Git hosting services. These credentials may be transmitted to those services. +- All credentials are encrypted before secure storage and not exposed in an unencrypted way by any Seqera API. + +# Support requests + +- When you submit a support request to the Australian Nextflow Seqera Service, we collect your email address and may ask for personal data as part of the support process. +- Data collected as part of a support request is stored in our instance of Freshdesk and is subject to the Freshworks Privacy Policy which can be found at [https://www.freshworks.com/privacy/](https://www.freshworks.com/privacy/). + +# Modifying and removing personal data + +Registered account holders can request removal of personal information at any time by emailing
|
+ + |
+ + Availability + | ++ Comments + | ++ Contact + | +|
| + Business Hours + | ++ Other times + | +|||
|
+ Nextflow Seqera available |
+ 99% + | +95% + | ++ This is only for the web portal independent of the backend compute infrastructure availability. + | ++ seqera@biocommons.org.au + | +
|
+ + |
+ + Availability + | ++ Comments + | ++ Contact + | +|
| + Business Hours + | ++ Other times + | +|||
|
+ The service available at Pawsey |
+ 99% + | +95% + | ++ Setonix Status + | ++ Helpdesk: help@pawsey.org.au + | +
|
+ The service available at NCI |
+ 99% + | +95% + | ++ GADI Status + | ++ Helpdesk: help@nci.org.au + | +
|
+ The service available on AWS |
+ 99% + | +95% + | ++ | ++ | +
|
+ Other compute backend |
+ 99% + | +95% + | ++ If you are using your own compute infrastructure, you will be affected by the availability of Nextflow Seqera Platform and your own compute backend only. + | ++ | +
+
+
+ +*Questions or comments about the Australian Nextflow Seqera Service, including whether your intended use falls within these Terms of Use, should be directed to
+ diff --git a/user-guide/add-workflow.md b/user-guide/add-workflow.md index 9994213..1ff18d7 100755 --- a/user-guide/add-workflow.md +++ b/user-guide/add-workflow.md @@ -9,4 +9,4 @@ page_id: pg-4 ## Add a new workflow to a workspace -Please follow the steps mentioned at [Seqera documentation](https://docs.seqera.io/latest/launch/launchpad/#adding-a-new-pipeline) to add a pipeline. +Please follow the steps mentioned at [Seqera documentation](https://docs.seqera.io/platform/latest/launch/launchpad/#adding-a-new-pipeline) to add a pipeline. diff --git a/user-guide/compute-env.md b/user-guide/compute-env.md index d1f2c3e..cb93273 100755 --- a/user-guide/compute-env.md +++ b/user-guide/compute-env.md @@ -18,11 +18,11 @@ To access these compute infrastructures through Seqera, compute environments nee Seqera supports adding compute environments for HPCs that utilise [Slurm](https://docs.seqera.io/platform/latest/compute-envs/slurm/) and [PBS Pro](https://docs.seqera.io/platform/latest/compute-envs/altair-pbs-pro/) workload managers. -> Note: compute environments are shared across the users of the same workspace. +> Note: Compute environments are shared across all users of the same workspace. -In order to access compute infrastructre, you need to create access [credentials](https://docs.seqera.io/platform/latest/credentials/overview/#introduction). SSH keys or Tower agents can be used to access the HPC. SSH keys are easier to use but some HPC providers are restricted from sharing SSH keys with a third party (e.g. Seqera Platform). In addition, it can be tricky to use SSH keys if the HPC is in a private network and requires VPN access. +In order to access compute infrastructre, you need to create access [credentials](https://docs.seqera.io/platform/latest/credentials/overview/#introduction). SSH keys or Tower Agents can be used to access the HPC. SSH keys are easier to use but some HPC providers are restricted from sharing SSH keys with a third party (i.e. Seqera Platform). In addition, it can be tricky to use SSH keys if the HPC is within a private network and requires VPN access. -The following instructions is to configure compute environments for HPC through Tower agent credentials. +The following instructions are to configure compute environments for HPC through Tower agent credentials. ### Steps to configure HPC on the Australian Nextflow Seqera Service within an organisation workspace @@ -30,14 +30,14 @@ The following instructions is to configure compute environments for HPC through Prerequisites for configuration on an organisation workspace: 1. You have access to an organisation workspace -2. The user has an owner or admin role within this workspace. +2. You have an owner or administrator role within this workspace. The following steps need to be completed in order unless they have been completed before and are to be reused. 1. Create Personal Token -2. Creating Tower agent credentials -3. Configuring the compute infrastructure +2. Create Tower Agent credentials +3. Configure the compute infrastructure **Detailed Instructions:** @@ -51,7 +51,7 @@ The following steps need to be completed in order unless they have been complete
- You don’t need an access token if you intend to create SSH key credentials, but you will need it for the Tower agent credentials.
+ You don’t need an access token if you intend to create SSH key credentials, but you will need it for the Tower Agent credentials.
- The user can create an access token as described here.
- Keep it safe, create a new one if you lose it and delete lost tokens. @@ -69,15 +69,15 @@ The following steps need to be completed in order unless they have been complete
- Navigate to the workspace you want to add credentials to, then click on the
Credentialstab.
- - Click on the
Add Credentialsbutton underCredentialsto create a sharedAgent connection IDfor the tower agent.
+ - Click on the
Add Credentialsbutton underCredentialsto create a sharedAgent connection IDfor the Tower Agent. - A wizard interface will appear, with some scripts in the
Usagebox and fields to complete. - Keep the Agent interface open.
- Log in to infrastructure (the HPC). -
- In theory, you can run the agent from anywhere on the HPC. See our [best practices](/user-guide/hpc-recommendations) for some recommendations. -
- Copy the usage script from the Agent interface to any text editor and edit the Access token to provide your own token created above and provide the path to the work directory for the agent. -
- The work directory for the Agent (provided as a parameter in the command) must exist before running the agent. +
- In theory, you can run the Agent from anywhere on the HPC. See our [best practices](/user-guide/hpc-recommendations) for some recommendations. +
- Copy the usage script from the Agent interface to any text editor and edit the access token to provide your own token created above and provide the path to the work directory for the Agent. +
- The work directory for the Agent (provided as a parameter in the command) must exist before running the Agent.
- Run the edited script relevant to your infrastructure (PBS or SLURM) bash.
- This script will download the Tower Agent script to the current work directory on the HPC and make it executable.
- Then, it runs the Agent by providing the connection id and access token. @@ -99,7 +99,7 @@ The following steps need to be completed in order unless they have been complete
- Select workdirectory
-
-
- This can remain as $TW_AGENT_WORK or specify your own +
- This can remain as
$TW_AGENT_WORKor specify your own
- Leave the launch directory field empty. @@ -170,32 +170,32 @@ The following steps need to be completed in order unless they have been complete
-
-### Utilising compute environment with tower agent
+### Utilising compute environment with Tower Agent
-There are a few points to be considered when using tower agent:
+There are a few points to be considered when using Tower Agent:
1. The users of a workspace will share the same compute environment and credentials.
2. Each user needs to create their own personal access token.
-3. Each user needs to run Tower agent on their account on the HPC.
-4. Each user needs to pass their personal access token and the shared connection id (of the credential) to their instance of tower agent on the HPC.
+3. Each user needs to run Tower Agent on their account on the HPC.
+4. Each user needs to pass their personal access token and the shared connection id (of the credential) to their instance of Tower Agent on the HPC.
To do that:
1. Create a personal access token or use a pre-created access token as described here (access1).
2. Obtain the connection id for the compute environment from its credential page (conn_id).
-3. Run Tower agent using access1 and conn_id, and an independent work directory (any directory you have access to).
+3. Run Tower Agent using access1 and conn_id, and an independent work directory (any directory you have access to).
4. The compute environment will be available and usable as long as the agent is running.
-
## Configuring commerical cloud
-The easiest way is using AWS Batch and tower forge permissions to allow tower to create the batch environment. In order to do this please follow this [documentation](https://docs.seqera.io/platform/latest/compute-envs/aws-batch/).
+The easiest way is using AWS Batch and Batch Forge permissions to allow tower to create the batch environment. In order to do this please follow this [documentation](https://docs.seqera.io/platform/latest/compute-envs/aws-batch/).
## Other infrastructures
diff --git a/user-guide/hpc-recommendations.md b/user-guide/hpc-recommendations.md
index 25d8485..e6b8a5d 100755
--- a/user-guide/hpc-recommendations.md
+++ b/user-guide/hpc-recommendations.md
@@ -1,7 +1,7 @@
---
title: Recommendations and best practice
contributors: [Ziad Al-Bkhetan, Johan Gustafsson]
-description: General and infrastructre-specific recommendations and best practice to configure your workspace's resources.
+description: General and infrastructure-specific recommendations and best practice to configure your workspace's resources.
toc: false
type: guides
page_id: pg-5
@@ -9,9 +9,6 @@ page_id: pg-5
## General recommendations:
-1. Make sure that the working directory of the agent is not used as a working directory by other users. This can cause access permission issues.
-2. Consider running the agent in the background or using jobs on the HPC, so you don’t need to start the agent very often.
+1. Make sure that the working directory of the Agent is not used as a working directory by other users. This can cause access permission issues.
+2. Consider running the Agent in the background or using jobs on the HPC, so you don’t need to start the Agent very often.
-## Recommendations on configuring compute environments on Setonix at Pawsey.
-
-## Recommendations on configuring compute environments on GADI at NCI.
diff --git a/user-guide/service-access.md b/user-guide/service-access.md
index 902a31c..463662c 100755
--- a/user-guide/service-access.md
+++ b/user-guide/service-access.md
@@ -9,31 +9,27 @@ page_id: pg-1
## Access an organisation workspace
-Accessing an organisation workspace requires joining the [pilot project of the Australian Nextflow Seqera Service](/main/nextflowseqera/join_us.md).
+1. Login to the Australian Nextflow Seqera Service and share your email with BioCommons.
-Under the pilot project, you can:
+2. BioCommons will create a workspace for you and add you as an administrator to the workspace.
-1. Create workspaces for your community.
-
-2. Contact BioCommons to arrange a new workspace for your organisation/group.
-
-3. Login to the Australian Nextflow Seqera Service and share your email with BioCommons.
-
-4. BioCommons will create a workspace for you and add you as an administrator to the workspace.
-
-5. You can add other users to your workspace by following the instructions at below.
+3. You can add other users to your workspace by following the instructions below.
## Join an existing organisation workspace
-1. The user who needs to access a workspace should log in to the Australian Nextflow Seqera Service.
+1. First log in to the Australian Nextflow Seqera Service.
-2. The email address used in the user’s login should then be shared with the workspace administrator, and the admin can add the user directly as described in the [Seqera documentation](https://docs.seqera.io/platform/23.4.0/administration/overview#membership-administration). This should be done under the participants' tab in the workspace rather than the organisation.
+2. The email address used in the user’s login should then be shared with the workspace administrator, and the admin can add the user directly as described in the [Seqera documentation](https://docs.seqera.io/platform/latest/administration/overview#membership-administration). This should be done under the participants' tab in the workspace rather than the organisation.
## Managing participants' access to organisation workspace
-Visit the [Seqera documentation](https://docs.seqera.io/platform/23.4.0/administration/overview#workspace-administration) to learn more about different access roles that can be assigned to workspace participants. This will allow for better management of resources and minimise risks.
+Visit the [Seqera documentation](https://docs.seqera.io/platform/latest/administration/overview#workspace-administration) to learn more about different access roles that can be assigned to workspace participants. This will allow for better management of resources and minimise risks.
+
+## Create a new organisation workspace
+
+Please contact the team at and ensure you are familiar with the [workspace requirements](/nextflow-seqera/main/access-models)
## Access a personal workspace
-By default, each user on the Australian Nextflow Seqera Service will have their own personal workspace which can be accessed directly after logging into the service. There is no need to contact the Australian BioCommons to use your own personal workspace. This model is not currently supported, and users are responsible to configure and use their personal workspace.
+By default, each user on the Australian Nextflow Seqera Service will have their own personal workspace which can be accessed directly after logging into the service. There is no need to contact us to use your own personal workspace. This model is not currently supported, and users are responsible to configure and use their personal workspace.
Tower agent is software that runs on the HPC and communicates with Seqera API to perform all tasks needed on the HPC including launching a pipeline and monitoring its execution. For an admin to create a tower agent credential, follow the following steps:
+Tower Agent is software that runs on the HPC and communicates with the Seqera API to perform all tasks needed on the HPC, including launching a pipeline and monitoring its execution. For an admin to create a tower agent credential, follow these steps:
The steps above will help to create the credentials and understand the Agent’s parameters and how it runs. In practice, this can be better optimised by having all scripts and tokens in config files and bash scripts. See best practice recommendations. The procedure is also described in the Seqera documentation at Quick Start.
+The steps above will help to create the credentials and understand the Agent’s parameters and how it runs. In practice, this can be better optimised by having all scripts and tokens in config files and bash scripts. See [best practice recommendations](/user-guide/hpc-recommendations). The procedure is also described in the Seqera documentation at Quick Start.
Note
Users of the same workspace share credentials, so there is no need to create a credential per user within a workspace.
@@ -138,7 +138,7 @@ The following steps need to be completed in order unless they have been completeNote
-Tower agent does not support service accounts on the HPC. In other words, you can not use one agent for multiple users on tower.
+Tower Agent does not support service accounts on the HPC. In other words, you can not use one Agent for multiple users.
-
The agent should be able to access the internet.
+The Agent should be able to access the internet.