How to use the Pipelines tool
This page describes how to use the sandbox Pipelines tool to submit analyses to the cloud and see their status
Introduction
To handle the multiple steps required for FinnGen analyses, from data preparation to final output, we created streamlined analysis pipelines. By combining validation, parallel processing, and output generation into one workflow, we can easily manage complex GWAS, fine-mapping, and genetic correlation analyses across thousands of endpoints. We have made these workflows available to all users to make performing these analyses faster and easier. These analysis workflows are available in the Pipelines tool.
Analyses submitted in the Pipelines tool are run in Google Cloud and not on your local virtual machine and thus allows for larger and more parallelized analyses.
Accessing the Pipelines tool
To open the Pipelines tool, simply open up the sandbox "start menu" by clicking Applications (usually in the top-left corner of the screen). From that menu, click Sandbox top open a second menu and then click Pipelines.
This will open the Pipelines page in the Firefox web browser with three boxes:
FinnGen workflows
User-defined workflows
Submitted jobs
Submitting an analysis
There are two ways to submit an analysis in the Pipelines tool. One is to use pre-built workflows made available to users, using FinnGen workflows, and the other is to create your own custom workflows, using User-defined workflows. For most users, there will be a pre-built workflow that will meet their needs, but more advanced users with knowledge of WDL (Workflow Descriptor Language; the coding language used to write the workflows) are able to create their own analysis workflows.
FinnGen workflows
The simplest way to run an analysis in the cloud is to submit it using pre-built and centrally maintained workflows. These allow you to input your own curated FinnGen data into standard pipelines and, once completed, access the results. There are two types of workflows: Green workflows (a.k.a. "unmodifiable" workflows) and Red workflows (a.k.a. "modifiable" workflows).
With Green workflows, you can choose the input data but cannot make any changes to how the specific analysis runs (i.e., the existing workflow .wdl cannot be edited). The advantage is that your results will be automatically copied to the green library and to the user-results PheWeb, with no download requested needed. Note: Green/Unmodifiable analysis workflows are only available for FinnGen release 12 (R12) and later.
With Red workflows, you can choose the input data and also make changes to how the specific analysis runs (by editing the existing workflow .wdl). As a consequence, the results cannot be automatically validated and thus copied to the green library or user-results PheWeb. A download request is required to access the results from outside of the sandbox environment.
Green workflows
Clicking on the Green workflows link takes you to a page listing the available green "unmodifiable" analysis workflows. The workflow Name indicates the analysis is performed and the FinnGen release that data that is used. For example, "UnmodifiableRegenieDF13" indicates that the workflow is the "unmodifiable" REGENIE (GWAS) workflow for release 13. If the workflow names are not clear, please refer to the Description for more details.
The Green "unmodifiable" workflows currently available in the Pipelines tool are (click links for usage instructions):
Each Green "unmodifiable" workflow has a different set of input options, so please follow the relevant instructions for the specific workflow.
Browse the list of workflows until you find the one you are interested in, then click Download or Create. Downloading the analysis workflow allows you submit your analysis to the cloud using the terminal, but in this case the analysis results are not automatically copied to the green library or uploaded to the user-results PheWeb because we cannot ensure that the workflow has not been modified. Please refer to How to submit a pipeline from the command line for more information.
Clicking Create loads the page of your selected workflow, which shows the WDL code that performs the analysis and, below it, the Input JSON which contains the input options and is where you need to specify your input files and analysis options. To see all the input options, expand the Input JSON text box by clicking and dragging the triangular tab in the bottom-right corner of the box.
The Pipelines tool won't check whether your input options are correct, but it will check that the syntax is correct (e.g., that you haven't accidentally deleted a necessary double-quote or comma symbol). The result of this check can be seen in the bottom-left of the Input JSON box. If the syntax is correct, it will display "JSON validity: valid ✓" and if incorrect will display "JSON validity: invalid ! ". Do not submit your analysis if the check states that your Input JSON is invalid, as it will fail to run. Instead, carefully check your edits to find the mistake. The check will state that it is valid once you corrected the mistakes.
Once you have edited the text in the Input JSON section with your chosen options and the relevant paths to your desired input files, click Submit to send the analysis to the cloud. If you don't wish to submit yet and plan to return later to finishing editing the input options, click Save Draft to store the edits you have already made. In a similar fashion, the Load Draft button can be used to load a previously saved draft.
Once jobs have been submitted, you can see their status in the Pipelines tool - see Submitted jobs.
Red workflows
Clicking on the Red workflows link takes you to a page listing the available red "modifiable" analysis workflows. As with the Green workflows page, the workflow Name indicates the analysis is performed and the FinnGen release that data that is used. For example, "HLAassociatedDF13" indicates that the workflow is the "modifiable" HLA allele association (GWAS) workflow for release 13. If the workflow names are not clear, please refer to the Description for more details.
The Red "modifiable" workflows currently available in the Pipelines tool include (click links for usage instructions):
Finemapping (releases 7, 11 and 12)
Conditional analysis of GWAS results (releases 10, 11 and 12)
Conditional analysis of GWAS results using custom-define regions (release 12 only)
GATE survival GWAS (releases 6, 10, 11 and 12)
HLA-allele only GWAS (release 13 only)
Polygenic Risk Score (PRS) calculator (releases 12 and 13)
Polygenic Risk Score weights generator (release 12 only)
Find the workflow you wish to use, then click Download or Create. Downloading the analysis workflow allows you submit your analysis using the terminal, but at the cost of analysis results not being automatically copied to the green library or uploaded to the user-results PheWeb. Please refer to How to submit a pipeline from the command line for more information.
As with the Green workflows, clicking Create loads the page of your selected workflow, which shows the WDL code that performs the analysis and, below it, the Input JSON where analysis input options and files are specified.
Unlike with the Green "unmodifiable" workflow page, this time the WDL code can be edited to modify the analysis itself. Here, you can made your edits to the workflow to customise its behavior before submitting the analysis to the cloud. As you are making edits, the Pipelines tool will track whether the syntax is correct and will display a warning stating
! Unexpected WDL format. Please check your WDL!
if incorrect. If the warning is displayed, the red box beneath the warning will inform you of where the Pipelines tool thinks that the syntax is incorrect, which should help identify mistakes in the code.
Beneath the WDL code section is the Input JSON text box. This is where you can specify all the input files and input options needed for your specific analysis. To be able to see all the input options, you can expand the Input JSON text box by clicking and dragging the triangular tab in the bottom-right corner of the box.
The Pipelines tool won't check whether your input options are correct, but (as with the WDL code) it will check that the syntax is correct but (unlike with the WDL code) cannot tell you where the mistake is. You can see the result of this check in the bottom-left of the Input JSON box, which will display "JSON validity: valid ✓" if correct and "JSON validity: invalid ! " if incorrect. If the check returns invalid, carefully check your Input JSON and correct mistakes until the check once again states valid. Do not submit your analysis if the check states that your Input JSON is invalid, as it will fail to run.
Once you have made your desired edits to the WDL code and added your input file paths and optons to the Input JSON section, click Submit to send the analysis to the cloud. If you don't wish to submit yet and plan to return later to finishing editing the input options, click Save Draft to store the edits you have already made. In a similar fashion, the Load Draft button can be used to load a previously saved draft.
Once jobs have been submitted, you can see their status in the Pipelines tool - see Checking the status of submitted jobs.
Edit draft
Clicking Edit draft will open the WDL and Input JSON page of the last workflow that you saved the draft of (using the Save Draft button). This can be either a Green or Red workflow, depending on which was the last draft workflow to be saved. Therefore, always double-check which workflow you are editing.
User-defined workflows
If the pre-built Green and Red workflows are not suitable for your needs, then it is possible to create your own custom analysis workflows for running in the cloud. Analyses are written in WDL (Workflow Descriptor Language) and require you to specify a docker image which contains the operating system and software that the analysis will be run in. This option is typically for more advanced users due to the technical difficulty of writing a workflow in WDL. For a basic guide on creating your first WDL workflow, click here.
Create your own workflow
Clicking on the Create your own workflow link will load a page with two empty boxes:
WDL: Here you put your analysis code, written in WDL. You can either write your code from scratch or copy code from either your own existing WDL(s) or pre-existing WDLs. We recommend the creating your own workflow (in a .wdl file) elsewhere in the sandbox and then copy-pasting into the WDL box on this page.
Input JSON: Here the input options and file paths can be specified as a .json format file. This is where you can put variables that will change between different runs of your custom analysis, so that you can avoid rewriting your WDL workflow for every analysis.
After filling in these two boxes, the Pipelines tool will check for the validity of both the WDL and input json syntax. If there is a syntax mistake, a warning message stating " ! Unexpected WDL format. Please check your WDL!" will appear at the bottom of the WDL box, otherwise if the syntax is correct no message will be displayed. If the WDL syntax warning is displayed, the red box beneath the warning will inform you of where the Pipelines tool thinks that the syntax is incorrect, which should help identify mistakes in the code.
In the Input JSON, the Pipelines will also check that the syntax is correct but (unlike for the WDL code) cannot tell you where the mistake is. The result of this check is seen in the bottom-left of the Input JSON box, which will display "JSON validity: valid ✓" if correct and "JSON validity: invalid ! " if incorrect. If the check returns invalid, carefully check your Input JSON and correct mistakes until the check once again states valid. Do not submit your analysis if the check states that your Input JSON is invalid, as it will fail to run.
Note: The Pipelines tool does not check the logic of your workflow and whether specified input files exist - it only checks whether your code and input options are consistent (e.g., that all variables declared in WDL are specified in the Input JSON.
Once you are satisfied that the WDL code and Input JSON are correct, click Submit to send the analysis to the cloud. If you don't wish to submit yet and plan to return later to finishing editing the input options, click Save Draft to store the edits you have already made. The Load Draft button can be used to load a previously saved draft.
Once jobs have been submitted, you can see their status in the Pipelines tool - see Checking the status of submitted jobs.
Edit draft
Clicking Edit draft will open the WDL and Input JSON page of the last workflow that you saved the draft of (using the Save Draft button), allowing you to continue editing your draft analysis workflow before submitting it.
Checking the status of submitted workflows
On the Pipelines tool main page, you are also able to find the status of your submitted jobs using the links in the Submitted jobs box. The links are:
Show pipelines jobs: See a list of all jobs that you have run or are running, regardless of their status. You can also change the filters so that you can see analyses submitted by other users within the same sandbox.
Running: See a list of your jobs that are currently running
Succeeded: See a list of your jobs that have successfully completed
Failed: See a list of your failed jobs
Clicking Show pipeline jobs will load the "Recent jobs" page which contains a table of all recent jobs that you have submitted. Clicking the Running, Succeeded or Failed links will also bring you to the same page, but will filter the displayed jobs on the selected job status. The "Recent jobs" page displays a table of your "jobs" (submitted workflows) with the following details for each:
State: one of "Submitted", "Running", "Succeeded" or "Failed". "Submitted" jobs are analyses that have not yet started
ID: Cromwell ID (in UUID format). This is a random string that is generated when the workflow starts running and is a unique identifier for your submitted workflow. Jobs in "Submitted" state may not have yet been assigned an ID.
Name: name of the workflow (e.g., "regenie_unmod" for the green "unmodifiable" REGENIE GWAS workflow). This is defined in the WDL code (.wdl file) by the name given to the
workflowfunction.User: First and second name of the user that submitted the workflow
Started At: Date and time that the workflow analyses was submitted.
Ended At: Date and time that the workflow analyses ended (i.e., entered "Succeeded" or "Failed" state). Jobs in "Submitted" and "Running" state will not have an entry in this field.
By default, only workflows that you submit in the past month will be displayed in this table. If you wish to see the status of all users' jobs within your sandbox, click the All Jobs button above the table (default is My Jobs) to the right of "Show only". Similarly, if you wish to see the status of workflows submitted more than a month ago, click either the Last Year or All Time buttons above the table and to the right of "started within" to see all workflows submitted in the past year or all workflows that have ever been submitted. For the purpose of speed, the "Recent jobs" table only shows 10 jobs at a time - you can browse through multiple pages using the pagination (page numbers) at the bottom right of the "Recent jobs" page.
The Search + box above the table allows you to search the list of jobs based on any attribute. For instance, typing "Failed" in the box will show jobs with the string Failed in any field. This allows you to search for specific IDs, workflow names and even dates and times (e.g., typing "17.04.26" will filter for jobs that either started or ended on that date).
Detailed information about a submitted workflow
On the "Recent jobs" page, you can click on any job in the table to load the "Job details" page with detailed information on that job. There will be four tabs:
Info: This is where basic information about the job is displayed, workflow output variable and file paths (under Outputs) if the job has "Succeeded" and (displayed in a red box) any errors the job has had so far (if still "Running") or had (if "Succeeded" or "Failed").
WDL: The submitted WDL code and Input JSON text for this workflow.
Sub WDLs: Submitted WDL code for any sub-WDLs provided when submitting the workflow (it can be advantageous to split WDL code into multiple modular scripts - some pre-built workflows use this feature). If no sub-WDLs used, then this tab will be blank.
Calls & Steps: Similar to the Info tab, but with detailed information for specific tasks within the workflow. Tasks can be select by clicking Select Call ▾, which opens a drop-down menu with the individual tasks. Clicking on any of these tasks in the drop-down menu will display information about that task, including status, start and end time and date, the docker image the task used and the paths of the standard output (stdout) and standard error (stderr) files. The standard output and error files can be useful for diagnostics if the code failed or produced unexpected output. If the workflow is still "Running" and hasn't reached a specific task or if the workflow failed before reaching a specific task, the task will not be displayed in the drop-down menu.
At the bottom of the "Job details" page, you can find the Download metadata (zip) and Clone as new job buttons. Clicking Download metadata (zip) will open a prompt to save a zip file, which contains the detailed Cromwell logs (in .json format) of the job. The "metadata" logs contained within this zip file, along with the error messages in the Info tab and the standard error and output files for specific tasks, are vital for identifying why a job failed. For more information on debugging job failures, please see If your pipeline job fails.
Clicking the Clone as new job button will load the same page that you submitted your workflow from (Green workflows or Red workflows in FinnGen workflows or Create your own workflow in User-defined workflows) and fill the WDL and Input JSON boxes with the text of your submitted workflow. This allows you to make edits to WDL (if not a Green workflow) and Input JSON before resubmitting (e.g., in the case where your previous job failed due to a correctable issue).
Related
Last updated
Was this helpful?