# How to use the Pipelines tool

### Open the Pipelines tool

You can find the Pipelines in sandbox from Applications -> FinnGen -> Pipelines

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-c2bc106fab4cee5536cda5522ffb29cb2e7e31d5%2Fimage%20(168).png?alt=media" alt=""><figcaption></figcaption></figure>

It will open a new browser page with a view like this:

New front page:

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-003bac1060f7f75ff42f3cad0eb1308f1da94eee%2Fimage%20(59).png?alt=media" alt=""><figcaption></figcaption></figure>

Old front page:

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-9ce2c44f52cb68dd1c0c45302336695619cf6086%2FPipelines_new_view.png?alt=media" alt=""><figcaption></figcaption></figure>

You can submit a new job in 2 different ways from pipelines (under `Create a new job`):

1. [New job](#submit-a-new-job)
2. [New standard job](#submit-a-standard-job)

### Submit a New job

You can submit a **New job** by **copy+pasting** your `.wdl` and `.json` files from `New job` button:

![](https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-7f1a2022030eb8c2b3fd972c0825fb435f6e0a83%2Fimage%20\(423\).png?alt=media)

That will open open a new page where you need to copy-paste your wdl (workflow definition language) script (the easiest way to do this is to write it as a separate file, cat file and copy-paste) and press the ‘Validate’ button at the bottom of the wdl.

![](https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-733d8fc4ea6cae1dd94d015b945fa8d6cfc12ff9%2Fimage%20\(47\).png?alt=media)

This will enable you to define your inputs.

![](https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-d42f646e04497dc2b36cfd3897b7af60e742cb07%2Fimage%20\(32\).png?alt=media)

You can either define the inputs manually or click the `Input as JSON` button and copy-paste your own .json file. If your inputs are ok and valid compared to wdl, you will see `JSON validity: valid` and you can press `Accept`. If your job has no Sub WLDs select `Submit`.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-2c15098f8c01f6daafd6929c1829cba29c67b03e%2Fimage%20(79).png?alt=media" alt=""><figcaption></figcaption></figure>

From Sandbox v.10.3 onwards, the Pipeline tool supports sub-workflow inputs. To include sub-workflow tick the Sub WDLs mark. Click Browse and select the sub.wld.zip file for your job. Then select `Submit`.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-889912768eb4b307470cf7d25ebc44e2d3385b91%2Fimage%20(463).png?alt=media" alt=""><figcaption></figcaption></figure>

Your run will go either as the first on the list (if it goes directly to the ‘Running’ state). If there is a queue, it might go to the ‘Submitted’ state first, in which case you might not see it at the top right away. To monitor your run, it is good to save your jobs ‘ID’ `[workflow_ID]` (the long string of numbers and letters), and also the `workflow_name` of your job (this is the string defined in your wdl as ‘workflow’, for example, if your wdl’s first row would be workflow saige\_pipeline\_ps, ‘saige\_pipeline\_ps’ would be your job’s name).

### Submit a Standard job

For some of the most used pipelines, you can get the `.wdl` and example inputs (`.json`) directly from the standard pipelines.

You can also submit a new **standard job** by selecting `New standard job`, which will open a selection of implemented standard pipelines to use:

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-3ced1f95faecf0c431b20c8d20fcd8582a06e496%2Fstandard_job.png?alt=media" alt=""><figcaption></figcaption></figure>

Currently standard pipeline includes:

* Regenie pipeline (DF10)
* GATE pipeline (DF6 and DF10)
* Finemapping pipeline (DF7)
* Conditional GWAS pipeline using regenie (DF10)
* Example pipeline (hello wdl)

When chosen the standard pipeline you want to use, click `Create`. This will open a view with ready implemented `.wdl`, sub-wdl(s) and `.json`, which all you can edit before submitting the pipelines, from the button below the inputs:

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-e2b369c670cf7a71f009c3d0e62267066a5552dd%2Fspipes%20(1).png?alt=media" alt=""><figcaption></figcaption></figure>

You can also save your changes as draft for later (`Save Draft`), and then `Load Draft` later.

### View Pipeline job info

[Open the Pipelines tool](#open-pipelines-tool) from the Sandbox Application menu, and click on the `Submitted jobs` on the right. You can click on either all jobs in your sandbox (`Show recent jobs`), or only `Running`, `Succeeded` or `Failed` jobs.

When you click on any of them, you'll see a list of jobs to match your selection. You can further filter listed jobs in the Search window. You can for example list only `gate`/`regenie`/`saige` runs, only succeeded runs, or by your `WORKFLOW_ID`. Remember to press '+' sign to add the filter.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-bf98adc8df4f7bd2aa93aaddaf563a7ea4408f2f%2Fpipelines_search.png?alt=media" alt=""><figcaption></figcaption></figure>

Clicking one of the jobs will open the information page of that run. On the sub-pages, you can see Info, WLD, sub-WDLs, and Calls & Steps information for the run. Save your Pipeline ID as you may need it later.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-6c2ab6525c3c7bd0a6d4f89b55ba73d7c29ecb4c%2Fimage%20(103)%20(1).png?alt=media" alt=""><figcaption></figcaption></figure>

### Output

When/if your job (standard or non-standatd) is successfully done (state is Succeeded), the state of your run changes to ‘Succeeded’, and you can check your results from:

`/finngen/pipeline/cromwell/workflows/[workflow_name]/[workflow_ID]`

**Related:**

* [If your pipeline job fails](https://docs.finngen.fi/working-in-the-sandbox/running-analyses-in-sandbox/pipelines-tool-instructions/if-your-pipeline-job-fails)
