# If your pipeline job fails

Pipeline jobs can fail on few of reasons and on different layers of data processing.

### Pipeline job failing during wdl, json or input data evaluation

The before cromwell starts data processing wdl, json and input data is evaluated. Runtime logs (see below) are not produced since the process has not started yet. Pipeline GUI provides failure information from workflow metadata if available.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-13fa9095fb41a2d2b76741861c88badf9d297903%2Fimage%20(479).png?alt=media" alt=""><figcaption></figcaption></figure>

Above example the path of file in library red bucket is not found as Sandbox alias "LIBRARY\_RED/" is not translated to the correct google cloud bucket "gs\://finngen-production-library-red" equivalent. In the Call & Step there is a path to the stderr file but the file does not exist:

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-6a58ddae34507720961e4e4d330d7c1bc9c95d4d%2Fimage%20(682).png?alt=media" alt=""><figcaption></figcaption></figure>

It is possible to inspect pipeline job metadata with command line tool to get additional cues for debugging:

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-b875e9d400d10c46d7d7e9237b23ab998056ebf5%2Fimage%20(730).png?alt=media" alt=""><figcaption></figcaption></figure>

### Pipeline job failing during virtual machine runtime of specific pipeline call

All pipeline call output files will be saved in Sandbox. These folders contain information on call executions that are important in source of debugging or you workflow. The most informative file is `/finngen/pipeline/cromwell/workflows/[workflow_name]/[workflow_ID]`. Your job will be saved as a pipeline job even if it is conducted with [the custom GWAS tools](https://docs.finngen.fi/working-in-the-sandbox/which-tools-are-available/untitled). You can search for a pipeline job from [the pipeline tools](https://docs.finngen.fi/working-in-the-sandbox/running-analyses-in-sandbox/pipelines-tool-instructions/how-to-use-the-pipelines-area) based on the date the custom GWAS was run, or by browsing the RECENTS page in [the custom GWAS tool](https://docs.finngen.fi/working-in-the-sandbox/which-tools-are-available/untitled/custom-gwas-tool).

If your pipeline job fails, first read the error report and log file to find out the specific reason for the failure of your pipeline job.

## How to check why the job failed

In the **Pipelines** application, go to the **Failed** section.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-8fefa8720541abd7f6b233116dc2fe015039ca3a%2Fpipelines1.png?alt=media" alt=""><figcaption></figcaption></figure>

Search for your own name (or the name of the person who made the pipeline job) and click the pipeline job you want to examine. You can also search for the job ID (see [Tips on how to find a pipeline job ID](https://docs.finngen.fi/working-in-the-sandbox/running-analyses-in-sandbox/pipelines-tool-instructions/tips-on-how-to-find-a-pipeline-job-id)).

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-1bfa505c558fe30919dea3539f3f9ea9c9621ee6%2Fpipelines2.png?alt=media" alt=""><figcaption></figcaption></figure>

The **Job details** page will open. Here you can again see the pipeline ID, Name, State, Submitting, Starting and Ending date and time, and more information about the error type.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-fc7c55de9a4b03ebae70cfcb8dbbe073168e3e2d%2Fpipelines33.png?alt=media" alt=""><figcaption></figcaption></figure>

Click over to the WDL tab to see the WDL code, input files, and settings. Ensure that you are viewing the correct run, and that it was generated from the input files you expected.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-f43b86f37cf618fd0cd5166a0a35e6cef3129fd6%2Fpipelineswdl.png?alt=media" alt=""><figcaption></figcaption></figure>

Next, go back to the Info page and click on **Download metadata**.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-6c5bf8b7ebea1dfa6d3f0223c7895ed988914773%2Fpipeline3%20(1).png?alt=media" alt=""><figcaption></figcaption></figure>

Once the metadata has been downloaded, open the file with RStudio and examine the point(s) of failure. A good way to do this is with Control + F (Windows) or Command + F (Mac) and search for "error" or "fail".

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-107adeb3c3244f128ea1041e36d99274983062af%2Fpipelines4.png?alt=media" alt=""><figcaption></figcaption></figure>

If this method doesn't give enough information about the cause of failure, it's a good idea to check out the `stderr` error file of the pipeline job. It can be found from **File Manager** with the path `/finngen/pipeline/cromwell/workflows/regenie/[your_pipeline_job_ID]`.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-fe91959694adaa34d2b20bdffe5b776e29ca0b66%2Fpipelines5.png?alt=media" alt=""><figcaption></figcaption></figure>

Select `call-sub_step1` and go through the folders in it until you reach the file containing `stderr`. This might require a little searching, but `stderr` usually is found from this file.

<figure><img src="https://3072695768-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MhYL0UTLjqsuIdK0SSO%2Fuploads%2Fgit-blob-f77af92ae132dd5e71b1f8519932eb30f46c2318%2Fpipelines6.png?alt=media" alt=""><figcaption></figcaption></figure>

After finding `stderr`, open it and try to find out a cause of failure.

## Solution to common errors

### Error: "file not found"

This can happen if:

1. Some of the input files specified in the workflow `.json` file are not at the correct place.\
   **Solution:** Fix the paths of the input files in the workflow `.json` file.
2. The input files put in `/finngen/…` were not yet synced.\
   **Solution:** Wait until the files are synced. More info here: [Why do I get error 'files missing from /finngen/red/'?](https://github.com/finngen-user-documentation/Draft_of_FG_Analyst_Handbook/blob/master/working-in-the-sandbox/running-analyses-in-sandbox/broken-reference/README.md)