# How to know if my pipeline job was failed due preemption of worker VM

In many FinnGen pipelines jobs like Regenie GWAS the task are computed using cheapest VMs available in Google Cloud Platform. These are called SPOT VMs, the noteworthy characteristic of SPOT VM is that it can be preempted. This means that other Google cloud user can reserve computing capacity and take it away from your job. Preemption can occur in any time point of the job and will naturally terminate all processes in the SPOT instance. The task will fail with preemption error and no output or results are provided.

Pipeline in pipeline pipeline runtime environment user can configure the amount how many times task is of retried with SPOT VM. Removal of "preemptible" parameter means that job is run with Standard VM that is roughly 5 time more expensive than SPOT VM.

```
runtime {
...  
    preemptible: 2 ####Fisrt run + 2 retries= 3 trials####  
...
}
```

Cromwell using Google Cloud BATCH API back-end does not currently support automatic retrial of preempted task (all trials failed) with standard VM (22.11.2024, Note there is active development on this feature so hopefully this is soon outdated, <mark style="color:red;">**Solved since 28.11.2024**</mark>: <mark style="color:red;">**Now there is automatic retry with standard VM)**</mark>.

User can see from Pipeline job metadata if the job is failed by preemption. The metadata can be downloaded from Pipelines GUI. [See this link](https://finngen.gitbook.io/finngen-handbook/working-in-the-sandbox/running-analyses-in-sandbox/if-your-pipeline-job-fails#pipeline-job-failing-during-wdl-json-or-input-data-evaluation).

<figure><img src="/files/BoUGBHjlYVmACnu6XTso" alt=""><figcaption><p>Example of preemption error in metadata</p></figcaption></figure>

The easiest way to fix preemption error is to cloning and rerunning the pipeline job. This will add additional 3 preemption trials. Note cromwell is using call caching so it will does not repeat already successfull tasks. One pipeline can be failed even two consecutive time if there is high demand of computing capacity in GCP.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.finngen.fi/faq/about-sandbox/how-to-know-if-my-pipeline-job-was-failed-due-preemption-of-worker-vm.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.