> For the complete documentation index, see [llms.txt](https://docs.finngen.fi/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.finngen.fi/working-in-the-sandbox/running-large-local-models-llms-in-the-sandbox.md).

# Running large local models (LLMs) in the sandbox

**Summary:** Currently there is no hosted LLM service (such as Gemini or Vertex AI) available inside the Sandbox. You can, however, run an open-weight model **locally on your own IVM**, using the GPU machines. Local models are treated like any other data processing in the Sandbox, so the usual rules apply: an approved analysis proposal and the Sandbox IVM tool request handling policy.

***

## What is and isn't available

| Hosted LLM services (Gemini, Vertex AI, other external APIs) | **Not generally available**. The Sandbox has no internet access. We plan to enable LLM models via Vertex AI in the future, but there is no date yet. Piloting is ongoing in selected Sandboxes |
| ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Running an open-weight model locally on your IVM             | **Allowed**, subject to the rules below.                                                                                                                                                       |
| GPU machines                                                 | **Available.** The standard GPU machine has an NVIDIA Tesla T4 (16 GB).                                                                                                                        |
| Larger GPU machines                                          | **On request, no guarantee.** These are scarce in our data centres. See "Choosing a machine".                                                                                                  |
| Pre-installed model weights                                  | **None.** You need to bring your own (see "Getting model weights into the Sandbox").                                                                                                           |

Because the model runs inside your IVM, the data you give it stays inside the Sandbox.

## Before you start

1. **Analysis proposal.** Work on FinnGen data in the Sandbox, including using an LLM on it, must be covered by an approved analysis proposal. See [When do I need to submit an analysis proposal?](https://docs.finngen.fi/finngen-data-specifics/about-analysis-proposals/when-do-i-need-to-submit-an-analysis-proposal). Make sure the proposal describes the LLM-based processing.
2. **Tool request.** A model you bring in is a user-imported tool, so the [Sandbox IVM tool request handling policy](https://docs.finngen.fi/working-in-the-sandbox/quirks-and-features/sandbox-ivm-tool-request-handling-policy) applies. In practice, send a request to <finngen-servicedesk@helsinki.fi> that names the model, where it comes from and its licence. Prefer `safetensors` files and say where the model comes from (see below).
3. **Licence.** The policy asks for free tools under an open-source licence approved by the OSI. Check the licence on the model card before you ask. Models under custom or research-only licences may not qualify.

## Getting model weights into the Sandbox

The Sandbox has no internet access, so you cannot download weights from Hugging Face on the IVM. Instead:

1. Download the model files on a machine that has internet access, from the official model repository. Keep the licence file.
2. Upload them to your own IVM through the **greenuploads** bucket. Follow [How to upload to your own IVM via FinnGen Green](https://docs.finngen.fi/working-in-the-sandbox/quirks-and-features/how-to-upload-to-your-own-ivm-via-finngen-green). Your bucket name is in the notification you received, or in the `buckets.txt` file on your IVM desktop.
3. Wait for the automatic virus scan (about 5 to 20 minutes). Cleared files appear under `/finngen/green` on your IVM.
4. Copy the model to your home disk and load it from that local path.

**Size limit.**  If a single file is larger, split or compress it before you upload.

#### Use trusted models and safe file formats

Model files are code-adjacent: a malicious model can compromise your IVM and the data on it. The virus scan does not check for this.

* **Only use models from sources you can trust.** Use the official repository of the model's developer (for example the organisation's own page on Hugging Face) and not re-uploads, "fine-tuned" or "uncensored" variants from unknown accounts. Custom models from unknown sources are a known supply-chain risk. Say in your tool request exactly where the model comes from.
* **Prefer the `safetensors` format.** It stores only the weights and cannot carry executable code. Avoid pickle-based files (`.bin`, `.pt`, `.pth`, `.pkl`, `.ckpt`), because loading them can run arbitrary code. If a model is only available in a pickle-based format, ask the service desk before you upload it.
* **Don't allow custom code from the model.** Don't pass `trust_remote_code=True` when loading a model. It runs Python code shipped with the model repository.

## Choosing a machine

Open a GPU profile when you start your IVM. The standard one has a Tesla T4 with 16 GB of GPU memory. As a rough guide for what fits:

* Model weights in 16-bit precision need about 2 GB of GPU memory per billion parameters, plus room for the context and activations. A 3B model fits comfortably. A 7B model is very tight on a T4, so use an 8-bit or 4-bit quantised version.
* The T4 does not support bfloat16 well. Use float16.
* If your model does not fit, ask <finngen-servicedesk@helsinki.fi> about a larger GPU machine. We can try to book one, but we cannot guarantee availability.

## Setting up and running

* Install Python packages (for example Hugging Face Transformers and PyTorch) from the internal PyPI repository, in a virtual environment on your home disk.
* Large installs and model files can fill `/tmp` or your home disk ("no space left on device"). Check free space with `df -h /home/ivm`. If you need more room, attach a persistent disk to your IVM (you can delete it when you no longer need it) and use a folder on it as `TMPDIR` for the install, for example `TMPDIR=<your-disk>/tmp pip install --no-cache-dir <package>`. You can keep the virtual environment and the model files on that disk as well.
* Load the model from the local path, and set `HF_HUB_OFFLINE=1` so that no library tries to reach the internet.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
path = "/home/ivm/models/<your-model>"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype="float16", device_map="auto", use_safetensors=True)
```

## Results and downloads

Your outputs follow the normal Sandbox rules. Downloading results from the Sandbox requires an approved analysis proposal and goes through the usual download process.\
**Finetuned/retrained model weigths cannot be exported from the Sandbox environment with reqular Download request**. These will need separate revision and user is responsible to provide evidence that the model data contains only anonymised data (See futher information <https://findata.fi/en/services-and-instructions/producing-anonymous-results/#Examples-to-support-the-application-of-the-instructions>).
