Anaconda Python module with ready set of scientific packages

The Python Lmod modules provide a convenient environment for working with different Python versions (3.9.18, 3.10.13, 3.11.7, 3.12.0), each equipped with pre-installed packages. These modules come pre-configured with both conda and pip environments, allowing users to seamlessly install additional packages using pip.

Loading Python Lmod Modules

To utilize the Python Lmod modules, load the desired version (for instance 3.10.3) with the following command:

module load fg-python/3.10.13

Package Management

• Listing Installed Packages: To view the packages installed in the current Python environment, use:

  pip list

• Checking Package Availability: Confirm if a specific package is available in the environment:

  pip show <package name>

• Interactive Python Session: Open an interactive Python session effortlessly:

  python

• Installing Additional Packages: Install a new package using pip:

  python -m pip install <package name>

Jupyter Integration

After loading the Python Lmod module, you can launch JupyterLab for an interactive working environment

jupyter lab

By default, JupyterLab will start in your current working directory (typically your home directory), which is where your notebooks should be saved.

Accessing data from /finngen/red

The FinnGen RED directory is read-only and intended for data access only. You do not need to copy data from /finngen/red into your home directory to use it in Jupyter.

To make /finngen/red available inside Jupyter, bind-mount it using Singularity:

export SINGULARITY_COMMAND_OPTS="-B /finngen/red"

You may bind-mount multiple directories by adding more -B options if needed.

After this, start JupyterLab from a writable directory, such as your home directory:

jupyter lab

or explicitly:

jupyter lab --notebook-dir="$HOME"

This setup allows you to:

• Read data from /finngen/red (read-only)

• Save notebooks and outputs in your home directory

Troubleshooting: Installed packages not visible in Jupyter

If you have installed Python packages after loading a Python module but they are not available in Jupyter notebooks, you may need to create an IPython kernel for that environment.

To create the kernel, run the following command:

python -m ipykernel install [--user] [--name <machine-readable-name>] [--display-name <"User Friendly Name">]

For example:

python -m ipykernel install --user --name python310

This will create a kernel named python310 ,which can be selected inside JupyterLab and will have access to the packages installed in the loaded module. For more information, please visit the IPython documentation on kernel installation.

Click here to visit the site with the full Jupyter official documentation.

BigQuery Integration

The following BigQuery packages are pre-installed:

• Google Cloud BigQuery

• Pandas_GBQ

  Example command to check BigQuery data in the terminal:

  python -c 'from google.cloud import bigquery; import pandas_gbq; client = bigquery.Client(); query = """ SELECT FINNGENID FROM finngen-production-library.sandbox_tools_r10.finngen_r10_minimum_v1 LIMIT 10 """; df = pandas_gbq.read_gbq(query, project_id = client.project); print(df) '

  The above command retrieves 10 FINNGENIDs from the DF10 release minimum table in BigQuery.

Visualization Packages

The Python Lmod modules include the following visualization packages:

• Plotly

• Matplotlib

• Seaborn

• UpSetPlot

More visualization packages can be added based on your requirements.

Working in Virtual Environments

To activate virtual environments, shell into the container using alias commands that you can see when you do:

module spider <module name>

To create a virtual environment inside the fg-python/3.10.13 Lmod module for instance, follow these steps:

1. Load the Python module:

   module load fg-python/3.10.13

2. Shell into the environment:

   fg-python-shell

3. Create a virtual environment named my-venv:

   python3 -m venv --system-site-packages my-venv

4. Activate the virtual environment:

   source my-venv/bin/activate

5. Install packages within the virtual environment:

   python -m pip install <package name>