Deploy containers on a supercomputer
Overview
Teaching: 12 min
Exercises: 23 minQuestions
How can I execute commands in a container with Singularity or Shifter?
How are variables and directories shared between host and container?
Objectives
Download and run containers on a supercomputer
Manage sharing of variables and directories with the host
Run a real-world bioinformatics application in a container
Download and run containers
Before starting, let us cd into the exercises subdirectory of the tutorial repository directory (see also Setup page):
cd ~/sc20-tutorial/exercises
Singularity
Download an old Ubuntu image using:
singularity pull docker://ubuntu:14.04
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 2e6e20c8e2e6 done
Copying blob 95201152d9ff done
Copying blob 5f63a3b65493 done
Copying config a7ecfa19d8 done
Writing manifest to image destination
Storing signatures
[..]
INFO: Creating SIF file...
INFO: Build complete: ubuntu_14.04.sif
Note how you need to prepend docker:// to the image name, to tell Singularity you’re downloading an image in Docker format (the default would be to download a SIF image).
The image file is just in your current directory:
ls
ubuntu_14.04.sif
Now let’s execute some Linux commands from within the container, whoami and lsb_release -a:
singularity exec ubuntu_14.04.sif whoami
tutorial
singularity exec ubuntu_14.04.sif lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.6 LTS
Release: 14.04
Codename: trusty
Note how with Singularity the user in the container is the same as in the host machine.
Singularity has a dedicated syntax to open an interactive shell prompt in a container:
singularity shell ubuntu_14.04.sif
Singularity>
Exit the shell by typing exit or hitting Ctrl-D.
Finally, note you can request Singularity to execute a container straight away, if the image is not available locally it will be pulled under the hood, and stored in the Singularity cache:
singularity exec docker://ubuntu:16.04 cat /etc/os-release
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 2c11b7cecaa5 done
Copying blob 04637fa56252 done
Copying blob d6e6af23a0f3 done
Copying blob b4a424de92ad done
Copying config 4f98a1bce7 done
Writing manifest to image destination
Storing signatures
[..]
INFO: Creating SIF file...
NAME="Ubuntu"
VERSION="16.04.7 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.7 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
By default, the cache is stored in ~/.singularity; this location can be customised using the environment variable SINGULARITY_CACHEDIR.
A subcommand, singularity cache, can be used to manage the cache.
Shifter
Let’s download the same Ubuntu image as above, using shifterimg:
shifterimg pull ubuntu:14.04
2020-11-02T22:38:46 Pulling Image: docker:ubuntu:14.04, status: READY
Locally stored images are managed by Shifter itself:
shifterimg images
mycluster docker READY 0e855866b8 2020-11-02T22:38:46 ubuntu:14.04
What’s the container user with Shifter? Let’s use both id -u and whoami:
shifter --image=ubuntu:14.04 whoami
livetau
shifter --image=ubuntu:14.04 id -u
1000
Again, these come from the host.
You can try more Linux commands:
shifter --image=ubuntu:14.04 lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.6 LTS
Release: 14.04
Codename: trusty
NOTE: If you need to open an interactive shell in the container with Shifter, just execute bash in the container.
Share host environment variables
By default, host variables are shared with the container:
export HELLO="world"
You can access that variable both with Singularity:
singularity exec ubuntu_14.04.sif bash -c 'echo $HELLO'
world
and with Shifter:
shifter --image=ubuntu:14.04 bash -c 'echo $HELLO'
world
Singularity has additional user options to tune the shell environment. In some cases, e.g. when using Python containers, you might need to isolate the container shell from the host, using -e:
singularity exec -e ubuntu_14.04.sif bash -c 'echo $HELLO'
If you need to pass variables to the container in this situation, you can use a dedicated syntax:
export SINGULARITYENV_BYE="moon"
singularity exec -e ubuntu_14.04.sif bash -c 'echo $BYE'
moon
Use host directories
Regarding the default working directory, the two container engines have different behaviours. Let’s see this with an example, the container image marcodelapierre/ubuntu_workdir:14.04, which has with a custom WORKDIR in the Dockerfile:
FROM ubuntu:14.04
WORKDIR "/workdir"
How does Singularity behave?
singularity exec docker://marcodelapierre/ubuntu_workdir:14.04 pwd
/home/tutorial/sc20-tutorial/exercises
Singularity always uses the host current working directory.
Now, how about Shifter?
shifterimg pull marcodelapierre/ubuntu_workdir:14.04
shifter --image=marcodelapierre/ubuntu_workdir:14.04 pwd
/workdir
shifter --image=ubuntu:14.04 pwd
/home/tutorial/sc20-tutorial/exercises
Shifter follows WORKDIR as defined in the Dockerfile. If undefined, it then defaults to the host current directory.
As a consequence, if you want to be sure you’re running in the host current directory, you need to amend the runtime command:
shifter --workdir=$(pwd) --image=marcodelapierre/ubuntu_workdir:14.04 pwd
/home/tutorial/sc20-tutorial/exercises
With both Singularity and Shifter, the container filesystem is read-only, so if you want to write output files you must do it in a bind-mounted host directory.
Typically, HPC administrators will configure the container engine for you, so that host filesystems containing data and software are mounted by default.
In the unlikely scenario where you need to bind-mount additional paths, Singularity offers handy methods for users. For instance:
singularity exec ubuntu_14.04.sif ls /data2
ls: cannot access /data2: No such file or directory
singularity exec -B /data2 ubuntu_14.04.sif ls /data2
file2
or
export SINGULARITY_BINDPATH="/data2"
singularity exec ubuntu_14.04.sif ls /data2
file2
Do It Yourself: BLAST example
Now you’re going to run a BLAST (Basic Local Alignment Search Tool) example with a container from BioContainers.
BLAST is a tool bioinformaticians use to compare a sample genetic sequence to a database of known sequences; it’s one of the most widely used bioinformatics packages.
This example is adapted from the BioContainers documentation.
For this exercise, use Singularity.
Try and achieve what you’re asked to do, use the solution only if you’re lost.
Before you start, change directory to blast:
cd ~/sc20-tutorial/exercises/blast
Pull the image
First, download the following container image:
quay.io/biocontainers/blast:2.9.0--pl526h3066fca_4Solution
singularity pull docker://quay.io/biocontainers/blast:2.9.0--pl526h3066fca_4
Run a test command
Now, run a simple command using that image, for instance
blastp -help, to verify that it actually works.Solution
$ singularity exec blast_2.9.0--pl526h3066fca_4.sif blastp -helpUSAGE blastp [-h] [-help] [-import_search_strategy filename] [..] -use_sw_tback Compute locally optimal Smith-Waterman alignments?
Now, the exercise directory contains a human prion FASTA sequence, P04156.fasta and a gzipped reference database to blast against, zebrafish.1.protein.faa.gz.
First, uncompress the database (you can use host commands for this):
gunzip zebrafish.1.protein.faa.gz
Run the analysis
We need to perform two tasks:
- prepare the zebrafish database with
makeblastdb:makeblastdb -in zebrafish.1.protein.faa -dbtype prot- run the alignment with
blastp:blastp -query P04156.fasta -db zebrafish.1.protein.faa -out results.txtStart from these commands and adapt them so you can run them from within a container.
Solution
singularity exec blast_2.9.0--pl526h3066fca_4.sif makeblastdb -in zebrafish.1.protein.faa -dbtype protBuilding a new DB, current time: 11/16/2019 19:14:43 New DB name: /home/ubuntu/singularity-containers/demos/blast_db/zebrafish.1.protein.faa New DB title: zebrafish.1.protein.faa Sequence type: Protein Keep Linkouts: T Keep MBits: T Maximum file size: 1000000000B Adding sequences from FASTA; added 52951 sequences in 1.34541 seconds.singularity exec blast_2.9.0--pl526h3066fca_4.sif blastp -query P04156.fasta -db zebrafish.1.protein.faa -out results.txt
The final results are stored in results.txt:
less results.txt
Score E
Sequences producing significant alignments: (Bits) Value
XP_017207509.1 protein piccolo isoform X2 [Danio rerio] 43.9 2e-04
XP_017207511.1 mucin-16 isoform X4 [Danio rerio] 43.9 2e-04
XP_021323434.1 protein piccolo isoform X5 [Danio rerio] 43.5 3e-04
XP_017207510.1 protein piccolo isoform X3 [Danio rerio] 43.5 3e-04
XP_021323433.1 protein piccolo isoform X1 [Danio rerio] 43.5 3e-04
XP_009291733.1 protein piccolo isoform X1 [Danio rerio] 43.5 3e-04
NP_001268391.1 chromodomain-helicase-DNA-binding protein 2 [Dan... 35.8 0.072
[..]
When you’re done, quit the view by hitting the q button.
Key Points
Download a container image with
singularity pullorshifterimg pullExecute commands in a container with
singularity execorshifterBy default Singularity and Shifter pass all host variables to the container
By default Singularity uses the host current directory as the container working directory, whereas Shifter gives precedence to
WORKDIRfrom the DockerfileDefine container specific shell variables with Singularity by prefixing them with
SINGULARITYENV_Mount additional host directories with Singularity with the flag
-B, or the variableSINGULARITY_BINDPATH