Example - Tensorflow
The following is a brief example of building a tensorflow container from a DockerHub recipe, and using via the batch command to run a tensorflow project without loading any specific modules.
First, I go to the official dockerHub Tensorflow page, and decide on which release I am interested in. For this example, I will use the latest python3 deployment. Following the instructions provided in the Building Images section:
singularity build <finalImageName>.simg docker://<dockerHubURL>
becomes
singularity build tf-py3.simg docker://tensorflow/tensorflow:1.14.0
Once entering the command, singularity will begin to compile the container. During the build process, the .singularity directory is used as a temporary cache location. Be aware of this is if you are nearing or at the hard quota limit in your home directory.
[[email protected] ~]$ singularity build tf-py3.simg docker://tensorflow/tensorflow:latest-py3
Docker image path: index.docker.io/tensorflow/tensorflow:latest-py3
Cache folder set to /gpfs_home/mk100/.singularity/docker
[6/6] |===================================| 100.0%
Importing: base Singularity environment
Exploding layer: sha256:7b722c1070cdf5188f1f9e43b8413157f8dfb2b4fe84db3c03cb492379a42fcc.tar.gz
Exploding layer: sha256:5fbf74db61f1459176d8647ba8f53f8e6cf933a2e56f73f0e8da81213117b7e9.tar.gz
Exploding layer: sha256:ed41cb72e5c918bdbd78e68f02930a3f1cf1d6079402b0a5b19de8508e67b766.tar.gz
Exploding layer: sha256:7ea47a67709ebea8efed59fbda703dbd00a0d2cae7e2808959744bfa30bfc0e9.tar.gz
Exploding layer: sha256:38e64f576f172feb568fa37796b3f563d0e60a2ba24b1eac2fad31df27214dc5.tar.gz
Exploding layer: sha256:ccae7f1b5fc083f600889513fd8b1d7601bc2923f1ae68f9d2bf66bb090832f9.tar.gz
Exploding layer: sha256:101128cef006ac70fa8be20e4a37ee7e63f24f873e26b44a013fe454894abc8e.tar.gz
Exploding layer: sha256:823f589d99e57b951ef7b273171950b53d68d0bc04eb822f30350057006b59a0.tar.gz
Exploding layer: sha256:602fd2bed0f1c6c68a6c0fb5aec0a864b0fe8858d5a854c4288782373d569599.tar.gz
Exploding layer: sha256:1313b3c833db39b4bf83b56b436cbcd245cfc485c50a300793d5d09c7df210e1.tar.gz
Exploding layer: sha256:51b06a4999c72e68d37588051e489e7864d9a166e3cdd9cfd4e0df9efff476f1.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: tf-py3.simg
Cleaning up...
Once built, you will have a new file in your current directory named tf-py3.simg.
Now that we have our tensorflow container, we can run it either interactively (interactive slurm job using singularity shell) or as a batch job through the existing slurm sbatch command. I will demonstrate a simple sbatch script which will submit a job to launch the container and execute a python script which requires tensorflow.
The sbatch script seen below
#!/bin/bash
#SBATCH -J myTestSingularity
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=1:00:00
#SBATCH --mem=14GB
#SBATCH --partition=batch
# Print key runtime properties for records
echo Master process running on `hostname`
echo Directory is `pwd`
echo Starting execution at `date`
echo Current PATH is $PATH
# Launch singulairty container and execute python command to run tf_NN.py
singularity exec $HOME/simgTest/tensorflowPy3.simg python $HOME/simgTest/tf_NN.py
Notice for previous workflows (non-singularity based), we would have loaded python3 and tensorflow, then had our sbatch script run
python3 myProgram. Now, instead you run the singularity exec command with the desired singularity image (containing tensorflow), and provide the corresponding python myProgram as a command argument.See below attached the sample batch script, and tf_NN.py tensorflow python code to run as this demo.
tf_NN.py
2KB
Text
Tensorflow Sample Python Code
singularity_sbatch_tf3.sh
483B
Text
Sample Batch Script