Skip to content

Commit 00db8b2

Browse files
authored
Merge pull request #234 from NYU-RTS/mdweisner-singularity-overlay-2
Update 07_jupyter_with_conda_singularity.mdx
2 parents 6ca4526 + 1b5a5fb commit 00db8b2

File tree

1 file changed

+26
-6
lines changed

1 file changed

+26
-6
lines changed

docs/hpc/09_ood/07_jupyter_with_conda_singularity.mdx

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,25 @@
11
# Jupyter Notebook with Conda/Singularity in OOD
2+
Please note that Greene and Torch organize overlay files and Singularity images in different directories.
23

4+
## Greene Directories
5+
Overlay Files:
6+
```
7+
/scratch/work/public/overlay-fs-ext3/
8+
```
9+
Singularity Files:
10+
```
11+
/scratch/work/public/singularity/
12+
```
13+
14+
## Torch Directories
15+
Overlay Files:
16+
```
17+
/share/apps/overlay-fs-ext3
18+
```
19+
Singularity Files:
20+
```
21+
/share/apps/images/
22+
```
323
## OOD + Singularity + conda
424
This page describes how to use your Singularity with conda environment in OOD GUI at Torch.
525

@@ -38,16 +58,16 @@ The above code automatically makes your environment look for the default shared
3858
```bash
3959
[NetID@log-1 ~]$ mkdir /scratch/$USER/my_env
4060
[NetID@log-1 ~]$ cd /scratch/$USER/my_env
41-
[NetID@log-1 my_env]$ cp -rp /scratch/work/public/overlay-fs-ext3/overlay-15GB-500K.ext3.gz .
61+
[NetID@log-1 my_env]$ cp -rp /share/apps/overlay-fs-ext3/overlay-15GB-500K.ext3.gz .
4262
[NetID@log-1 my_env]$ gunzip overlay-15GB-500K.ext3.gz
4363
```
4464
Above we used the overlay file `overlay-15GB-500K.ext3.gz` which will contain all of the installed packages. There are more optional overlay files. You can find instructions on the following pages: [Singularity with Conda](../07_containers/03_singularity_with_conda.md), [Squash File System and Singularity](../07_containers/04_squash_file_system_and_singularity.md).
4565

4666
### Launch Singularity Environment for Installation
4767
```bash
48-
[NetID@log-1 ~]$ singularity exec --overlay /scratch/$USER/my_env/overlay-15GB-500K.ext3:rw /scratch/work/public/singularity/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif /bin/bash
68+
[NetID@log-1 ~]$ singularity exec --overlay /scratch/$USER/my_env/overlay-15GB-500K.ext3:rw /share/apps/images/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif /bin/bash
4969
```
50-
Above we used the Singularity OS image `cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif` which provides the base operating system environment for the conda environment. There are other Singularity OS images available at `/scratch/work/public/singularity`
70+
Above we used the Singularity OS image `cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif` which provides the base operating system environment for the conda environment. There are other Singularity OS images available at `/share/apps/images`
5171

5272
Launching Singularity with the `--overlay` flag mounts the overlay file to a new directory: `/ext3` - you will notice that when not using Singularity `/ext3` is not available.
5373

@@ -125,7 +145,7 @@ Singularity> exit
125145

126146
# wait to be assigned a node
127147

128-
[NetID@cm001 my_env]$ singularity exec --overlay /scratch/$USER/my_env/overlay-15GB-500K.ext3:rw /scratch/work/public/singularity/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif /bin/bash
148+
[NetID@cm001 my_env]$ singularity exec --overlay /scratch/$USER/my_env/overlay-15GB-500K.ext3:rw /share/apps/images/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif /bin/bash
129149

130150
Singularity> source /ext3/env.sh
131151
# activate the environment
@@ -153,11 +173,11 @@ At the bottom of the file we have the template singularity command.
153173
```bash
154174
singularity exec $nv \
155175
--overlay /scratch/$USER/my_env/overlay-15GB-500K.ext3:ro \
156-
/scratch/work/public/singularity/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif \
176+
/share/apps/images/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif \
157177
/bin/bash -c "source /ext3/env.sh; $cmd $args"
158178
```
159179
:::warning
160-
If you used a different overlay (/scratch/$USER/my_env/overlay-15GB-500K.ext3 shown above) or .sif file (/scratch/work/public/singularity/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif shown above), you MUST change those lines in the command above to the files you used.
180+
If you used a different overlay (/scratch/$USER/my_env/overlay-15GB-500K.ext3 shown above) or .sif file (/share/apps/images/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif shown above), you MUST change those lines in the command above to the files you used.
161181
:::
162182

163183
Edit the default `kernel.json` file by setting PYTHON_LOCATION and KERNEL_DISPLAY_NAME using a text editor like nano/vim.

0 commit comments

Comments
 (0)