Gitlab Community Edition Instance

Skip to content
Snippets Groups Projects
Commit 50dcfb30 authored by Hauke Kirchner's avatar Hauke Kirchner
Browse files

Update docs.hpc.gwdg.de links in README.md

parent d714bd6b
Branches
No related tags found
No related merge requests found
...@@ -34,7 +34,7 @@ me@local-machine~ % ssh glogin ...@@ -34,7 +34,7 @@ me@local-machine~ % ssh glogin
[me@glogin9 ~]$ bash build_dlc-dlwgpu.sh [me@glogin9 ~]$ bash build_dlc-dlwgpu.sh
``` ```
The container can be built on our various HPC clusters using the `build_dlc-*.sh` scripts. Details on how to use Apptainer and the process of building containers are described in our [documentation](https://docs.hpc.gwdg.de/software/apptainer/) and our blog article [Decluttering your Python environments](https://gitlab-ce.gwdg.de/hpc-team-public/science-domains-blog/-/blob/main/20230907_python-apptainer.md). The container can be built on our various HPC clusters using the `build_dlc-*.sh` scripts. Details on how to use Apptainer and the process of building containers are described in our [documentation](https://docs.hpc.gwdg.de/software_stacks/list_of_modules/apptainer/index.html) and our blog article [Decluttering your Python environments](https://gitlab-ce.gwdg.de/hpc-team-public/science-domains-blog/-/blob/main/20230907_python-apptainer.md).
Running `build_dlc-dlwgpu.sh` will build an image with the software used for our workshop example, as defined in `dlc-dlwgpu.def`. Contrary to the traditional way of using conda to install all packages defined in a `requirements.txt` file, pip is used here to reduce the number of software packages used. However, there are good reasons to use conda, so `build_dlc-conda-example.sh` shows a minimal example of installing Python packages in a container using conda (see `dlc-conda-example.def`). Running `build_dlc-dlwgpu.sh` will build an image with the software used for our workshop example, as defined in `dlc-dlwgpu.def`. Contrary to the traditional way of using conda to install all packages defined in a `requirements.txt` file, pip is used here to reduce the number of software packages used. However, there are good reasons to use conda, so `build_dlc-conda-example.sh` shows a minimal example of installing Python packages in a container using conda (see `dlc-conda-example.def`).
If you encounter errors while building the container, have a look at `build_dlc-dlwgpu.log` or `build_dlc-conda-example.log`. You can also use `cat` (in a different terminal) to see the progress of the image building process. If you encounter errors while building the container, have a look at `build_dlc-dlwgpu.log` or `build_dlc-conda-example.log`. You can also use `cat` (in a different terminal) to see the progress of the image building process.
...@@ -73,7 +73,7 @@ Below are two examples of how to use HPC JupyterHub. The first example shows the ...@@ -73,7 +73,7 @@ Below are two examples of how to use HPC JupyterHub. The first example shows the
## Using the container via CLI on the HPC 🏭 ## Using the container via CLI on the HPC 🏭
The full potential of an HPC cluster can only be utilised using the command line interface (CLI). Workflows can be optimized for the available hardware, such as different accelerators (e.g. GPUs, NPU, ...) and highly parallel workflows. Here a simple workflow using our workshops example is shown, based on the same container, that was used in JupyterHub. For more details please have a look at our [documentation](https://docs.hpc.gwdg.de/index.html), e.g. on [Slurm/GPU Usage](https://docs.hpc.gwdg.de/usage_guide/slurm/gpu_usage/index.html) and [GPU Partitions](https://docs.hpc.gwdg.de/compute_partitions/gpu_partitions/index.html). The full potential of an HPC cluster can only be utilised using the command line interface (CLI). Workflows can be optimized for the available hardware, such as different accelerators (e.g. GPUs, NPU, ...) and highly parallel workflows. Here a simple workflow using our workshops example is shown, based on the same container, that was used in JupyterHub. For more details please have a look at our [documentation](https://docs.hpc.gwdg.de/index.html), e.g. on [Slurm/GPU Usage](https://docs.hpc.gwdg.de/how_to_use/slurm/gpu_usage/index.html) and [GPU Partitions](https://docs.hpc.gwdg.de/how_to_use/compute_partitions/gpu_partitions/index.html).
### Interactive HPC usage ### Interactive HPC usage
...@@ -82,7 +82,7 @@ Using an HPC cluster interactively gives you direct access to the cluster's comp ...@@ -82,7 +82,7 @@ Using an HPC cluster interactively gives you direct access to the cluster's comp
```bash ```bash
cd path/to/deep-learning-with-gpu-cores/code/ cd path/to/deep-learning-with-gpu-cores/code/
# get available GPUs with (see docs.hpc.gwdg.de/usage_guide/slurm/gpu_usage/) # get available GPUs with (see https://docs.hpc.gwdg.de/how_to_use/slurm/gpu_usage/index.html)
# sinfo -o "%25N %5c %10m %32f %10G %18P " | grep gpu # sinfo -o "%25N %5c %10m %32f %10G %18P " | grep gpu
# grete # grete
...@@ -106,7 +106,7 @@ exit ...@@ -106,7 +106,7 @@ exit
### HPC usage via batch scripts ### HPC usage via batch scripts
Finally, the defined workflow can be submitted as a job to the workload manager [SLURM](https://docs.hpc.gwdg.de/usage_guide/slurm/index.html). To do this, a job script needs to be defined and submitted via `sbatch`. As you already have a lot of experience with the containerised software environment, starting on your local machine and JupyterHub, a major point of failure when scaling up your analysis and moving to HPC systems has become unlikely. Finally, the defined workflow can be submitted as a job to the workload manager [SLURM](https://docs.hpc.gwdg.de/how_to_use/slurm/index.html). To do this, a job script needs to be defined and submitted via `sbatch`. As you already have a lot of experience with the containerised software environment, starting on your local machine and JupyterHub, a major point of failure when scaling up your analysis and moving to HPC systems has become unlikely.
```bash ```bash
sbatch path/to/deep-learning-with-gpu-cores/code/submit_train_dlc.sh sbatch path/to/deep-learning-with-gpu-cores/code/submit_train_dlc.sh
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment