Gitlab Community Edition Instance

Skip to content
Snippets Groups Projects
Commit a369c70e authored by Hauke Kirchner's avatar Hauke Kirchner
Browse files

Update README.md

parent f87756a3
Branches
No related tags found
No related merge requests found
......@@ -116,6 +116,63 @@ sbatch path/to/deep-learning-with-gpu-cores/code/submit_train_dlc.sh
### 1 Reproduce the given examples.
Here you should get some experience with building containers on the cluster, and how to use the containerized software in your workflows on JupyterHub and the HPC. All the relevant code, commands and settings are given above in this README.
### 2 Adapt the container to one of your own use cases.
### 3 **Advanced**: Try to build and/or run containers on your local machine.
\ No newline at end of file
The real power of containers is that you can define your own personal set of software that you can take with you wherever you need it. So let's start with the interesting part and build your own custom containers that fit your needs. Below we have prepared some examples that have worked for us in various projects to give you some inspiration.
```
mkdir ~/tmp && cd $_
cp ~/deep-learning-with-gpu-cores/code/container/{dlc-conda-example.def,build_dlc-conda-example.sh} .
```
You can install RStudio with custom R packages or you chosse to install Julia. Just extend the `%post` section int the definition file. Here you can find some inspiration, but feel encourged to try it with your set of relevant software.
```
%post
# Install RStudio
apt install -y --no-install-recommends software-properties-common dirmngr
wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc
add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/"
add-apt-repository ppa:c2d4u.team/c2d4u4.0+
apt update
add-apt-repository ppa:ubuntugis/ubuntugis-unstable
apt-get update
apt-get install -y libgdal-dev libgeos++-dev libudunits2-dev libproj-dev libx11-dev libgl1-mesa-dev libglu1-mesa-dev libfreetype6-dev libxt-dev libfftw3-dev
apt install -y \
r-base \
r-cran-caret \
r-cran-crayon \
r-cran-devtools \
r-cran-forecast \
r-cran-hexbin \
r-cran-htmltools \
r-cran-htmlwidgets \
r-cran-plyr \
r-cran-randomforest \
r-cran-rcurl \
r-cran-reshape2 \
r-cran-rmarkdown \
r-cran-rodbc \
r-cran-rsqlite \
r-cran-shiny \
r-cran-tidyverse
```
```
# Install Julia
apt install -y julia
echo 'ENV["HTTP_PROXY"] = "http://www-cache.gwdg.de:3128"' >> /etc/julia/startup.jl
echo 'ENV["HTTPS_PROXY"] = "http://www-cache.gwdg.de:3128"' >> /etc/julia/startup.jl
```
The software is now available on the HPC via your own container, but can also be used in our [JupyterHub service](https://jupyter.hpc.gwdg.de/).
### 3 **Advanced**: Try to build and/or run containers on your local machine.
So far we have limited ourselves to the use of containers on our GWDG systems/HPC. You have seen that the whole workflow, from container building to interactive use via JupyterHub to SLURM-based job execution using all the potential of HPC, can be implemented based on our services. However, a big selling point of containers is that you can take them with you. So let us try to build and/or run containers on your local machine. As this is highly dependent on your operating system, we will not provide step-by-step instructions, you will have to figure it out for yourself. In this way you will become more familiar with containers and how to use them on different systems. The Science Domains blog post [Declutter your Python environments](https://gitlab-ce.gwdg.de/hpc-team-public/science-domains-blog/-/blob/main/20230907_python-apptainer.md) gives an example for [Linux systems](https://gitlab-ce.gwdg.de/hpc-team-public/science-domains-blog/-/blob/main/20230907_python-apptainer.md?ref_type=heads#i-want-to-test-it-on-my-own-system). In addition, [UTM](https://mac.getutm.app/) can be a good choice and has been tested for MacOS.
\ No newline at end of file
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment