diff --git a/code/container/README.md b/code/container/README.md index 31baadf981fda8a15c0a563256c10125b2f07e7a..b5d3b8bf19a35a6f0c08cf95e579974e01f0ad6d 100644 --- a/code/container/README.md +++ b/code/container/README.md @@ -116,6 +116,63 @@ sbatch path/to/deep-learning-with-gpu-cores/code/submit_train_dlc.sh ### 1 Reproduce the given examples. +Here you should get some experience with building containers on the cluster, and how to use the containerized software in your workflows on JupyterHub and the HPC. All the relevant code, commands and settings are given above in this README. + ### 2 Adapt the container to one of your own use cases. -### 3 **Advanced**: Try to build and/or run containers on your local machine. \ No newline at end of file +The real power of containers is that you can define your own personal set of software that you can take with you wherever you need it. So let's start with the interesting part and build your own custom containers that fit your needs. Below we have prepared some examples that have worked for us in various projects to give you some inspiration. + +``` +mkdir ~/tmp && cd $_ +cp ~/deep-learning-with-gpu-cores/code/container/{dlc-conda-example.def,build_dlc-conda-example.sh} . +``` + +You can install RStudio with custom R packages or you chosse to install Julia. Just extend the `%post` section int the definition file. Here you can find some inspiration, but feel encourged to try it with your set of relevant software. + +``` +%post + # Install RStudio + apt install -y --no-install-recommends software-properties-common dirmngr + wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc + add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/" + add-apt-repository ppa:c2d4u.team/c2d4u4.0+ + apt update + + add-apt-repository ppa:ubuntugis/ubuntugis-unstable + apt-get update + apt-get install -y libgdal-dev libgeos++-dev libudunits2-dev libproj-dev libx11-dev libgl1-mesa-dev libglu1-mesa-dev libfreetype6-dev libxt-dev libfftw3-dev + + + apt install -y \ + r-base \ + r-cran-caret \ + r-cran-crayon \ + r-cran-devtools \ + r-cran-forecast \ + r-cran-hexbin \ + r-cran-htmltools \ + r-cran-htmlwidgets \ + r-cran-plyr \ + r-cran-randomforest \ + r-cran-rcurl \ + r-cran-reshape2 \ + r-cran-rmarkdown \ + r-cran-rodbc \ + r-cran-rsqlite \ + r-cran-shiny \ + r-cran-tidyverse +``` + +``` + # Install Julia + apt install -y julia + + echo 'ENV["HTTP_PROXY"] = "http://www-cache.gwdg.de:3128"' >> /etc/julia/startup.jl + echo 'ENV["HTTPS_PROXY"] = "http://www-cache.gwdg.de:3128"' >> /etc/julia/startup.jl +``` + +The software is now available on the HPC via your own container, but can also be used in our [JupyterHub service](https://jupyter.hpc.gwdg.de/). + +### 3 **Advanced**: Try to build and/or run containers on your local machine. + +So far we have limited ourselves to the use of containers on our GWDG systems/HPC. You have seen that the whole workflow, from container building to interactive use via JupyterHub to SLURM-based job execution using all the potential of HPC, can be implemented based on our services. However, a big selling point of containers is that you can take them with you. So let us try to build and/or run containers on your local machine. As this is highly dependent on your operating system, we will not provide step-by-step instructions, you will have to figure it out for yourself. In this way you will become more familiar with containers and how to use them on different systems. The Science Domains blog post [Declutter your Python environments](https://gitlab-ce.gwdg.de/hpc-team-public/science-domains-blog/-/blob/main/20230907_python-apptainer.md) gives an example for [Linux systems](https://gitlab-ce.gwdg.de/hpc-team-public/science-domains-blog/-/blob/main/20230907_python-apptainer.md?ref_type=heads#i-want-to-test-it-on-my-own-system). In addition, [UTM](https://mac.getutm.app/) can be a good choice and has been tested for MacOS. \ No newline at end of file