.. _tutorials-dpp-2: ##################################################### Lesson 2: Introduction to the DynPkgs Package Manager ##################################################### .. image:: ../images/dynpkgs.png :align: left :width: 100px DynPkgs_ is a package managing system for the DynRes software stack. Its goal is to provide the same, simple and convient interface to work with the DynRes software stack on different platforms. To this end, it supports different package management backends and deployment methods. .. _DynPkgs: https://dynres.readthedocs.io/projects/DynPkgs/en/latest/ Interface --------- DynPkgs Environments ~~~~~~~~~~~~~~~~~~~~ Environments are used for consistent management of DynPkgs Packages. It is the recommended way to work with the DynRes software stack. The list of available environments can be found `here `_. Additionally custom environments can be defined. 1. Installing Environments: """"""""""""""""""""""""""" An environent (i.e. all contained packages) can be installed with the following command: .. code-block:: console user@host:~# dynpkgs env_install [environent] To reinstall (i.e., recompile) all packages of the environent the following command can be used. .. code-block:: console user@host:~# dynpkgs env_reinstall [environent] 2. Deploying Environments """"""""""""""""""""""""" An environent can be deployed with the following command: .. code-block:: console user@host:~# dynpkgs env_deploy [--nodes=NUM_NODES] [environent] .. code-block:: console (environment) user@host:~# ... .. code-block:: console (environment) user@host:~# exit .. code-block:: console user@host:~# The --nodes option can be used to specify the number of nodes to deploy the environment on. DynPkgs Packages ~~~~~~~~~~~~~~~~ A DynPkgs package defines the installation and environment settings of a single software package. While DynPkgs Environments are the preferred way to manage DynPkgs packages, sometimes the development process requires individual management of packages. The list of available packages can be found `here `_. Additionally, custom packages can be defined. 1. Installing Packages """""""""""""""""""""" Install a package: .. code-block:: console user@host:~# dynpkgs pkg_install [package] Reinstall a package: .. code-block:: console user@host:~# dynpkgs pkg_reinstall [package] 1. (Un)Loading Packages """"""""""""""""""""""" Load a package (i.e. set required environment variables): .. code-block:: console user@host:~# dynpkgs pkg_load [package] Unload a package (i.e. unset required environment variables): .. code-block:: console user@host:~# dynpkgs pkg_unload [package] Adding Packages and Environments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In addition to using predefined packages and environements, DynPkgs allows adding new custom packages and environments: Adding Packages: ~~~~~~~~~~~~~~~~ .. code-block:: console user@host:~# dynpkgs pkg_add [OPTIONS] OPTIONS: --name=[name] // The name of the package to add (lower-case) --giturl=[url] // The giturl to clone --git-branch=[branch] // The git branch to clone --git-commit=[hash] // The git commit to check out --url=[url] // The url to fetch a tarball (wget) --build-system=[system] // The build system to use: make, cmake, git-autotools, release-autotools --build-deps=[deps] // Comma seperated list of DynRes packages required for the build Please verify the correctness of the package recipes: - install/install_[package_name].sh - load/load_[package_name].sh - unload/unload_[package_name].sh Adding Environments: ~~~~~~~~~~~~~~~~~~~~ .. code-block:: console user@host:~# dynpkgs env_add [OPTIONS] OPTIONS: --name=[name] // The name of the environment --packages=[packages] // Comma seperated list of DynRes packages included in the environment Hands-On -------- **Managing Environments** We will explore environment management using the **preinstalled** 'tutorial_dynreshpc26' environment. **1. Start Deployment** Since all packages of the environment are already installed we can go on and **deploy** it: .. code-block:: console [mpiuser@n1 hpc]$ dynpkgs env_deploy tutorial_dynreshpc26 We can see that many packages are loaded. We have dropped into a new shell with all these packages available on all nodes. This is indicated by the (env_name) prefix: .. code-block:: console (tutorial_dynreshpc26) [mpiuser@n1 hpc]$ **2. Environment variables** Inside the deployment several environment variables are set, including: - PATH - LIBRARY_PATH - LD_LIBRARY_PATH - C_INCLUDE_PATH: .. code-block:: console (tutorial_dynreshpc26) [mpiuser@n1 hpc]$ echo $PATH **3. Exit Deployment** Once we are done with our deployment we can simply exit the deployment shell and come back to a clean environment .. code-block:: console (tutorial_dynreshpc26) [mpiuser@n1 hpc]$ exit [mpiuser@n1 hpc]$ echo $PATH **Managing Packages** Sometimes it is still necessary to manage packages individually: **1. Install/reinstall packages** - Reinstall a package: .. code-block:: console [mpiuser@n1 hpc]$ dynpkgs pkg_install timestamps [mpiuser@n1 hpc]$ dynpkgs pkg_reinstall timestamps **2. Load packages** .. code-block:: console [mpiuser@n1 hpc]$ echo $LD_LIBRARY_PATH [mpiuser@n1 hpc]$ dynpkgs pkg_load timestamps *********************************************** * Loading TIMESTAMPS... *********************************************** ==> Loading Timestamps @ /opt/hpc/install/timestamps [mpiuser@n1 hpc]$ echo $LD_LIBRARY_PATH /opt/hpc/install/timestamps/lib: - Unload a package **3. Unload packages** .. code-block:: console [mpiuser@n1 hpc]$ dynpkgs pkg_unload timestamps *********************************************** * Unloading TIMESTAMPS... *********************************************** ==> Unloading Timestamps @ /opt/hpc/install/timestamps *********************************************** [mpiuser@n1 hpc]$ echo $LD_LIBRARY_PATH **Adding packages** Let's add a new package called 'data_redist_test'. The source code is hosted on Gitlab and it uses a simple Makefile. The only dependecy is 'ompi'. .. code-block:: console [mpiuser@n1 hpc]$ dynpkgs pkg_add --name=data_redist_test --git-url=https://gitlab.inria.fr/dynres/utils/data_redist.git --build-system=make --build-deps=ompi **Add environments** To use the package we will add an environemnt called 'data_redist_test' containing 2 packages: 'ompi', and 'data_redist_test'. .. code-block:: console [mpiuser@n1 hpc]$ dynpkgs env_add --name=data_redist_test --packages=ompi,data_redist_test Since we have not yet installed all packages of the environment we will use the env_install command: .. code-block:: console [mpiuser@n1 hpc]$ dynpkgs env_install data_redist_test Our new package is now installed. We can go on and deploy the new environment: .. code-block:: console [mpiuser@n1 hpc]$ dynpkgs env_deploy data_redist_test (data_redist_test) [mpiuser@n1 hpc]$ echo $LD_LIBRARY_PATH /opt/hpc/install/data_redist_test/lib:/opt/hpc/install/ompi/lib: (data_redist_test) [mpiuser@n1 hpc]$ exit For Lesson 3 we will use the tutorial_dynreshpc26 environment deployment: .. code-block:: console [mpiuser@n1 hpc]$ dynpkgs env_deploy tutorial_dynreshpc26 Ongoing & Future Work: ---------------------- - Install backends: NixOs Compose, kameleon, Spack - Deploy backends: OAR-Nxc, OAR-kadeploy, SLURM-spack, ... - GitLab CI (integrated with Docker Cluster)