Lesson 2: Introduction to the DynPkgs Package Manager

../_images/dynpkgs.png

DynPkgs is a package managing system for the DynRes software stack. Its goal is to provide the same, simple and convient interface to work with the DynRes software stack on different platforms. To this end, it supports different package management backends and deployment methods.

Interface

DynPkgs Environments

Environments are used for consistent management of DynPkgs Packages. It is the recommended way to work with the DynRes software stack. The list of available environments can be found here. Additionally custom environments can be defined.

1. Installing Environments:

An environent (i.e. all contained packages) can be installed with the following command:

user@host:~# dynpkgs env_install [environent]

To reinstall (i.e., recompile) all packages of the environent the following command can be used.

user@host:~# dynpkgs env_reinstall [environent]

2. Deploying Environments

An environent can be deployed with the following command:

user@host:~# dynpkgs env_deploy [--nodes=NUM_NODES] [environent]

(environment) user@host:~# ...

(environment) user@host:~# exit

user@host:~#

The –nodes option can be used to specify the number of nodes to deploy the environment on.

DynPkgs Packages

A DynPkgs package defines the installation and environment settings of a single software package. While DynPkgs Environments are the preferred way to manage DynPkgs packages, sometimes the development process requires individual management of packages. The list of available packages can be found here. Additionally, custom packages can be defined.

1. Installing Packages

Install a package:

user@host:~# dynpkgs pkg_install [package]

Reinstall a package:

user@host:~# dynpkgs pkg_reinstall [package]

1. (Un)Loading Packages

Load a package (i.e. set required environment variables):

user@host:~# dynpkgs pkg_load [package]

Unload a package (i.e. unset required environment variables):

user@host:~# dynpkgs pkg_unload [package]

Adding Packages and Environments

In addition to using predefined packages and environements, DynPkgs allows adding new custom packages and environments:

Adding Packages:

user@host:~# dynpkgs pkg_add [OPTIONS]
         OPTIONS:
             --name=[name]               // The name of the package to add (lower-case)
             --giturl=[url]              // The giturl to clone
             --git-branch=[branch]       // The git branch to clone
             --git-commit=[hash]         // The git commit to check out
             --url=[url]                 // The url to fetch a tarball (wget)
             --build-system=[system]     // The build system to use: make, cmake, git-autotools, release-autotools
             --build-deps=[deps]         // Comma seperated list of DynRes packages required for the build

Please verify the correctness of the package recipes:

  • install/install_[package_name].sh

  • load/load_[package_name].sh

  • unload/unload_[package_name].sh

Adding Environments:

user@host:~# dynpkgs env_add [OPTIONS]
         OPTIONS:
             --name=[name]               // The name of the environment
             --packages=[packages]       // Comma seperated list of DynRes packages included in the environment

Hands-On

Managing Environments

We will explore environment management using the preinstalled ‘tutorial_dynreshpc26’ environment.

1. Start Deployment Since all packages of the environment are already installed we can go on and deploy it:

[mpiuser@n1 hpc]$ dynpkgs env_deploy tutorial_dynreshpc26

We can see that many packages are loaded. We have dropped into a new shell with all these packages available on all nodes. This is indicated by the (env_name) prefix:

(tutorial_dynreshpc26) [mpiuser@n1 hpc]$

2. Environment variables Inside the deployment several environment variables are set, including:

  • PATH

  • LIBRARY_PATH

  • LD_LIBRARY_PATH

  • C_INCLUDE_PATH:

(tutorial_dynreshpc26) [mpiuser@n1 hpc]$ echo $PATH

3. Exit Deployment Once we are done with our deployment we can simply exit the deployment shell and come back to a clean environment

(tutorial_dynreshpc26) [mpiuser@n1 hpc]$ exit
[mpiuser@n1 hpc]$ echo $PATH

Managing Packages

Sometimes it is still necessary to manage packages individually:

1. Install/reinstall packages

  • Reinstall a package:

[mpiuser@n1 hpc]$ dynpkgs pkg_install timestamps
[mpiuser@n1 hpc]$ dynpkgs pkg_reinstall timestamps

2. Load packages

[mpiuser@n1 hpc]$ echo $LD_LIBRARY_PATH

[mpiuser@n1 hpc]$ dynpkgs pkg_load timestamps

***********************************************
* Loading TIMESTAMPS...
***********************************************
  ==> Loading Timestamps @ /opt/hpc/install/timestamps

[mpiuser@n1 hpc]$ echo $LD_LIBRARY_PATH
/opt/hpc/install/timestamps/lib:

  • Unload a package

3. Unload packages

[mpiuser@n1 hpc]$ dynpkgs pkg_unload timestamps

***********************************************
* Unloading TIMESTAMPS...
***********************************************
  ==> Unloading Timestamps @ /opt/hpc/install/timestamps
***********************************************
[mpiuser@n1 hpc]$ echo $LD_LIBRARY_PATH

Adding packages

Let’s add a new package called ‘data_redist_test’. The source code is hosted on Gitlab and it uses a simple Makefile. The only dependecy is ‘ompi’.

[mpiuser@n1 hpc]$  dynpkgs pkg_add --name=data_redist_test --git-url=https://gitlab.inria.fr/dynres/utils/data_redist.git --build-system=make --build-deps=ompi

Add environments

To use the package we will add an environemnt called ‘data_redist_test’ containing 2 packages: ‘ompi’, and ‘data_redist_test’.

[mpiuser@n1 hpc]$ dynpkgs env_add --name=data_redist_test --packages=ompi,data_redist_test

Since we have not yet installed all packages of the environment we will use the env_install command:

[mpiuser@n1 hpc]$ dynpkgs env_install data_redist_test

Our new package is now installed. We can go on and deploy the new environment:

[mpiuser@n1 hpc]$ dynpkgs env_deploy data_redist_test

(data_redist_test) [mpiuser@n1 hpc]$ echo $LD_LIBRARY_PATH
/opt/hpc/install/data_redist_test/lib:/opt/hpc/install/ompi/lib:

(data_redist_test) [mpiuser@n1 hpc]$ exit

For Lesson 3 we will use the tutorial_dynreshpc26 environment deployment:

[mpiuser@n1 hpc]$ dynpkgs env_deploy tutorial_dynreshpc26

Ongoing & Future Work:

  • Install backends: NixOs Compose, kameleon, Spack

  • Deploy backends: OAR-Nxc, OAR-kadeploy, SLURM-spack, …

  • GitLab CI (integrated with Docker Cluster)