// HACKER NEWS — CYBERSECURITY
Back to the future: modules for Guix packages (2022)
Some things in our software world are timeless. The venerable
Environment Modules are one of these.
If you’ve ever used a high-performance cluster in the last three
decades, chances are you’re already familiar with it. Modules is about
managing software environments, just like Guix is—or, perhaps more
accurately, guix shell.
You will be delighted, or surprised, to learn that Guix
now has a compatibility layer with
Modules.
As Furlani’s 1991 introductory paper
explains,
Modules were—and still are—a key enabler for Unix users, especially in
high-performance computing (HPC). The module command lets users
manipulate their software environment in terms of packages, without
having to be Unix or shell experts; they let them compose packages and
build the software environment of their choice, without interfering with
other users; they give a level of flexibility that Unix alone wouldn’t
provide. The command-line interface is easily understood:
“loads” GCC 11.2 in your shell. You can
“load” and “unload” software components at will:
As an interface, Modules are easy to use and understand.
However, they leave it up to sysadmins (sometimes users) to
actually deploy the software. The common approach has been for
sysadmins to build and install, by themselves, the software that
Modules refer to. The end result is that modules vary from machine to
machine. For example the gcc module shown above might refer to
GCC 11.2 on one cluster and GCC 8 on another; it might have an entirely
different name on a third cluster. Likewise, the python/3.8 module
above might refer to different patch-level versions of Python 3.8, or
it might refer to a variant of Python
built with different dependencies or different build flags.
These issues have been largely mitigated by package managers such as
EasyBuild and Spack: both
automate package builds, and both can generate module
files—Tcl
snippets that define environment variables to set when “loading” a
module. With EasyBuild and Spack, it becomes possible to not only
automate deployment and module file generation, but also to deploy
similar software on different machines.
“Similar”, though, does not mean “the same”. Software built with Spack
or EasyBuild depends on software already available on the host system:
it is built on top of a GNU/Linux distribution, which could be
CentOS 7.4 (released in 2017), or Ubuntu 22.04, or really anything else.
Thus, software installed with these tools depends on software provided
by the underlying distribution, at build time and at run time.
This “hidden dependency” makes it hard to redeploy the exact same
environment on a different machine or at a different point in time: the
same build process
might
fail, or it might succeed
but the resulting software might behave
differently.
Our approach in Guix is to not
have that “hidden dependency”. Instead, the package dependency graph
that Guix manipulates is self-contained: it includes package
definitions for all the user-land software one may use.
The news today is the release of
Guix-Modules, a new tool to
generate module files from
Guix packages. The primary goal, as with the module file generation
tools in EasyBuild and Spack, is to make it easy for HPC cluster
sysadmins to provide a set of modules for their users—more on that
below. Guix-Modules is an extension of Guix. To use it, you need to
install it and to set the GUIX_EXTENSIONS_PATH environment variable,
like so:
Let’s say you want to generate modules to /opt/modules for selected
packages; you can do so by running: