spack_logoThe Advanced Research Computing (ARC) high-performance computing (HPC) cluster has integrated the Scientific Package Manager Spack (https://spack.io) to streamline software development for users. Spack allows users to build software from its source code or install pre-built binaries directly into their environments. The ARC-provided Spack is configured to utilize existing softwares installed on our HPC system as a foundation for handling build dependencies, and simplifying the process of managing various software components required for HPC tasks.

Moreover, users have the flexibility to customize the already existing configuration to their unique needs, such as using custom compilers, or software management for labs, or groups of individuals. This level of customization ensures that users can optimize their computing environment according to their unique requirements, enhancing efficiency and performance for their HPC workflows.

To get started, first thoroughly study and review the official Spack documentation (https://spack.readthedocs.io/en/latest/).

Once familiar with the official Spack documentation, login to our HPC cluster. ARC Spack is provided as a cluster-wide software, available through our module system.

$ module load spack

Once loaded, that’s it! Spack’s setup script is sourced into your linux environment, allowing you to use Spack through the terminal. You can test that Spack is available by entering $ spack or $ echo $SPACK_ROOT

ARC Spack is configured save all custom configurations, environments, software installations (binaries, scripts, source code), and generated module files into your home directories (~/.spack) folder. If this folder does not exist, it’s automatically created. Deletion of this folder results in all custom configurations, environments, and software’s installed being permanently deleted. 

The spack command has many subcommands. You’ll only need a small subset of them for typical usage. To view a list of out-of-box software available for installation run the command

$ spack list

To get more information on a particular package from the Spack list, use Spack info and Spack spec to show the dependencies, and variants enabled by default. The example below illustrates with gmake.

$ spack info gmake


Package: gmake
Description:
GNU Make is a tool which controls the generation of executables and
other non-source files of a program from the program's source files.
Homepage: https://www.gnu.org/software/make/
Preferred version:
4.4.1 https://ftpmirror.gnu.org/make/make-4.4.1.tar.gz
Safe versions:
4.4.1 https://ftpmirror.gnu.org/make/make-4.4.1.tar.gz
4.4 https://ftpmirror.gnu.org/make/make-4.4.tar.gz
4.3 https://ftpmirror.gnu.org/make/make-4.3.tar.gz
4.2.1 https://ftpmirror.gnu.org/make/make-4.2.1.tar.gz
4.0 https://ftpmirror.gnu.org/make/make-4.0.tar.gz
Deprecated versions:
None
Variants:
build_system [generic] generic
Build systems supported by the package
guile [false] false, true
Support GNU Guile for embedded scripting
Build Dependencies:
guile pkgconfig
Link Dependencies:
guile
Run Dependencies:
None
Licenses:
GPL-3.0-only

$ spack spec gmake


Input spec
--------------------------------
- gmake
Concretized
--------------------------------
[+] gmake@4.4.1%gcc@13.2.0~guile build_system=generic arch=linux-rhel8-skylake_avx512
[+] ^gcc-runtime@13.2.0%gcc@13.2.0 build_system=generic arch=linux-rhel8-skylake_avx512

The spec command shows that gmake has only one dependency, and that is the gcc-runtime libraries.

To install $ spack install gmake. To uninstall $ spack uninstall gmake

All Spack installed software automatically integrates with the system’s Lmod module system. Which means you can load Spack software in one of two ways.

$ spack load gmakeor $ module load gmake. Strongly preferred to use $ module load

To verify the software is loaded correctly, and the correct (spack installed version is being used), you can run the following commands:

$ which gmake
~/.spack/opt/spack/gmake-4.4.1-qpr6-gcc-13.2.0/bin/gmake

Note: Every time you intend to load Spack installed software, the Spack module must be loaded as well. Spack installed software are appended to your linux environment only If Spack is available, therefore it’s imperative and important that Spack be loaded as a module dependency. 

That’s it! gmake is now available for use.

Spack simplifies the process of finding available software packages through two primary methods:

  1. Command Line: Users can utilize the command $ spack list within the Spack environment. This command generates a comprehensive list of all available software packages accessible through Spack. It provides a convenient way for users to view the catalog of software available for installation or customization within their computing environment.
  2. Online Repository: Alternatively, users can access the Spack package repository through the website https://packages.spack.io/. This web-based repository serves as an online catalog of software packages compatible with Spack. Users can browse through the repository to explore various available packages, their versions, dependencies, and related information. This web interface provides an intuitive way to search and discover software packages that can be easily integrated into their computational environment using Spack’s package management capabilities.

Once package(s) to be installed are identified, activate the desired Spack environment. $ spack env activate ENV-NAME. Verify you have enough space in your home directory for the installation. $ home-quota will tell you how much space you have remaining. Typically should reserve at least 10GB or more, depending on the size of the installation.

To create a Spack environment (https://spack.readthedocs.io/en/latest/environments.html), run $ spack env create ENV-NAME. A comprehensive discussion is provided in the official Spack documentation.

Spack commands are environment sensitive. For example, the find command shows only the specs in the active environment if an environment has been activated. Similarly, the install and uninstallcommands act on the active environment.

$ spack find
==> In environment myenv
==> No root specs
==> 0 installed packages

Adding Abstract Specs

An Abstract spec is the user-specified spec before Spack has applied any defaults or dependency information. Users can add abstract specs to an environment using the spack add command.

$ spack env activate myenv
$ spack add gmake

The spack addcommand is environment aware. It adds to the currently active environment. All environment aware commands can also be called using the spack -eflag to specify the environment.

Concretizing

A complete detailed discussion on Spack’s concretizing command is here (https://spack.readthedocs.io/en/latest/environments.html#environments-concretization-config). Once specs have been added, they can be concretized.

$ spack concretize

You can also check the programs, and packages to be installed using $ spack spec -lI. The first letter after the hyphen is a lowercase “L”, and the second letter is an uppercase “i”.
This command identifies what dependencies Spack needs in order to install your desired packages along with how it will obtain them. Assuming their are no problems, then it will print a list of the packages and their dependencies, where entries that begin with a green [+] have already been installed somewhere in your local Spack installation, while those that begin with a green [^] are referencing the system installation, and those beginning with a gray - will need to be downloaded and installed.

Installing an Environment

In addition to installing individual specs into an environment, these specs can be installed collectively as the entire environment to your local installation directory (~/.spack) or an alternative directory If you’ve specified in your custom configuration.

$ spack install

Once installed, that’s it! The software packages are available via $ spack load or $ module load.

Generating Lmod modulefiles  for an  Environment

After software has installed successfully, to generate Lmod modulefiles (if not automatically generated), run $ spack modules lmod refresh

The ARC Spack is a cluster-wide software module, preconfigured for a general use case. It comes with preconfigured compilers, flags, software libraries, and Spack specific settings to suit typical usage scenarios across the cluster for scientific research computing. However, we understand that every research is unique, and that can sometimes require customization beyond what is the default. Custom Spack configurations are supported and are discussed in section 1.7.

The following commands can be useful in viewing the details of ARC Spack’s configurations. A full discussion can be found here (https://spack.readthedocs.io/en/latest/configuration.html)

 

Spack Configuration $ spack config get config Display’s general Spack configuration, such as installation directories (reference)
Spack packages $ spack config get packages Display’s package configurations (reference)
Spack compilers $ spack config get compilers Display’s compiler configurations, and availability (reference)
Spack repos $ spack config get repos Display’s available repositories (reference)
Spack Upstreams $ spack config get upstreams Display’s chained Spack installations, package, and module branches (reference)

Multiple compilers and compiler versions are available for building software using Spack, on Great Lakes, Armis2 and Lighthouse. Compilers can be made available to Spack by specifying them manually in compilers.yaml (or spack.yaml for custom environments), or automatically by running $ spack compiler find. To view the list of compilers available, run $ spack compilers

$ spack compilers or $ spack compiler list

==> Available compilers
-- gcc rhel8-x86_64 ---------------------------------------------
gcc@8.4.1 gcc@13.2.0 gcc@11.2.0 gcc@10.3.0
-- intel rhel8-x86_64 -------------------------------------------
intel@2022.0.2

Compiler Info

To view the specific compiler information, run $ spack compiler info. For example, $ spack compiler info gcc@13.2.0

gcc@13.2.0:
paths:
----- cc = /sw/pkgs/arc/gcc/13.2.0/bin/gcc
----- cxx = /sw/pkgs/arc/gcc/13.2.0/bin/g++
----- f77 = /sw/pkgs/arc/gcc/13.2.0/bin/gfortran
----- fc = /sw/pkgs/arc/gcc/13.2.0/bin/gfortran
modules = []
operating system = rhel8

Selecting a Compiler when Installing Software

A compiler and its version can be specified when using the spack install <pkg-name> command, a compiler is specified with % and its version using @, for example, to install gmake using GCC 10.3.0, run: $ spack install gmake%gcc@10.3.0

Default Spack Installations

If you specified a view, you can point your $PATH, $LD_LIBRARY_PATH, etc to the view. If you did not, you can use  $ module load to load individual packages into your environment, ready to use. Note, have to load the Spack module first as a dependency.

$ module load spack boost

To learn more about Spack views see (https://spack.readthedocs.io/en/latest/packaging_guide.html#views). Spack modules are made available through Lmod (https://lmod.readthedocs.io/en/latest) which is an environment management system. For example, by default, Spack software is installed to ~/.spack/opt and the module files are saved in ~/.spack/share/lmod. Therefore, configure Lmod such that it can see module files located at this location, you have to prepend to your $MODULEPATH the exact path where your module files are saved.
This prepending is already done automatically when you load the Spack module. Below is an explanation of how this is done. If you’re install Spack software at it’s default location, then you DO NOT have to perform these steps. However, If you’re installing software in a custom location, such as a Turbo volume, then the following guide could be useful.

Configuring Lmod to View Spack Modules Installed in Non-Default Locations

For example, consider software installed at <code>/turbo/myproject1/software/my_project</code> Given a directory tree structure as:

$ tree /turbo/myproject1/software/my_project/share/spack/lmod/linux-rhel8-x86_64
/turbo/myproject1/software/my_project/share/spack/lmod/linux-rhel8-x86_64
├── gcc
│   ├── 10.3.0
│   │   ├── gcc-runtime
│   │   │   └── 10.3.0-pglmvn4.lua
│   │   └── gmake
│   │   └── 4.4.1-tdqh6jn.lua
│   └── 13.2.0
│   ├── boost
│   │   └── 1.83.0-uejfb3g.lua
│   └── gcc-runtime
│   └── 13.2.0-msqwczb.lua

To make packages built by the gcc compiler, version 10.3.0 available through the module system, prepend to $MODULEPATH /turbo/myproject1/software/my_project/share/spack/lmod/linux-rhel8-x86_64/gcc/10.3.0

In addition, make packages built by the gcc compiler, version 13.2.0 available through the module system, prepend to $MODULEPATH /turbo/myproject1/software/my_project/share/spack/lmod/linux-rhel8-x86_64/gcc/13.2.0

$ export MODULEPATH=/turbo/myproject1/software/my_project/share/spack/lmod/linux-rhel8-x86_64/gcc/10.3.0:$MODULEPATH
$ export MODULEPATH=/turbo/myproject1/software/my_project/spack/lmod/linux-rhel8-x86_64/gcc/13.2.0:$MODULEPATH

Verify Result:
$ module avail

-------------------- /turbo/myproject1/software/my_project/share/spack/lmod/linux-rhel8-x86_64/gcc/13.2.0 --------------------
boost/1.83.0-uejfb3g gcc-runtime/13.2.0-msqwczb (D)

-------------------- /turbo/myproject1/software/my_project/share/spack/lmod/linux-rhel8-x86_64/gcc/10.3.0 --------------------
gcc-runtime/10.3.0-pglmvn4 gmake/4.4.1-tdqh6jn

Customized Spack Configuration & Environment

The ARC Spack configuration aims to make installing software simple, so spack install gmakeshould, in many cases, just work!, and install software under your $HOME/.spack/ directory. However in most cases it is better to create and use a Spack environment fine-tuned to your preferences.

  • You want to install different software to different locations (for example, you build software for two different projects, or research groups, and want to install some software to /turbo/project1 and some to /turbo/project2
  • You want to install a collection of software into a single tree, where said three has all the binaries, and alternative tree has all the libraries, etc.
  • You have software built outside of Spack, and need Spack to build software on your external build, rather than its default approach for building each dependency package.
  • You want to use different conventions for directory structure or module file names than the defaults set by ARC.

A Spack environment (https://spack.readthedocs.io/en/latest/environments.html) is a directory with a spack.yamlfile that outlines software to build, and the configuration modifications to use while building it. To create and activate an environment.

$ module load spack
$ spack env create my-env
$ spack env activate my-env

By default, this environment will be identical to the ARC Spack configuration, i.e it will install software to your default location, but you could change this in the spack.yaml file. For example:


spack:
config:
install_tree:
# install things into the `install` subdirectory of this environment:
root: /path/to/install
# alternatively: install things into my project's /turbo/myproject1/software:
# root: /turbo/myproject1/software/my_project
# put modulefiles generated by Spack in this environment directory:
module_roots:
tcl: /path/to/install/modules
lmod: /path/to/install/lmod
view: /path/to/my_tree
specs: []

Additionally, If your software stack needs something that was installed external to Spack you can add a packages section, to account for it:

spack:
# [snip]
packages:
# use the ncl from "module load ncl"
ncl:
externals:
- spec: ncl@6.5.0
modules: [ncl/6.5.0]
# use my local install of fftw 3.3.8, as the preferred fftw version:
fftw:
version: [3.3.8]
externals:
- spec: fftw@3.3.8
prefix: /path/to/my/fftw/install

For complete, comprehensive and further details about Spack environments, see https://spack.readthedocs.io/en/latest/environments.html

Architecture

Package

“Source code” describing a piece of software and how to build it (actually a Python class, defined in package.py), in a directory with any patches that might need to be applied first. The Spack Packaging Guide has details about creating and maintaining packages (https://spack.readthedocs.io/en/latest/packaging_guide.html).

Repo

A collection (“repository” of packages. Almost everything is in the “builtin” repo, but Spack has a “repos” config section where you can specify locations and order of repos to search. See the Spack package repository documentation (https://spack.readthedocs.io/en/latest/repositories.html).

Spec

Spack defines a domain-specific-language for describing the build parameters and dependencies with which a package was, or should be, built. The Spack specs and dependencies documentation describes this in detail (https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies).

Variant

A variant is a selectable build option for a package. Available variants for a package are defined in the Spack package. You can, in spec, enable (+) or disable(- or ~) or set (name=value) a variant. Variants usually correspond to a ./configure or cmake setting.

Installation tree

A directory for Spack-installed software. Spack allows specifications of a directory structure and naming convention for installed software. At ARC, this defaults to your $HOME directories .spack folder (~/.spack). This can be modified through Spack environments.

Upstream

Spack uses Upstream to satisfy dependencies in other locations, called “upstream” locations. At ARC there are two upstream locations. To view about them, run $ spack config get upstreams

Buildcache

As a way to avoid repeatedly building the same software, Spack supports packaging up built software for reuse. These pre-built packages are held in a “buildcache”, and a subsequent spack installcan simply unpack the cached package, instead of rebuilding it.