|Node Type||Standard||Large Memory||GPU (K40m)||GPU (V100)|
|Number of Nodes||129||6||6||5|
|Processors||2x 2.5 GHz Intel Haswell (Xeon E5-2680v3)||2x 3.0 GHz Intel Skylake (Xeon Gold 6154)||2x 2.2 GHz Intel Broadwell (Xeon E5-2630v4)||2x 2.5 GHz Intel Cascade Lake (Xeon Gold 6248)|
|Cores per Node||24||36||20||40|
|RAM||128 GB (122.8 GB requestable)||1.5 TB (1,542 GB requestable)||64 GB (58.3 GB requestable)||191 GB (184.3 GB requestable)|
|GPU||N/A||N/A||4x Nvidia Tesla K40m||3x Nvidia Tesla V100|
The compute nodes are all interconnected with InfiniBand networking. The InfiniBand fabric is based on the Mellanox enhanced data rate (EDR) platform in the Voltaire GridDirector 4700, which provides 100 Gbps of bandwidth and sub-5μs latency per host. Five Grid Director 4700 switches are connected to each other with 240 Gbps of bandwidth each.
In addition to the InfiniBand networking, there is a gigabit Ethernet network that also connects all of the nodes. This is used for node management and NFS file system access.
To discuss high-speed connections to the Armis2 cluster, please contact firstname.lastname@example.org.
The high-speed home and scratch file systems are provided by Turbo Research Storage. Turbo is a high-capacity, fast, reliable, and secure data storage service that allows investigators across U-M to connect their data to the computing resources necessary for their research, including our Armis2 HPC cluster. Turbo supports storage of sensitive data.
Computing jobs on Armis2 are managed completely through the Slurm workload manager. See the Armis2 User Guide for directions on how to submit and manage jobs. For advanced information on how to use Slurm on Armis2, see the Slurm User Guide for Armis2.
There are three layers of software on Armis2.
The Armis2 cluster runs CentOS Linux 7. We update the operating system on Armis2 as CentOS releases new versions and our library of third-party applications offers support. Due to the need to support several types of drivers (AFS file system drivers, InfiniBand network drivers and NVIDIA GPU drivers) and dozens of third party applications, we are cautious in upgrading and can lag CentOS’s releases by months.
Compilers and Parallel and Scientific Libraries
Armis2 supports the Gnu Compiler Collection, the Intel Compilers, and the PGI Compilers for C and Fortran. The Armis2 cluster’s parallel library is OpenMPI, and the default versions are 1.10.7 (i686) and 3.1.2 (x86_64), and there are limited earlier versions available. Armis2 provides the Intel Math Kernel Library (MKL) set of high-performance mathematical libraries. Other common scientific libraries are compiled from source and include HDF5, NetCDF, FFTW3, Boost, and others.
Software installed on Armis2 must be compatible with these compilers and libraries.
Armis2 supports a wide range of application software. We license common engineering simulation software (e.g. Ansys, Abaqus, VASP). We also have software for statistics, mathematics, debugging and profiling, etc. Please contact us if you wish to inquire about the current availability of a particular application.
Armis2 has 12 total Nvidia Tesla K40m GPUs connected to three nodes, and 15 total Nvidia Tesla V100 GPUs connected to five nodes.
|GPU Model||Nvidia Tesla K40m||Nvidia Tesla V100|
|Peak double precision floating point perf.||1.43 Tflops||7 Tflops|
|Peak single precision floating point perf.||4.29 Tflops||14 Tflops|
|Memory bandwidth (ECC off)||288 GB/sec||900 GB/sec|
|Memory size (GDDR5)||12 GB||16 GB|