HPC Updates (Great Lakes, Armis2, and Lighthouse)

We are currently planning the exact times of the maintenance, so we may be updating the duration as our planning continues. Due to maintenance, the high-performance computing (HPC) clusters and their storage systems (/home and /scratch) will be unavailable:

• Great Lakes, Armis2, Lighthouse : Monday, June 5, 2023, 8am – Friday, June 9, 2023, 5pm

Status Pages

Great Lakes

Lighthouse

Armis2

Copy any files you might need during the maintenance window to your local drive using Globus File Transfer.

Use the command “maxwalltime” to discover the amount of time remaining until maintenance begins at the command-line of any cluster login node. Jobs that request more walltime than remains until maintenance will automatically be queued and start once maintenance is complete.

Contact arc-support@umich.edu if you have any questions.

System settings changes

Armis2

• Armis2 now has a 15 minute idle timeout (down from the previous 2 hour limit), to maintain Armis2 PHI compliance.

System hardware changes

Great Lakes

• Recabling the networking for the single-precision GPUs (SGPGU)
• Potentially compete update to the ethernet networking
• Networking switch removals

Armis2 and Lighthouse

• Maintenance on the Modular Data Center (MDC) (power will be out)
• Hardware reorganization at the MDC
• Upgrades to the ethernet and InfiniBand networking systems

System software changes

Great Lakes, Armis2 and Lighthouse

Slurm Changes

Changes in behavior:

• — srun will no longer read in SLURM_CPUS_PER_TASK. This means you will implicitly have to specify –cpus-per-task on your srun calls, or set the new SRUN_CPUS_PER_TASK env var to accomplish the same thing
• — srun –ntasks-per-core now applies to job and step allocations. Now, use of –ntasks-per-core=1 implies –cpu-bind=cores and –ntasks-per-core>1 implies –cpu-bind=threads.
• — srun –overlap now allows the step to share all resources (CPUs, memory, and GRES), where previously –overlap only allowed the step to share CPUs with other steps

NEW Features:

• GPU Compute Mode option in srun/sbatch 
  • –gpu_cmode=<shared|exclusive|prohibited>
    • Set the GPU compute mode on the allocated GPUs to shared, exclusive or prohibited. Default is exclusive

• Email checkbox

• • Prior all email was enabled, now users can opt out with a check-box at the top of the form that is enabled by default.

• Auto Slurm Account list

• • All apps have been updated with a new auto account feature that will collect a list of the account the user has access to and create a pull down list instead of having the user type the account.

• Auto Partition list

• • Partitions are automatically detected, if a new PI is added to LH or A2, or if a new partition is added to GL. 
• Add SLURM_JOB_START_TIME and SLURM_JOB_END_TIME environment variables.
• extra=<string>
  • An arbitrary string enclosed in double quotes if using spaces or some special characters.  Available in job details.Example:srun –time=00:10:00 –gpus-per-node=1 –nodes=1  -A hpcstaff –extra=”Extra Metadata Field :)” –pty /bin/bashscontrol show job 151 | grep -i extraExtra=Extra Metadata Field 🙂
• tres-per-task 
  • For salloc/sbatch/srun. Unified tres in one call
  • Example:
    • without new option:
      • srun –time=00:10:00  –nodes=1  -A hpcstaff -p ‘spgpu’ –ntasks=1 –cpus-per-task=8 –gpus-per-task=1 –pty /bin/bash
    •    with new option:
      • srun –time=00:10:00  –nodes=1  -A hpcstaff -p ‘spgpu’ –ntasks=1 –tres-per-task=cpu:8,gres:gpu:1 –pty /bin/bash

SLURM Install path, now default

• Since new paths are system default paths, SLURM will be available without having to set up an env path. If you have hard-coded the `/opt` path make sure you remove it
• The following are the updated path changes:

Globus changes

• New Armis2 endpoint: “umich#armis2 v2”

User software changes

• During testing, we discovered the transports being chosen for OpenMPI multi-node jobs was causing failures.   We have added an update to the openmpi module which sets the variable `UCX_TLS` to not use the TCP transport layer.  By adding this, when launching a multi-node job, only the InfiniBand network and protocol will be used.  No action or changes are necessary – all necessary updates are applied when loading the openmpi module.

Storage Updates (Turbo, Locker, and Data Den)

We are not planning any major changes to the storage services themselves, the main reason for storage outages will be a network migration to the new MACC core network.  This date is still flexible and subject to change.  If you need access to any data during the outage, you should copy any that you need to your desktop and/or laptop using Globus File Transfer and work on it directly there during the maintenance.

• Storage (Turbo, Locker and Data Den): Tuesday, June 6, 2023, 8am to Thursday, June 8, 2023, 6 pm.

Status Pages

Storage

All Storage Services

• Network migration for all storage systems