Beta User Guide

Beta is a preview cluster that allows users to become familiar with SchedMD’s Slurm Workload Manager, which will be used in our upcoming Great Lakes cluster. All standard compute nodes in the Beta cluster have 16 core Intel(R) Xeon(R) CPU E5-2670 processors at 2.60GHz and 64GB memory. There is one GPU node with 8 K20x GPUs.  Beta will leverage some existing infrastructure (minimal software via modules, storage, etc).

Software modules

The Beta cluster uses the Lmod modules system to provide access to centrally installed software. If you used a cluster at U-M previously, then you should review the documentation for the module system as we have changed the configuration to match that used at most national clusters and most other university clusters.

In particular, you should use the command module keyword to look for a module and do not use module available to search for software, as module available will only show software for which all the dependencies (or prerequisites) are already loaded.

So, to search for the software package FFTW, use

$ module keyword fftw

That will show which versions are installed and provide a command to determine what is needed to load it.

Please see our page on using the Lmod modules system for more details and examples.

There are two main categories of software available on the system: software that is installed as part of the installation of the operating system and software that is installed separately. No special action is needed to use the software installed with the operating system. The separately installed software is set up so that you will use a module to use it. The module will set up the environment and make the software available. We do it this way to enable having multiple versions of the same package and to avoid having conflicts between software packages that have mutually exclusive system requirements.

Requesting software licenses

Many of the software packages that are licensed for use on ARC clusters are licensed for a limited number of concurrent uses. If you will use one of those packages, then you must request a license or licenses in your submission script. As an example, to request one Stata license, you would use

#SBATCH --licenses=stata@slurmdb:1

The list of software can be found from Beta by using the command

$ scontrol show licenses

This section will be updated when system level changes are made to Beta. There are currently no updates.

Terms of Usage and User Responsibility

1. Data is not backed up. None of the data on Beta is backed up. The data that you keep in your home directory, /tmp or any other filesystem is exposed to immediate and permanent loss at all times. You are responsible for mitigating your own risk. ARC provides more durable storage on Turbo, Locker, and Data Den. For more information on these, look here.
2. Your usage is tracked and may be used for reports. We track a lot of job data and store it for a long time. We use this data to generate usage reports and look at patterns and trends. We may report this data, including your individual data, to your adviser, department head, dean, or other administrator or supervisor.
3. Maintaining the overall stability of the system is paramount to us. While we make every effort to ensure that every job completes with the most efficient and accurate way possible, the stability of the cluster is our primary concern. This may affect you, but mostly we hope it benefits you. System availability is based on our best efforts. We are staffed to provide support during normal business hours. We try very hard to provide support as broadly as possible, but cannot guarantee support on a 24 hours a day basis. Additionally, we perform system maintenance on a periodic basis, driven by the availability of software updates, staffing availability, and input from the user community. We do our best to schedule around your needs, but there will be times when the system is unavailable. For scheduled outages, we will announce them at least one month in advance on the ARC home page; for unscheduled outages we will announce them as quickly as we can with as much detail as we have on that same page. You can also track ARC on Twitter (@umichARC).
4. Beta is intended only for non-commercial, academic research and instruction. Commercial use of some of the software on Beta is prohibited by software licensing terms. Prohibited uses include product development or validation, any service for which a fee is charged, and, in some cases, research involving proprietary data that will not be made available publicly. Please contact hpc-support@umich.edu if you have any questions about this policy, or about whether your work may violate these terms.
5. You are responsible for the security of sensitive codes and data. If you will be storing export-controlled or other sensitive or secure software, libraries, or data on the cluster, it is your responsibility that is is secured to the standards set by the most restrictive governing rules.  We cannot reasonably monitor everything that is installed on the cluster, and cannot be responsible for it, leaving the responsibility with you, the end user.
6. Data subject to HIPAA regulations may not be stored or processed on the cluster.

• For more information on HIPAA, see the ITS Guide
• For questions about Protected Health Information (PHI), contact Michigan Medicine Corporate Compliance at compliance-Group@med.umich.edu.

USER RESPONSIBILITIES

Users must manage data appropriately in their various locations:

• /home
• /scratch
• /tmp and /var/tmp
• customer-provided NFS

SECURITY ON BETA / USE OF SENSITIVE DATA

The Beta high-performance computing system at the University of Michigan has been built to provide a preview of the Great Lakes HPC cluster. Beta has the same security stance as the Flux cluster.

Applications and data are protected by secure physical facilities and infrastructure as well as a variety of network and security monitoring systems. These systems provide basic but important security measures including:

• Secure access – All access to Beta is via ssh or Globus. Ssh has a long history of high-security.
• Built-in firewalls – All of the Beta computers have firewalls that restrict access to only what is needed.
• Unique users – Beta adheres to the University guideline of one person per login ID and one login ID per person.
• Multi-factor authentication (MFA) – For all interactive sessions, Beta requires both a UM Kerberos password and Duo authentication. File transfer sessions require a Kerberos password.
• Private Subnets – Other than the login and file transfer computers that are part of Beta, all of the computers are on a network that is private within the University network and are unreachable from the Internet.
• Flexible data storage – Researchers can control the security of their own data storage by securing their storage as they require and having it mounted via NFSv3 or NFSv4 on Beta. Another option is to make use of Beta’s local scratch storage, which is considered secure for many types of data. Note: Beta is not considered secure for data covered by HIPAA.