News

The Great Lakes HPC Cluster is the university-wide, shared computational discovery and high-performance computing (HPC) service. It is designed to support both compute- and data-intensive research. 

Great Lakes is available without charge for student teams and organizations who need HPC resources. This program aims to enable students access to high-performance computing to enhance their team’s mission; it is not meant to be used for faculty-led research. Jobs submitted from a student team account will have lower priority and will be able to run when there are sufficient resources available. Currently, we are limiting the resources available to each team (see below), but in the future, we expect to increase the available resources. If your team or organization needs more resources or access to the large memory nodes, an administrator for your team can also request a paid Slurm account which will not have any of the restrictions mentioned below and can work in conjunction with the no-cost team account. 

Access

Your student team/organization must be registered as a Sponsored Student Organizations (SSO). If your team is an SSO and would like an account on Great Lakes, please have a sponsoring administrator email us at arcts-support@umich.edu with your team name, the uniqnames of the users who are allowed to use the account, and a list of the uniqnames of the account administrators who can make decisions about the account.

If you are a member of a team or organization that isn’t registered as an SSO, please email us at arcts-support@umich.edu with the details of your organization and what your HPC needs are.

Limits

Once your account is created, your team will be able to use up to 100 CPUs in the standard partition and 1 GPU in the gpu partition. Jobs will be limited to 24 hours. If members of your team do not have Linux experience at the command-line, they can also use Great Lakes through their browser via Open OnDemand.

If you have any questions, please send an email to arcts-support@umich.edu.

The 2021-22 rates for the Great Lakes Cluster have been approved. These rates represent cost recovery for the Great Lakes Cluster, and do not include any support your unit may choose to provide.

The monthly rate is based on a 30-day month; per-minute rates are included for reference. Charges are based on the percentage of the machine your job requests (in terms of the amount of cores, memory, and GPUs), and its actual runtime.

Estimating Maximum Job Costs

The command can estimate the maximum cost of job based on cpu, memory, gpu, and partition selection. The amount returned is the most a job could cost and does not account for any unit or other subsidies you have access to.

my_job_estimate --help 
my_job_estimate --cores 20 --memory 10gb --partition standard

College of Engineering Funding

The College of Engineering will cost-share 44% of the Great Lakes Cluster rates for sponsored research accounts for researchers in these units who are paying to use the Great Lakes Cluster on unit Shortcodes.

Medical School Funding

The Medical School will cost-share 80% of the Great Lakes Cluster rates for sponsored research accounts for researchers in these units who are paying to use the Great Lakes Cluster on unit Shortcodes.

College of Literature, Science, & the Arts Funding

The College of Literature, Science, & the Arts (LSA) is pleased to announce a cost-share increase for the rest of the 2021-2022 fiscal year from 44% to 80% of the Great Lakes Cluster rates for research accounts for researchers in College units paying for their use of the Great Lakes Cluster on unit Shortcodes. NOTE: This cost-sharing is in place temporarily while a cost-sharing strategy for LSA is devised. Questions should be directed to Luke Tracy (ltracy@umich.edu), Manager of Research Computing Services, LSA. LSA researchers who do not have access to any other account may be eligible to use the accounts provided centrally by LSA. The usage policy and restrictions on these accounts is described in detail on the LSA’s public Great Lakes accounts page. Questions about access or use of these accounts should be sent to arcts-support@umich.edu.

See the LSA funding page for information on funding courses at the College of Literature, Science, and the Arts.

School of Public Health Funding

The School of Public Health will cost-share 80% of the Great Lakes Cluster rates for sponsored research accounts for researchers in these units who are paying for their use of Great Lakes Cluster on unit Shortcodes. NOTE: The SPH cost-sharing is limited to a finite budget. Therefore, SPH faculty and staff must check with the Assistant Dean for Finance to verify if/how much cost-sharing is available.

Getting a Slurm Account

To establish a Slurm account for a class please contact us at arcts-support@umich.edu with the following information:

• Students to be put on the account
• Shortcode to be used for billing
• List of individuals to administer the account

Any limits to be placed on the either the users or the account as a whole

Please note: courses are currently free of charge through the end of winter term 2020. All students will need to have a user login to use the account. Use the Login Request form to get an account.

Example jobs and their charges¹

To help illustrate how the job charges work, here are some examples of differently-sized jobs.

¹ The charges above have been rounded for readability.

If you have questions, please send email to arcts-support@umich.edu.

What is Great Lakes?

Great Lakes is an ARC-TS managed HPC cluster available to faculty (PIs) and their students/researchers. All computational work is scheduled via the Slurm resource manager and task scheduler. For detailed hardware information, see the configuration page. Great Lakes is not suitable for HIPAA or other sensitive data.

What forms do I need to fill out?

1. The Principal Investigator (PI) needs to request a Slurm account, specifying users that can access the account, the people which can administer that account, and payment details.
2. Each user given access to the account must request a user login. Please refer to the Great Lakes User Guide for additional steps and usage information.

What is a UMRCP account and how can I get one on Great Lakes?

The University of Michigan Research Computing Package (UMRCP), provided by ITS, is an investment into the U-M research community via simple, dependable access to several ITS-provided high-performance computing clusters and data storage resources. For more information, please visit our UMRCP page.

Will my Turbo storage be available on Great Lakes?

Since Turbo is a storage service independent of Great Lakes, users that utilized Turbo on Flux will still be able to access their data on Great Lakes.  The cost of Turbo will not change and no data needs to be transferred.  If you have trouble accessing Turbo, please contact arcts-support@umich.edu.

How do I submit jobs using a web interface?

Great Lakes utilizes Open OnDemand to enable web-based job submission, manage the files in their home directory, view/delete active jobs, and open a web terminal session. Users can also use Matlab, Jupyter Notebooks, RStudio, and get a remote desktop.

You must be on campus or on the VPN to connect to Great Lakes OnDemand.  For more information, see the OnDemand section of the Great Lakes User Guide.

How do I view the resource usage on my account?

To view TRES (Trackable RESource) utilization by user or account, use the following commands (substitute bold variables):
Shows TRES usage by all users on account during date range:

sreport cluster UserUtilizationByAccount start=mm/dd/yy end=mm/dd/yy account=test --tres type

Shows TRES usage by specified user(s) on account during date range:

sreport cluster UserUtilizationByAccount start=mm/dd/yy end=mm/dd/yy users=un1,un2 account=test --tres type

Lists users alphabetically along with TRES usage and total during date range:

sreport cluster AccountUtilizationByUser start=mm/dd/yy end=mm/dd/yy tree account=test --tres type

Possible TRES types:
cpu
mem
node
gres/gpu

To view disk usage and availability by user, type:

home-quota -u uniqname

For more reporting options, see the Slurm sreport documentation.

What is a “root (_root) account”?

Each PI or project has a collection of Slurm accounts which could be used for different purposes (e.g. different grants or focuses of research) with different users.  These Slurm accounts are contained within the PI/project’s root account (e.g. research_root).  For example:

researcher_root
    researcher
        user1
        user2
    researcher1
        user2
        user3

These accounts can have different limits on them, and are also collectively limited for /scratch usage and overall cluster usage.

As a PI, how can I limit usage on my account?

Principal Investigators can request that CPU, GPU, memory, billing units, and walltime be limited per user or group of users on their account.  For more information, see the Great Lakes policy documentation.

Limits must be requested by emailing arcts-support@umich.edu.

As a PI, can I purchase my own nodes for Great Lakes?

PIs may purchase hardware for use on the Lighthouse cluster by emailing arcts-support@umich.edu to develop a hardware plan. Lighthouse utilizes the same Slurm job scheduler and infrastructure as Great Lakes, but purchased nodes can be used exclusively by the PI’s group.

What does my job status mean?

When listing your submitted jobs with squeue -u uniqname, the final column titled “NODELIST(REASON)” will give you the reason that the job is not running yet. The possible statuses are:

Resources

This job is waiting for the resources (CPUs, Memory, GPUs) it requested to become available. Resources become available when currently running jobs complete. The job with Resources in the NODELIST(REASON) column is the top priority job and should be started next.

Priority

This job is not the top priority, so it must wait in the queue until it becomes the top priority job. Once it becomes the top priority job, the NODELIST(REASON) column will change to “Resources”. The priority of all pending jobs can be shown with the sprio command. A job’s priority is determined by two factors: fairshare and age. The fairshare factor in a job’s priority is influenced by the amount of resources that have been consumed by members of your Slurm account. More recent usage means a lower fairshare priority. The age factor is determined by the job’s queued time. The longer the job has been waiting in the queue, the higher the age priority.

AssocGrpCpuLimit

This job was submitted with a Slurm account that has a limit set on the number of CPUs that may be used at one time. This limit is set for all jobs by all users of the same Slurm account. Once some of the jobs running under this Slurm account complete, this reason will change to Priority or Resources unless there is some other limit or dependency. All jobs running under a given Slurm account can be viewed by running squeue --account=account_name

AssocGrpGRES

This job was submitted with a Slurm account that has a limit set on the number of GPUs that may be used at one time. This limit is set for all jobs by all users of the same Slurm account. Once some of the jobs running under this Slurm account complete, this reason will change to Priority or Resources unless there is some other limit or dependency. All jobs running under a given Slurm account can be viewed by running squeue --account=account_name

AssocGrpMem

This job was submitted with a Slurm account that has a limit set on the amount of memory that may be used at one time. This limit is set for all jobs by all users of the same Slurm account. Once some of the jobs running under this Slurm account complete, this reason will change to Priority or Resources unless there is some other limit or dependency. All jobs running under a given Slurm account can be viewed by running squeue --account=account_name

AssocGrpBillingMinutes

This job was submitted with a Slurm account that has a limit set on the amount of monetary charges that may be accrued. Jobs that are pending with this reason will not start until the limit has been raised or the monthly bill has been processed.

Dependency

This job has a dependency on another job. It will not start until that dependency is met. The most common dependency is waiting for another job to complete.

QOSMinGRES

This job was submitted to the GPU partition, but did not request a GPU. This job will never start. This job should be deleted and resubmitted to a different partition or if a GPU is needed, resubmitted to the GPU partition with a GPU request. A GPU can be requested by adding the following line to a batch script: #SBATCH --gres=gpu:1

How Can I Access On-Campus Restricted Software?

From the Command Line

Log into an on-campus login node via ssh client to gl-campus-login.arc-ts.umich.edu

From Open On-Demand

Open your browser (Firefox, Edge, or Chrome in an incognito tab – recommended) and navigate to greatlakes-oncampus.arc-ts.umich.edu.

What are the SSH pub keys for Great Lakes?

If you wish to pre-populate your SSH client configuration with the publicly available keys for Great Lakes, they are as follows:

ECDSA:

greatlakes.arc-ts.umich.edu,greatlakes-oncampus.arc-ts.umich.edu,gl-login?.arc-ts.umich.edu,141.211.192.38,141.211.192.39,141.211.192.40,141.211.192.41 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHWel/rAXqIJYxexVzMSlgy/fICWukn8DaOGMPpAomH1E5AhCjrH2zMMTJHtXYsRA+brm/sTbn21Zw+pgpgJSYA=

ED25519:

greatlakes.arc-ts.umich.edu,greatlakes-oncampus.arc-ts.umich.edu,gl-login?.arc-ts.umich.edu,141.211.192.38,141.211.192.39,141.211.192.40,141.211.192.41 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICwaAq9LI48vVO4qbt35Xfz1pi+RE1Krq1iIeJQqoFEw

RSA:

greatlakes.arc-ts.umich.edu,greatlakes-oncampus.arc-ts.umich.edu,gl-login?.arc-ts.umich.edu,141.211.192.38,141.211.192.39,141.211.192.40,141.211.192.41 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA16eDiBWF3SgPQXEeJsH8dsxO8x3o5KkdqWMg/lK57Kpwf4QGXJNvYy0jxSAuKTRim/ob6+nDRH8zIOwnl9tlyEw+8VN3WR8nqBqxX6Km2yzTOMO8Lh7fLuMTZHOdEz0uOn6tBP8LTMtHN9h/fANjKFVl8N+jsejMXrPf0w7jGjc=

On a Mac or Linux machine you’ll add the keys to your known_hosts file.

On a Mac this file is: /Users/<username>/.ssh/known_hosts.
On Linux: /home/<username>/.ssh/known_hosts.

The known_hosts file should have 644 (i.e. -rw-r--r--) permissions.

If you are using an SSH client that is not part of your operating system (e.g. Windows using PuTTY), please see the client documentation referring to host key verification.

A good start for PuTTY users can be found here (section A.2.9 “Is there an option to turn off the annoying host key prompts?)”

Can I get a refund on a failed job?

Any refunds (if any) are at the discretion of ARC and will only only be enacted during system-wide preventable issues. This does not include hardware failure, power failures, job failures, or similar issues.  For more information, see the Great Lakes policies.

If you have a problem not listed here, please send an email to arcts-support@umich.edu.