1Before you can use Lighthouse, the Principal Investigator (PI) must establish a Slurm account by contacting arcts-support@umich.edu.
Email SupportLighthouse users must be authorized by the PI or a user authorized to make changes to the account. Contact HPC support to request user changes.
EMAIL SUPPORTSee the Lighthouse cheat sheet for a list of common Linux (Bash) and Slurm commands, including Torque and Slurm comparisons.
CHEAT SHEET (PDF)
| Cluster Defaults | Default Value |
|---|---|
| Max walltime | 2 weeks |
| Default walltime (if not specified in job script) | 60 minutes |
| Default memory per CPU (if not specified in job script) | 768 MB |
| Default number of CPUs (if not specified in job script) | no memory specified: 1 core Memory specified: memory/768 = # of cores (rounded down) |
| /scratch quota per root account | 10 TB (see Scratch Storage Policies below) |
| Max queued jobs per user per account | 5,000 |
| Shell timeout if idle: | 24 hours |
DUO two-factor authentication is required to access the majority of U-M services and all HPC services. If you need to set up DUO please visit this page.
You must establish a user login on Lighthouse by filling out this form.
You must be on campus or on the VPN to connect to Lighthouse. If you are trying to log in from off campus, or using an unauthenticated wireless network such as MGuest, you have a couple of options:
Open Terminal and type:
ssh uniqname@lighthouse.arc-ts.umich.edu
You will be required to enter your Kerberos level-1 password to log in. Please note that as you type your password, nothing you type will appear on the screen; this is completely normal. Press “Enter/Return” key once you are done typing your password.
When you’re connecting for the first time, it’s not uncommon to see a message like this one:
The authenticity of host 'lighthouse.arc-ts.umich.edu (141.211.102.84)' can't be established. RSA key fingerprint is 6f:8c:67:df:43:4f:e0:fc:80:5b:49:1a:eb:81:cc:54. Are you sure you want to continue connecting (yes/no)?
This is normal. By saying “yes” you’re accepting the public SSH key for the system. This key will be stored in a local known_hosts file on your system so you won’t be prompted in the future. The keys from Lighthouse will NOT change. So, for example, if you get a new computer and SSH to Lighthouse, you’ll be prompted to add the key again.
We encourage you to compare the fingerprint you’re presented with, when connecting for the first time, to one of the fingerprints below. The format of the fingerprint you’re presented could be dictated by the SSH client on your machine.
| RSA | 6f:8c:67:df:43:4f:e0:fc:80:5b:49:1a:eb:81:cc:54 |
| ECDSA | Dae1G3gu0mtro2Rm15U6l8aQg4bGFnDQJhmGH3k+fKs |
| ED25519 | 9ho43xHw/aVo4q5AalH0XsKlWLKFSGuuw9lt3tCIYEs |
In the example message given above, we are presented with the RSA key fingerprint and its MD5 value, which is the same value as in the above table.
If you’re NOT seeing one of these fingerprints, submit a ticket to arcts-support@umich.edu and do NOT connect to the server via SSH until discussing with an ARC staff member to determine if there is a security issue.
To avoid being prompted to accept the key on a new system you may choose to pre-populate your SSH known_hosts file with the pub keys from Lighthouse. The keys can be found in the FAQ
Download and install PuTTY here.
Launch PuTTY and enter lighthouse.arc-ts.umich.edu as the host name, then click open.
If you receive a “PuTTY Security Alert” pop-up, this is completely normal, click the “Yes” option. This will tell PuTTY to trust the host the next time you want to connect to it. From there, a terminal window will open; you will be required to enter your UMICH uniqname and then your Kerberos level-1 password in order to log in. Please note that as you type your password, nothing you type will appear on the screen; this is completely normal. Press “Enter/Return” key once you are done typing your password.
At the “Enter a passcode or select one of the following options:” prompt, type the number of your preferred choice for DUO authentication.
You can use SFTP (best for simple transfers of small files) or Globus (best for large files or a commonly used endpoint) to transfer data to your /home directory.
On Windows, you can also use WinSCP with similar settings, available alongside PuTTY here.
To copy a single file, type:
scp localfile uniqname@lighthouse-xfer.arc-ts.umich.edu:./remotefile
To copy an entire directory, type:
scp -r localdir uniqname@lighthouse-xfer.arc-ts.umich.edu:./remotedir
These commands can also be reversed in order to copy files from Lighthouse to your machine:
scp -r uniqname@lighthouse-xfer.arc-ts.umich.edu:./remotedir localdir
You will need to authenticate via Duo to complete the file transfer.
Globus is a reliable high performance parallel file transfer service provided by many HPC sites around the world. It enables easy transfer of files from one system to another, as long as they are Globus endpoints.
Globus Online is a web front end to the Globus transfer service. Globus Online accounts are free and you can create an account with your University identity.
Globus Online also allows for simple installation of a Globus endpoint for Windows, Mac, and Linux desktops and laptops.
A non-standard use of Globus Online is that you can use it to copy files from one location to another on the same cluster. To do this use the same endpoint (umich#lighthouse as an example) for both the sending and receiving machines. Setup the transfer and Globus will make sure the rest happens. The service will email you when the copy is finished.
There are Command line tools for Globus that are intended for advanced users. If you wish to use these, contact HPC support.
This is a simple guide to get your jobs up and running. For more advanced Slurm features, see the Slurm User Guide for Lighthouse. If you are familiar with using the resource manager Torque, you may find the migrating from Torque to Slurm guide useful.
On Lighthouse, like the Flux Operating Environment, you can run on hosts owned by a researcher who has granted you access. Unless the researcher has placed additional limits, the only default limit is a 2-week maximum runtime (wallclock). You will submit your jobs to a partition (queue) named after the PI’s uniqname.
To help in the transition, we have included 2 test nodes in your partition. These 2 test nodes are shared with all FOE/Lighthouse researchers, so availability will depend on current demand. This is similar to the environment you previously tested on the Beta cluster (there is a short limit on run times for jobs). You can submit jobs to that partition to validate your workflows. Once the PI has moved their nodes from FOE to Lighthouse, the test nodes will be removed from the PI’s partition and the maximum runtime will be set to 2 weeks.
Most work will be queued to be run on Lighthouse and is described through a batch script. The sbatch command is used to submit a batch script to Slurm. To submit a batch script simply run the following from a shared file system; those include your home directory, /scratch, and any directory under /nfs that you can normally use in a job on Flux. Output will be sent to this working directory (jobName-jobID.log). Do not submit jobs from /tmp or any of its subdirectories.
$ sbatch myJob.sh
The batch job script is composed of three main components:
Example:
#!/bin/bash # The interpreter used to execute the script #“#SBATCH” directives that convey submission options: #SBATCH --job-name=example_job #SBATCH --mail-type=BEGIN,END #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --mem-per-cpu=1000m #SBATCH --time=10:00 #SBATCH --account=test #SBATCH --partition=standard # The application(s) to execute along with its input arguments and options: /bin/hostname sleep 60
How many nodes and processors you request will depend on the capability of your software and what it can do. There are four common scenarios:
This is the simplest case and is shown in the example above. The majority of software cannot use more than this. Some examples of software for which this would be the right configuration are SAS, Stata, R, many Python programs, most Perl programs.
#!/bin/bash #SBATCH --job-name JOBNAME #SBATCH --nodes=1 #SBATCH --cpus-per-task=1 #SBATCH --mem-per-cpu=1g #SBATCH --time=00:15:00 #SBATCH --account=test #SBATCH --partition=standard #SBATCH --mail-type=NONE srun --cpu-bind=none hostname -s
This is similar to what a modern desktop or laptop is likely to have. Software that can use more than one processor may be described as multicore, multiprocessor, or multithreaded. Some examples of software that can benefit from this are MATLAB and Stata/MP. You should read the documentation for your software to see if this is one of its capabilities.
#!/bin/bash #SBATCH --job-name JOBNAME #SBATCH --nodes=1 #SBATCH --cpus-per-task=4 #SBATCH --mem-per-cpu=1g #SBATCH --time=00:15:00 #SBATCH --account=test #SBATCH --partition=standard #SBATCH --mail-type=NONE srun --cpu-bind=none hostname -s
. This is the classic MPI approach, where multiple machines are requested, one process per processor on each node is started using MPI. This is the way most MPI-enabled software is written to work.
#!/bin/bash #SBATCH --job-name JOBNAME #SBATCH --nodes=2 #SBATCH --ntasks-per-node=4 #SBATCH --mem-per-cpu=1g #SBATCH --time=00:15:00 #SBATCH --account=test #SBATCH --partition=standard #SBATCH --mail-type=NONE srun --cpu-bind=none hostname -s
This is often referred to as the “hybrid mode” MPI approach, where multiple machines are requested and multiple processes are requested. MPI will start a parent process or processes on each node, and those in turn will be able to use more than one processor for threaded calculations.
#!/bin/bash #SBATCH --job-name JOBNAME #SBATCH --nodes=2 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=4 #SBATCH --mem-per-cpu=1g #SBATCH --time=00:15:00 #SBATCH --account=test #SBATCH --partition=standard #SBATCH --mail-type=NONE srun --cpu-bind=none hostname -s
| Description | Slurm directive (#SBATCH option) | Lighthouse Usage |
|---|---|---|
| Job name | --job-name=<name> | --job-name=lhjob1 |
| Account | --account=<account> | --account=test |
| Queue | --partition=<partition_name> | --partition=name
The partition name depends on the PI’s requested cluster |
| Wall time limit | --time=<hh:mm:ss> | --time=02:00:00 |
| Node count | --nodes=<count> | --nodes=2 |
| Process count per node | --ntasks-per-node=<count> | --ntasks-per-node=1 |
| Minimum memory per processor | --mem-per-cpu=<memory> | --mem-per-cpu=1000m |
| Request software license(s) | --licenses=<application>@slurmdb:<N> | --licenses=stata@slurmdb:1 requests one license for Stata |
| Request event notification | --mail-type=<events> Note: multiple mail-type requests may be specified in a comma separated list: --mail-type=BEGIN,END,NONE,FAIL,REQUEUE |
--mail-type=BEGIN,END,FAIL |
Please note that if your job is set to utilize more than one node, make sure your code is MPI enabled in order to run across these nodes and you must use srun rather then mpirun or mpiexec. More advanced job submission options can be found in the Slurm User Guide for Lighthouse.
An interactive job is a job that returns a command line prompt (instead of running a script) when the job runs. Interactive jobs are useful when debugging or interacting with an application. The salloc command is used to submit an interactive job to Slurm. When the job starts, a command line prompt will appear on one of the compute nodes assigned to the job. From here commands can be executed using the resources allocated on the local node. NOTE: On the Lighthouse cluster, both a partition and an account must be specified.
[user@lh-login1 ~]$ salloc --account=test --partition=test salloc: Granted job allocation 395814 salloc: Waiting for resource configuration salloc: Nodes lh1113 are ready for job [user@lh1113 ~]$ hostname lh1113.arc-ts.umich.edu [user@lh1113 ~]$
Jobs submitted with salloc and no additional specification of resources will be assigned the cluster default values of 1 CPU and 768MB of memory. The account must be specified; the job will not run otherwise. If additional resources are required, they can be requested as options to the salloc command. The following example job would be appropriate for an MPI job where one wants two nodes with four MPI processes using one CPU on each node with one GB of memory for each CPU in each task. MPI programs run from jobs should be started with srun or one of the other commands that will start MPI programs. Note the --cpu-bind=none option, which is recommended unless you know what an efficient processor geometry for your job is.
[user@lh-login1 ~]$ salloc --nodes=2 --account=test --partition=test --ntasks-per-node=4 --mem-per-cpu=1GB salloc: Granted job allocation 395815 salloc: Waiting for resource configuration salloc: Nodes lh[1112-1113] are ready for job [user@lh1112 ~]$ srun --cpu-bind=none hostname lh1113.arc-ts.umich.edu lh1113.arc-ts.umich.edu lh1113.arc-ts.umich.edu lh1113.arc-ts.umich.edu lh1112.arc-ts.umich.edu lh1112.arc-ts.umich.edu lh1112.arc-ts.umich.edu lh1112.arc-ts.umich.edu [user@lh-login1 ~]$
In the above example srun is used within the job from the first compute node to run a command once for every task in the job on the assigned resources. srun can be used to run on a subset of the resources assigned to the job, though that is fairly uncommon. See the srun man page for more details.
If your allocation includes GPUs, you can utilize them by including a count option from the table below. All counts can be represented by gputype:number or just a number (default type will be used). Available GPU types can be found with the command sinfo -O gres -p <partition>. GPUs can be requested in both Batch and Interactive jobs. Additionally, a user can select the compute mode of GPUs for each job as either exclusive (ARC’s default setting) or shared. Exclusive mode limits each GPU to run only one process at a time, while shared mode allows multiple processes to run simultaneously on a single GPU. See the CUDA Programming Guide for more details. Note, you may query the compute mode from any GPU node by entering the command nvidia-smi -q | grep "Compute Mode", where a result of Default refers to the NVIDIA default of shared mode, as opposed to the ARC default selection of exclusive mode. For example:
$ nvidia-smi -q |grep "Compute Mode"
Compute Mode : Default
| Description | Slurm directive (#SBATCH or srun option) | Example |
|---|---|---|
| GPUs per node | --gpus-per-node=<gputype:number> | --gpus-per-node=2 or --gpus-per-node=v100:2 |
| GPUs per job | --gpus=<gputype:number> | --gpus=2 or --gpus=v100:2 |
| GPUs per socket | --gpus-per-socket=<gputype:number> | --gpus-per-socket=2 or --gpus-per-socket=v100:2 |
| GPUs per task | --gpus-per-task=<gputype:number> | --gpus-per-task=2 or --gpus-per-task=v100:2 |
| Compute Mode | --gpu_cmode=<shared|exclusive> | --gpu_cmode=shared |
| CPUs required per GPU | --cpus-per-gpu=<number> | --cpus-per-gpu=4 |
| Memory per GPU | --mem-per-gpu=<number> | --mem-per-gpu=1000m |
You can see the status of a job with squeue -j <jobID> . Most of a job’s specifications can be seen by invoking scontrol show job <jobID>. More details about the job can be written to a file by using scontrol write batch_script <jobID> output.txt. If no output file is specified, the script will be written to slurm<jobID>.sh.
A job’s record remains in Slurm’s memory for 30 minutes after it completes. scontrol show job will return “Invalid job id specified” for a job that completed more than 30 minutes ago. At that point, one must invoke the sacct command to retrieve the job’s record from the Slurm database.
To view TRES (Trackable RESource) utilization by user or account, use the following commands (substitute your values for bolded parts):
sreport cluster UserUtilizationByAccount start=mm/dd/yy end=mm/dd/yy account=test --tres type
sreport cluster UserUtilizationByAccount start=mm/dd/yy end=mm/dd/yy users=un1,un2 account=test --tres type
sreport cluster AccountUtilizationByUser start=mm/dd/yy end=mm/dd/yy tree account=test --tres type
cpu
mem
node
gres/gpu
For more reporting options, see the Slurm sreport documentation.
You must use Duo authentication to log on to the Lighthouse OnDemand web service. Get more details on the Safe Computing Two-Factor page.
You must establish a user login on Lighthouse by filling out this form.
You must be on campus or on the VPN to connect to Lighthouse OnDemand. If you are trying to log in from off campus or using an unauthenticated wireless network such as MGuest, you should install VPN software on your computer.
Once you are on the University network, follow these instructions to connect:
If you receive a “Bad Request: Your browser sent a request that this server could not understand. Size of request header field exceeds server limit.” error, please clear your browser’s cookies and try to access Great Lakes OnDemand again.
This error can occur when the request header sent by the browser becomes too large. This is typically caused by accumulated cookies and cached data for a specific site; in some cases, corrupted cookies can also contribute to the problem.
At the top of the page, click “Files” and then “Home Directory”. A new tab will be created that contains the File Explorer:
Here you can navigate your home folder. The buttons do the following:
At the top of the home page, click “Jobs” and then “Job Composer”. A new tab will be created that contains the Job Composer:
Upon your first visit to this page, you’ll go through a helpful tutorial. The buttons do the following:
To view active job information, click “Jobs” and then “Active Jobs” from the home page.
This is a simple guide to get your jobs up and running. For more advanced Slurm features and job scripting information, see the Slurm User Guide for Lighthouse. If you are familiar with using the resource manager Torque, you may find the migrating from Torque to Slurm guide useful.
At the top of the home page, click “Interactive Apps” and then select your desired application.
Launches an interactive desktop in a new tab. Specify your account, hours, memory, cores, and partition (usually your PI’s uniqname):
Please note: At this time only node node may be requested with Remote Desktop.
Upon selecting “Launch”, your job will be queued on one of your nodes and shown on the “My Interactive Sessions” screen. As soon as the job’s status is “Running”, you can change remote desktop settings. For slower internet connections, you can try a higher compression and lower image quality using the sliders. Conversely, if you have a fast connection you can lower compression and raise image quality. You can also directly access your node’s terminal by clicking on the hostname (blue button). Once you’re ready to use the desktop, click on “Launch Lighthouse Remote Desktop”:
A remote desktop session will then be opened in a new tab for the requested amount of time. If you finish early, return to the “My Interactive Sessions” tab and delete the job.
Launches an interactive desktop with MATLAB configured and running in a new tab. Specify your desired version, account, hours, partition (usually your PI’s uniqname), and memory (4GB minimum):
Upon selecting “Launch”, your job will be queued on one of your nodes and shown on the “My Interactive Sessions” screen. As soon as the job’s status is “Running”, you can change remote desktop settings. For slower internet connections, you can try a higher compression and lower image quality using the sliders. Conversely, if you have a fast connection you can lower compression and raise image quality. You can also directly access your node’s terminal by clicking on the hostname (blue button). Once you’re ready to use the application, click on “Launch MATLAB”:
A remote desktop session running MATLAB will then be opened in a new tab for the requested amount of time. You may also use the terminal and other basic applications. If you finish early, return to the “My Interactive Sessions” tab and delete the job.
Launches an interactive desktop with RStudio configured and running in a new tab. Specify your desired version, account, hours, cores, partition (standard, gpu, largemem), and memory (2GB minimum):
Upon selecting “Launch”, your job will be queued on one of your nodes and shown on the “My Interactive Sessions” screen. As soon as the job’s status is “Running”, you can change remote desktop settings. For slower internet connections, you can try a higher compression and lower image quality using the sliders. Conversely, if you have a fast connection you can lower compression and raise image quality. You can also directly access your node’s terminal by clicking on the hostname (blue button). Once you’re ready to use the application, click on “Launch RStudio”:
A remote desktop session running RStudio will then be opened in a new tab for the requested amount of time. You may also use the terminal and other basic applications. If you finish early, return to the “My Interactive Sessions” tab and delete the job.
Launches a Jupyter Notebook Server in a new tab. Specify your desired Anaconda Python version, account, hours, partition (usually your PI’s uniqname), cores, and memory per core:
Upon selecting “Launch”, your job will be queued on one of your nodes and shown on the “My Interactive Sessions” screen. As soon as the job’s status is “Running”, you can click on “Connect to Jupyter”:
For instructions on using Jupyter Notebook, see the official documentation.
If you have GPUs in your partition, the form for your desktop or interactive app will automatically display two related fields, Number of GPUs and GPU Compute Mode. Enter the number of GPUs that you wish to use for the job in the appropriate field. Note the maximum allowed value depends on the number of GPUs on your node. ARC’s default setting for Compute Mode is exclusive, where only single process can run at a time on each GPU. Shared mode allows multiple processes to run simultaneously on a single GPU. See the CUDA Programming Guide for more details. Note, you may query the compute mode from any GPU node by entering the command nvidia-smi -q | grep "Compute Mode", where a result of Default refers to the NVIDIA default of shared mode, as opposed to the ARC default selection of exclusive mode.
For example:
$ nvidia-smi -q |grep "Compute Mode"
Compute Mode : Default
On Lighthouse, like the Flux Operating Environment, you can run on hosts owned by a researcher who has granted you access. Unless the researcher has placed additional limits, the only default limit is a 2-week maximum runtime (wallclock). You will submit your jobs to a partition (queue) named after the PI’s uniqname.
To help in the transition, we have included 2 test nodes in your partition. These 2 test nodes are shared with all FOE/Lighthouse researchers, so availability will depend on current demand. This is similar to the environment you previously tested on the Beta cluster (there is a short limit on run times for jobs). You can submit jobs to that partition to validate your workflows. Once the PI has moved their nodes from FOE to Lighthouse, the test nodes will be removed from the PI’s partition and the maximum runtime will be set to 2 weeks.
The Lighthouse cluster uses the Lmod modules system to provide access to centrally installed software. If you used a cluster at U-M previously, then you should review the documentation for the module system as we have changed the configuration to match that used at most national clusters and most other university clusters.
In particular, you should use the command module keyword to look for a module and do not use module available to search for software, as module available will only show software for which all the dependencies (or prerequisites) are already loaded.
So, to search for the software package FFTW, use
$ module keyword fftw
That will show which versions are installed and provide a command to determine what is needed to load it.
Please see our page on using the Lmod modules system for more details and examples.
There are two main categories of software available on the system: software that is installed as part of the installation of the operating system and software that is installed separately. No special action is needed to use the software installed with the operating system. The separately installed software is set up so that you will use a module to use it. The module will set up the environment and make the software available. We do it this way to enable having multiple versions of the same package and to avoid having conflicts between software packages that have mutually exclusive system requirements.
Many of the software packages that are licensed for use on ARC clusters are licensed for a limited number of concurrent uses. If you will use one of those packages, then you must request a license or licenses in your submission script. As an example, to request one Stata license, you would use
#SBATCH --licenses=stata@slurmdb:1
The list of software can be found from Lighthouse by using the command
$ scontrol show licenses
This section will be updated when system level changes are made to Lighthouse. There are currently no updates.
Slurm partitions represent collections of nodes for a computational purpose, and are equivalent to Torque queues.
Each PI’s standard compute nodes are identified by the PI’s uniqname and have a maximum job walltime of 14 days (can be increased up to 4 weeks at the PI’s request).
Limits can be set on a Slurm association or on an Slurm account. This allows a PI to limit individual users or the collective set of users in an account as the PI sees fit. The following values can be used to limit either an account or user association, unless noted otherwise below:
Current Lighthouse limits:
Please contact ARC if you would like to implement any of these limits.
Users must manage data appropriately in their various locations:
Every user has a /scratch directory for every Slurm account they are a member of. Additionally for that account, there is a shared data directory for collaboration with other members of that account. The account directory group ownership is set using the Slurm account-based UNIX groups, so all files created in the /scratch directory are accessible by any group member, to facilitate collaboration.
Example:
/scratch/msbritt_root/msbritt /scratch/msbritt_root/msbritt/bob /scratch/msbritt_root/msbritt/shared_data
Users are able to use /scratch with a size-quota of 10 TB and with an auto-purge policy on unaccessed files, which means that any unaccessed data will be automatically deleted by the system after 60 days. Scratch file systems are not backed up. Critical files should be backed up to another location.
If you are in need of more scratch space for your account please email us at arcts-support@umich.edu. Please note that these requests need to come from an administrator on the account and should include an explanation of why the increase is required.
The Lighthouse high-performance computing system at the University of Michigan has the same security stance as the Great Lakes cluster.
Applications and data are protected by secure physical facilities and infrastructure as well as a variety of network and security monitoring systems. These systems provide basic but important security measures including:
ARC is a division of ITS
