Turbo Storage User Guide

Turbo is a high-capacity, reliable, secure, and fast storage solution. It is tuned for large files (which corresponds roughly to be files in 1 megabyte in size or larger), but it is still capable of efficiently storing small files, such as: word documents, spreadsheets, image files, etc.

Turbo enables investigators across the University of Michigan to store and access data needed for their research via a local computer in a lab, office or our High Performance Computing Clusters, such as: Great Lakes, Lighthouse, and Armis2. Additionally, it supports the storing of sensitive data on HIPAA compliant systems, such as the Armis2 cluster managed by ARC.

Turbo Rates

• 10TB Turbo storage is available as part of the no-cost U-M Research Computing Package (UMRCP). Additional allocation is available for a fee as noted below (shortcode required).
• Unreplicated: $6.49 per terabyte per month ($77.88 per year)
• Replicated: $13.02 per terabyte per month ($156.22 per year)
• Not sure if this is the right storage service for you? Try the ITS Data Storage Finder to find storage solutions to meet your needs.

Get started

Visit our Ordering Storage page to see an overview of what is needed to set up a new Turbo volume. Use your UMICH (Level-1) credentials and Duo two-factor authentication to login via the ITS Service Request (SRS) Portal or the ITS General Computing provision page.

SRS support

Information about the SRS and how to use it can be found throughout the SRS system.

Here are some additional resources:

• Using the Service Request System
• Access and Permissions
• Chartfield Change Request
• Format the Inventory and Location Report

If you have any questions, please contact us at arc-support@umich.edu.

Be aware that if you have a multi-protocol or NFS Turbo volume that is mounted on the Great Lakes HPC Cluster, there is no need to set up a separate Globus endpoint.

Data Den is a service for preserving electronic data generated from research activities. It is a low-cost, highly durable storage system and is the largest storage system operated by ARC.

Storing of sensitive data (including HIPAA and FERPA) is now supported (visit the Sensitive Data Guide for full details). This service is part of the U-M Research Computing Package (UMRCP) that provides storage allocations to researchers. Most researchers will not have to pay for Data Den.

Data Den is a disk-caching, tape-backed archive optimized for data that is not regularly accessed for extended periods of time (weeks to years). Data Den can hold onto research data for many years, thereby meeting the needs of the grant requirements specifically NIH and NSF.

Data Den is not meant to replace active storage services like Turbo and Locker. It is best used for data that is set aside and is not being used at all, but still needs to be stored. Data Den is best accessed through the Globus data management sharing system to move data into and out of the service. Data Den is only available in a replicated format.

Data Den can be part of a well-organized data management plan providing international data sharing, encryption, and data durability.

Data Den Rates

• 100TB of Data Den storage is available as part of the no-cost U-M Research Computing Package (UMRCP). Additional allocation is available for a fee as noted below.
• Data Den service rates are $20.04 per replicated terabyte (TB) per year ($1.67/TB/month).
• Starting July 1, 2023, the new LSA Research Storage Funding Model will provide unlimited Data Den Research Archive Storage. Please see the LSA Research Storage Portal to learn more.
• Not sure if this is the right storage service for you? Try the ITS Data Storage Finder to find storage solutions to meet your needs.

Getting Started – UPDATED 2023

• Project admins can use the Portal to update user access lists
• Use Globus.org collection UMich ARC Non-Sensitive Data Den Volume Collection your path will be in  /<volume>
• Active Archive users that have archives mounted on ARC cluster login nodes path will be /nfs/dataden/<volume>

Requesting or Modifying a Data Den Storage Volume

Most should start with the U-M Research Computing Package rather than these instructions.

Visit the ITS Service Request System (SRS).

Please include the following:

• Amount of storage needed (in 1TB increments)
• MCommunity Group name (group members will receive service-related notification, and can request service changes)
• Numeric group ID of the group that will have access to files at the top level directory.
• Numeric user ID of person who will administer the top level directory and grant access to other users.

Globus Server Endpoint

Data Den supports the use of Globus servers to provide high performance transfers, data sharing and access to Data Den from off campus. To access Data Den via Globus, request your volume be added to Globus. The globus collection is UMich ARC Non-Sensitive Data Den Volume Collection for non-sensitive data.

Bundling Files

Because of the design of Data Den, often projects will need to be bundled to form larger single file archives. The most common tool do this with is tar. Tar can optionally compress the data also but can take much longer. We also recommend the use of archivetar when using the ARC HPC clusters and is also available as a container for Linux systems.

On the ARC HPC systems run archivetar will do the needed tar bundling and upload to Data Den. The following command will sort data into tar files of 100GB each and upload to the selected folder on Data Den using Globus. You can also use tar, zip or other commands manually.

cd <folder to archive>
module load archivetar
archivetar --prefix my-archive --dryrun
archivetar --prefix my-archive --destination-dir /<dataden-volume>/<folder>

The following tar command will bundle all file files in directory, store it in the file bundle.tar.bz2, and compress it with bzip2. It will also create a small text file bundle.index.txt that can be stored with it to quickly reference what files are in the bundle.

tar -cvjf bundle.tar.bz2 driectory | tee bundle.index.txt

To extract the bundle:

tar -xvjf bundle.tar.bz2

Optionally omit -j to save time compressing, and omit -v to not print the bundle progress as it runs.

Compressing an archive can be accelerated on multi-core systems using pigz and lbzip2. The following will work on all ARC systems:

tar --use-compress-program=lbzip2 -cvf bundle.tar.bz2 brockp | tee bundle.index.txt

To extract the bundle:

tar --use-compress-program=lbzip2 -xvf bundle.tar.bz2

Storage Resource Software

If you are unsure which of our storage services should be used to host your data, we have written some software that you can download and execute to analyze your files to understand how much of your data is stored in large files, how much of your data has been accessed recently, and the distribution of file sizes and access times. Use the Storage Guidance Tool to find file statistics and get a storage recommendation.

This software doesn’t examine the contents of any data files, it merely scans file attributes, it also does not store any file names after searching through the filesystem.

If you have any questions on this software, send an email to arc-support@umich.edu. If you are unsure about any of the recommendations the tool sends you, contact us at arc-support@umich.edu.

Policies

Small File Limitation and Optimal Size

Data Den’s underlying technology does not work well with small files. Because of this design, each 1TB of Data Den capacity provides only 10,000 files, and only migrates files 100 MByte or larger. The optimal file size range from 10 – 200 GBytes.

Maximum File Size

The maximum file size is 8 TByte, but files should optimally not be larger than 1 TByte. Larger archives can be split before uploading to Data Den with the split -b 200G filename command.

Sensitive Data — ePHI/HIPAA/ITAR/EAR/CUI

Data Den now supports sensitive data such as HIPAA, PHI, other data types. Refer to the Sensitive Data Guide for full details.

System Abuse

Abuse, generally by excessive recalls of data better suited for active storage, of Data Den intentionally or not may result in performance or access being limited to preserve performance and access for other users. In the event this happens, staff will be in contact with users to engineer more appropriate solutions.

Frequently Asked Questions

Q: How do I check Data Den space and file usage?
A: Contact arc-support@umich.edu

Q: I’m getting Disk Quota Exceeded when uploading multiple TB of data but I’m well below my quota why?
A: This means you overran the allocated cache space for your volume. This space size is estimated for how much new data would be ingested in 48 hours. Leave the transfer running and contact arc-support@umich.edu. We will increase the cache space for the ingest and Globus will automatically resume the transfer where it left off.

Q: How can I use Data Den in my data management plan?
A: Data Den can provide low cost storage and data sharing with Globus to allow access to select data on Data Den anywhere in the world on demand. Most can get Data Den for no cost through UMRCP.

Q: Why can’t Data Den be accessible directly from hosts or my personal machine?
A: Data Den can appear as a local disk, but is not currently allowed in most cases as it presents many opportunities to disrupt use of the services by other users. Anyone with more than 100TB of data and wishes to automate their data management may be interested in Data Den Active Archive and should contact arc-support@umich.edu.

Q: Can Data Den encryption be used with my restricted data?
A: Always refer to the Sensitive Data Guide as the definitive source for allowed data types. Data Den now supports sensitive data.

Q: Why can’t we use Data Den as general purpose storage?
A: Data Den’s design is inherently linear and cannot respond to user requests quickly, the way general purpose storage does.  Those with massive data volumes (100TB+) may find Active Archive a good solution to control storage costs and have a portion of data active and fast.

Q: I need to retrieve a lot (more than 50) of files from Data Den. Can you help me?
A: Yes, contact arc-support@umich.edu with the list of files you want to recall, and we will optimize the recall from tape, speeding access.

Q: I use Locker heavily and want to automate moving data as it ages to my Data Den space. Can you help me?
A: Yes. Data Den and Locker are inherently connected. Any Locker volume can potentially have a custom data migration policy automating data flow to Data Den via Active Archive. Contact arc-support@umich.edu to set up a custom migration policy.

Advanced Topics

System Configuration

Data Den consists of two IBM Spectrum Archive LTFSEE clusters. Each cluster is connected to a corresponding Locker cluster. The LTFSEE cluster consists of an IBM TS4500 tape library with 20PB of uncompressed capacity and 9 Linear Tape Open (LTO) 8 drives. Each drive provides optimally 360 MByte/s uncompressed performance. Each library can also add additional frames and drives for new tape technology, increasing the capacity and performance.

Each cluster has a copy of all data separated by miles protecting it against fire and other disasters. Each cluster has 2 data movers. The purpose of these data movers is to apply policies to specific exports in Locker. As data are written to these exports the data mover moves the largest and oldest files to the tape in the TS4500 library encrypting it in the process. The data mover leaves a 1 MByte stub preserving the file layouts and allowing the first 1 MByte of data to be read without involving the tape. When a file that has been migrated is read the data mover instructs the TS4500 to recall the data from the tape and place it back on the faster disk of Locker transparently to the user.

Thus Data Den is able to ingest new data quickly to disk, and over time send it to tape as it is ages and goes unused. If data is recalled it remains on disk until it again is migrated due to policy.

Locker is a cost-optimized, high-capacity, large-file storage service for research data. Locker provides high-performance for large files, and allows investigators across U-M Ann Arbor to connect their data to computing resources necessary for their research, including U-M’s HPC clusters.

Locker can only be used for research data. It is designed for data that does not require frequent access, for example, data that is at the end of its useful life-cycle. It is tuned for large files (1MB or greater), and large numbers of small files should not be stored on the service without being aggregated in some fashion (as through tar). Locker can be used in combination with the Globus data management sharing system for hosting and sharing data with external collaborators and institutes.

Locker rates

• Unreplicated: $2.28 per terabyte per month ($27.36 per year)
• Replicated: $4.53 per terabyte per month ($54.37 per year).
• A U-M shortcode is required to order this service.
• Not sure if this is the right storage service for you? Try the ITS Data Storage Finder to find storage solutions to meet your needs.

Get started

Visit our Ordering Storage page to see an overview of what is needed to set up a new Locker volume.Use your UMICH (Level-1) credentials and Duo two-factor authentication to login via the ITS Service Request (SRS) Portal or the ITS General Computing provision page.

SRS support

Information about the SRS and how to use it can be found throughout the SRS system.

Here are some additional resources:

• Using the Service Request System
• Access and Permissions
• Chartfield Change Request
• Format the Inventory and Location Report

Getting Started

Requesting or Modifying a Locker Storage Volume

• Common Internet File System (CIFS) Requests
• Network File System (NFS) Requests

Globus Server Endpoint

Locker can be made available on existing ARC Globus servers to provide high performance transfers, data sharing and access to Locker from off campus. To access Locker via Globus, request your Locker volume be added to Globus.

ARC Compute System Support

Locker can be accessed from any ARC compute service that supports the same data classifications as your export. To have your Locker export added to an ARC resource contact us with the export name and system name. Locker will be available on all login and data transfer nodes at a minimum.

Mounts will be located at
/nfs/locker/<export-name>/

Research groups may also request system group creation to control group access to Locker volumes.

Optional Features

Replication – (Recommended) Optional second copy of all data in a different geographic location.

Snapshots – (Highly Recommended) Tracking of how data in a volume changes over time allowing users to recover deleted, modified, or otherwise damaged data.

Access snapshots at:
<mount-location>/.snapshots/<date>

Using Locker

Mounting on Windows CIFS
Instructions provided when provisioned

Mounting on Linux NFS
Instructions provided when provisioned

Mounting on Apple OSX
Instructions provided when provisioned

Storage Resource Software

If you are unsure which of our storage services should be used to host your data, we have written some software that you can download and execute to analyze your files to understand how much of your data is stored in large files, how much of your data has been accessed recently, and the distribution of file sizes and access times. Use the Storage Guidance Tool to find file statistics and get a storage recommendation.

This software doesn’t examine the contents of any data files, it merely scans file attributes, it also does not store any file names after searching through the filesystem.

If you have any questions on this software, send us an email at arc-support@umich.edu with your inquiry. If you are unsure about any of the recommendations the tool sends you, contact us at arc-support@umich.edu with your inquiry.

Group Access Controls

Linux Set GID

Using Set GID (SGID) on a directory will force all files created in that directory to inherit the same group permissions as the parent directory even if the user creating them primary or effective group is different. The benefit of this combined with the creation of a group on shared systems is that all files will be created owned and accessible (by default) to members of that group

#list available group
groups
chgrp <groupname> folder
chmod g+s folder

Windows AD Groups

Contact arc-support@umich.edu

Policies

Small File Limitation

Locker’s target audience are those research projects with massive data volumes in large files. Because of this design each 1TByte of Locker capacity provides only 1 Million files, e.g., 10 TByte provides 10 Million files. This works out to 1Mbyte per file average size.

Sensitive Data (ePHI, HIPAA, ITAR, EAR, CUI)

Locker is not currently supported for PHI or other data types. It is scheduled to be reviewed for support at a later date.

System Abuse

Abuse of Locker intentionally or not may result in performance or access being limited to preserve performance and access for other users. In the event this happens staff will be in contact with the users to engineer solutions.

Frequently Asked Questions

Q: How do I Check Locker Space and File Usage?

A: Linux or OSX Terminal use:

    Space: df -h <mount-path>
    Files: df -h -i <mount-path>

Q: Can Locker be Mounted on All ARC-TS Cluster Compute Nodes?

A: Currently we do not allow Locker to be mounted by very large numbers of clients. This could change in the future so let us know if this would help you. Currently we recommend using Globus to stage data between cluster scratch and Locker between runs. Globus provides a CLI so you can script.

Q: Can I Simultaneously Access Locker from Linux and Windows?

A: Currently Locker supports NFS (Linux) OR CIFS (Windows), Apple OSX supports both. This is known as Multi-Protocol or simultaneous NFS and CIFS access. Because Linux and Windows have different permissions schemes this is complex to manage. We don’t currently support his on Locker but do support it on Turbo. To work around this we recommend using Globus to ease data movement between Locker and systems that cannot mount it natively.

Q: Why can’t we use Locker as general purpose storage?

A: To maintain performance, encryption, professional support, and a low cost, Locker’s design does not make it well suited for general purpose primary storage. For this see the Turbo and MiStorage services.

Q: I deleted data but Locker still reports full?

A: Likely your export has snapshots enabled. Snapshots store changes to Locker exports over time thus deleted data is just ‘moved’ to a snapshot. Eventually snapshots will age out and free space on their own. Snapshot consumption does count against volume space used. To delete or disable snapshots to free space early contact support.

Q: I have free space but Locker reports full?

A: Likely you are at your file quota and are running average file size smaller than 1MByte. This use case is outside the support of Locker’s design and the small files should move to another storage service.

Q: I don’t see my .snapshots folder?

A: Your volume might not have snapshots enabled. If it does it is a hidden file on Linux and OSX terminals use ls -a to view all files including hidden files. To show hidden files in OSX and Windows user interfaces varies by version and can be found in their documentation and online.

Q: My volume shows 2x the size I requested!

A: The system Locker is built on tracks all copies of data in its file system. If a volume requests replication (2 copies of all data) total space will represent the primary and replica copy in total. Thus 1TB of new data will consume 2TB of Locker space.

Advanced Topics

System Configuration

Locker consists of two DDN GS14KX-E Gridscaler clusters running IBM Spectrum Scale. Each cluster is located in different data centers with dedicated fiber for data replication between the two sites. Each GS14KX-E cluster can hold 1680 hard drives for capacity of 10PB usable using 8TByte drives. Each hard drive is 7200RPM self encrypting and can be added to the system online. If at capacity additional GS14KX-E can be clustered to add additional performance and capacity.

By not including dedicated metadata or flash/NVMe storage we are able to keep the cost of Locker lower than other solutions such at Turbo. Thus Locker will not perform well with small IO operations and is built for capacity. Thus why we offer both services. The GS14KX-E does have support for adding NVMe/Flash for meta-data and tiering at a later date should the price of such devices become more reasonable.

Locker is directly connected to the Data Den archive via dedicated data movers and to the ARC research network by two IBM Cluster Export Services (CES) nodes or Protocol Nodes. Each CES node is connected with 100Gbps network connections and work in an active-active high availability configuration. Outside the ARC network performance is limited to 40Gbps from the campus backbone.

Open Storage Research Infrastructure (OSiRIS) is a collaboration between U-M, Wayne State University, Michigan State University and Indiana University to build a distributed, multi-institutional storage infrastructure that will allow researchers at any of our three campuses to read, write, manage and share large amounts of data directly from their computing facility locations on each campus.

By providing a single data infrastructure that supports computational access on the data “in-place,” OSiRIS meets many of the data-intensive and collaboration challenges faced by our research communities and enable these communities to easily undertake research collaborations beyond the border of their own Universities.

OSiRIS will use commercial off-the-shelf hardware coupled with CEPH software to build a high performance software-defined storage system. The system is composed of a number of building-block components: storage head- nodes with SSDs + 60-disk SAS shelf, RHEV cluster (2-hosts), Globus Connect servers, a PERFSONAR network monitoring node and reliable, OpenFlow capable switches.

OSiRIS will deploy a software-defined storage (e.g., commodity hardware with storage logic abstracted into a software layer) service for our universities using the CEPH Storage Cluster as the primary means of organizing the storage hardware required.

OSiRIS is funded by a grant from the National Science Foundation; the Principal Investigator is Shawn McKee, Research Scientist in the Department of Physics and the Director of the Center for Network and Storage-Enabled Collaborative Computational Science (CNSECCS). CNSECCS and OSiRIS are operated under the auspices of the Michigan Institute for Computational Discovery and Engineering (MICDE).