Great Lakes

2024 ARC Summer Maintenance happening in June

By | News

Summer maintenance will be happening from June 3-5. Updates will be made to software, hardware, and operating systems to improve the performance and stability of services. ARC works to complete these tasks quickly to minimize the impact of the maintenance on research.

The dates listed below are the days the work will be occurring.

HPC clusters and storage systems (/home and /scratch) will be unavailable:

  • June 3-4: for Great Lakes
  • June 3-5: for Armis2 and Lighthouse

No downtime for ARC storage systems maintenance:

  • Turbo, Locker, and Data Den will not see any interruption in service

Queued jobs and maintenance reminders

Jobs will remain queued, and will automatically begin after the maintenance is completed. The command “maxwalltime” will show the amount of time remaining until maintenance begins for each cluster, so you can size your jobs appropriately. 

Status updates

  • Status updates will be posted to the ITS website and the ARC Twitter feed
  • ARC will send an email when the maintenance has been completed.

How can we help you?

For assistance or questions, contact ARC at arc-support@umich.edu.

PFAS research in the Michigan mother-infant pairs study, supported by ITS, SPH, MM, AGC

By | News

Three mothers holding their infants. Everyone is sitting on a couch..PFAS (per- and polyfluoroalkyl substances) are a class of chemicals that have been around since the 1940s and became more broadly used in the post-war 1960s era. PFAS are in our homes, offices, water, and even our food and blood. PFAS break down slowly and are difficult to process, both in the environment and our bodies. 

Scientific studies have shown that exposure to some PFAS in the environment may be linked to harmful health effects in humans and animals. Because there are thousands of PFAS chemicals found in many different consumer, commercial, and industrial products, it is challenging to study and assess the human health and environmental risks. 

Fortunately, some of the most persistent PFAS are being phased out. The EPA has been working on drinking water protections, scientists are working on ways to break down and eliminate PFAS, and PFAS are being addressed at a national level

A team of University of Michigan researchers from the School of Public Health DoGoodS-Pi Environmental Epigenetics Lab and Michigan Medicine are working to understand how behaviors and environments during pregnancy can cause changes to the way genes work in offspring. This emerging field is known as toxicoepigenetics. 

Jackie Goodrich, Ph.D., research associate professor at the U-M School of Public Health, led the team. “PFAS may impact the development of something we all have called the epigenome. The epigenome is a set of modifications on top of our DNA that controls normal development and function. Environmental exposures like PFAS can alter how the epigenome forms, and this impacts development and health. Our study expands on current knowledge about PFAS and the epigenome by focusing on a type of epigenetic mark that is not usually measured.”

Vasantha Padmanabhan, Ph.D., M.S., professor emerita (in service), Department of Pediatrics, Michigan Medicine, built the Michigan Mother-Infant Pairs study over the past decade with an emphasis on identifying harmful exposures during pregnancy that impact women and their newborns. “I am so grateful to those who engaged in this study. PFAS are complex, and mothers’ and infants’ involvement helped us work toward a solution that impacts us all. I want to acknowledge the contributions of the U-M Department of Obstetrics and Gynecology, Michigan Institute for Clinical & Health Research (MICHR), and the Von Voigtlander Women’s Hospital that made this study possible.” 

Rebekah Petroff, Ph.D., a research fellow with Environmental Health Sciences, led the computation portion of the research. She said that using Turbo for storing the raw data and Great Lakes for high-performance computing (HPC) enabled a much faster analysis that was needed for the study with so much data to analyze. 

Turbo and Great Lakes are services provided by Advanced Research Computing, a division of Information and Technology Services (ITS). ARC facilitates powerful approaches to complex research challenges in fields ranging from physics to linguistics, and from engineering to medicine.

Petroff said, “This analysis would have taken over a month straight of computing time on a regular desktop computer. The first job we submitted to Great Lakes ran so fast—I had results the next morning! Great Lakes made this research possible, and I believe that our study results can be broadly impactful to public health and toxicoepigenetics going forward.”

Support for using this complex technology also came from Dan Barker, a UNIX systems admin with the U-M School of Public Health Biostatistics Department. Barker assisted with the code needed to use Great Lakes. “We started with a test run of a few hundred pairs of genomes. Once we were successful with that, we ran the entire nearly 750,000 epigenetic marks across 141 people and seven different PFAS.”

Barker also helped design and submit array jobs which are a series of identical, or near identical, tasks that are run multiple times. This is a common technique used by researchers when leveraging HPC. Array jobs allow for essential analytical comparisons among the test results. Petroff said, “In our study, we used an array job to split up our computations so that they ran much more efficiently!”

The U-M Advanced Genomics Core (AGC) performed the epigenetic assays, a kind of laboratory technique which measures marks on your DNA, for this project. AGC is part of the campus-wide laboratories that develop and provide state-of-the-art scientific resources to enable biomedical research known as Biomedical Research Core Facilities (BRCF). Other BRCF cores also worked on this project, including the Epigenomics Core and the Bioinformatics Core.

Genotyping is similar to reading a few words scattered on a page. This process gives researchers small packets of data to compare. Genotyping looks for information at a specific place in the DNA where we know important data will be. This project used a type of genotyping called microarrays (also known as “arrays”) and help researchers understand how regulation of DNA—including methylation and hydroxymethylation measured in this study—are impacted by exposures like PFAS.  

Brock Palen, ARC director, said, “This research is of human interest and impacts all of us. I’m pleased that ARC assisted their research with staff expertise, equipment, and no-cost allocations from the U-M Research Computing Package.”

Petroff said that follow up studies are needed to better understand if the results are universal or specific to this cohort of infants and parents. If the results hold steady, then a significant discovery has been made that will lead to more comprehensive PFAS mitigation solutions. “Although steps are being taken to mitigate PFAS, exposure is still prevalent, and a deeper understanding of how it impacts humans is needed,” said Dana Dolinoy, Ph.D., chair, NSF International Department Chair of Environmental Health Sciences and epigenetics expert.

Read the full article: Mediation effects of DNA methylation and hydroxymethylation on birth outcomes after prenatal per- and polyfluoroalkyl substances (PFAS) exposure in the Michigan mother–infant pairs cohort.

Funding was provided by grants from the National Institutes of Health, the U.S. Environmental Protection Agency, and the National Institute of Environmental Health Sciences Children’s Health Exposure Analysis Resource program.

Precision Health and ARC team up on a self-service tool for genetic research

By | Great Lakes, HPC, News

Encore is a self-serve genetic analysis tool that researchers can now run using a point-and-click interface without the need to directly manipulate the genetic data. Only a phenotype file is needed to build a GWAS model with SAIGE (genetics analysis software), launch and monitor job progress, and interactively explore results.

It is geared for a range of disciplines and specialties including biostatistics, epidemiology, neuroscience, gastroenterology, anesthesiology, clinical pharmacy, and bioinformatics.

The tool was developed at the U-M School of Public Health Center for Statistical Genetics and is managed by Precision Health and supported by ITS’s Advanced Research Computing (ARC).  

Brock Palen, ARC director, “When someone uses Encore they are actually running on Great Lakes, and we are happy to provide the computational performance behind Encore.”

Using Encore is easy. No coding, command-line/Linux knowledge is required to run GWAS in Encore. Researchers also do not need to have knowledge of batch job submission or scheduling, or have direct access to a high-performance computing cluster. Encore automatically prepares job submission scripts and submits the analysis to the Great Lakes High-Performance Computing Cluster. 

Great Lakes is the university’s flagship open-science high-performance computing cluster. It is much faster and more powerful than a laptop, and provides quicker answers and optimized support for simulation, genomics, machine learning, life science, and more. The platform provides a balanced combination of computing power, I/O performance, storage capability, and accelerators.

Visit the Encore wiki page to learn more

To get started, send an email to PHDataHelp@umich.edu

For questions about Great Lakes, contact arc-support@umich.edu

Understanding the strongest electromagnetic fields in the universe

By | Data, Great Lakes, HPC, Research, Uncategorized

Alec Thomas is part of the team from the U-M College of Engineering Gérard Mourou Center for Ultrafast Optical Science that is building the most powerful laser in the U.S.

Dubbed “ZEUS,” the laser will be 3-petawatts of power. That’s a ‘3’ with 15 zeros. All the power generated in the entire world is 10-terawatts, or 1000 times less than the ZEUS laser. 

The team’s goal is to use the laser to explore how matter behaves in the most extreme electric and magnetic fields in the universe, and also to generate new sources of radiation beams, which may lead to developments in medicine, materials science, and national security. 

A simulation of a plasma wake.

This simulation shows a plasma wake behind a laser pulse. The plasma behaves like water waves generated behind a boat. In this image, the “waves” are extremely hot plasma matter, and the “boat” is a short burst of powerful laser light. (Image courtesy of Daniel Seipt.)

“In the strong electric fields of a petawatt laser, matter becomes ripped apart into a `plasma,’ which is what the sun is made of. This work involves very complex and nonlinear physical interactions between matter particles and light. We create six-dimensional models of particles to simulate how they might behave in a plasma in the presence of these laser fields to learn how to harness it for new technologies. This requires a lot of compute power,” Thomas said. 

That compute power comes from the Great Lakes HPC cluster, the university’s fastest high-performance computing cluster. The team created equations to solve a field of motion for each six-dimensional particle. The equations run on Great Lakes and help Thomas and his team to learn how the particle might behave within a cell. Once the field of motion is understood, solutions can be developed. 

“On the computing side, this is a very complex physical interaction. Great Lakes is designed to handle this type of work,” said Brock Palen, director of Advanced Research Computing, a division of Information and Technology Services. 

Thomas has signed up for allocations on the Great Lakes HPC cluster and Data Den storage. “I just signed up for the no-cost allocations offered by the U-M Research Computing Package. I am planning to use those allocations to explore ideas and concepts in preparation for submitting grant proposals.”

Learn more and sign up for the no-cost U-M Research Computing Package (UMRCP).

Prof. Thomas’ work is funded by a grant from the National Science Foundation.

3-2-1…blast off! COE students use ARC-TS HPC clusters for rocket design

By | Educational, General Interest, Great Lakes, Happenings, HPC, News
MASA team photo

The MASA team has been working with the ARC-TS and the Great Lakes High-Performance Computing Clusters to rapidly iterate simulations. What previously took six hours on another cluster, takes 15 minutes on Great Lakes. (Image courtesy of MASA)

This article was written by Taylor Gribble, the ARC-TS summer 2020 intern. 

The Michigan Aeronautical Science Association (MASA) is a student-run engineering team at U-M that has been designing, building, and launching rockets since its inception in 2003. Since late 2017, MASA has focused on developing liquid-bipropellant rockets—which are rockets that react to a liquid fuel with a liquid oxidizer to produce thrust—in an effort to remain at the forefront of collegiate rocketry. The team is made up of roughly 70 active members including both undergraduate and graduate students who participate year-round.

Since 2018, MASA has been working on the Tangerine Space Machine (TSM) rocket which aims to be the first student-built liquid-bipropellant rocket to ever be launched to space. When completed, the rocket’s all-metal airframe will stand over 25 feet tall. The TSM will reach an altitude of 400,000 feet and will fly to space at over five times the speed of sound.

MASA is building this rocket as part of the Base 11 Space Challenge which was organized by the Base 11 Organization to encourage high school and college students to get involved in STEM fields. The competition has a prize of $1 million, to be awarded to the first team to successfully reach space. MASA is currently leading the competition, having won Phase 1 of the challenge in 2019 with the most promising preliminary rocket design.

Since the start of the TSM project, MASA has made great strides towards achieving its goals. The team has built and tested many parts of the complete system, including custom tanks, electronics, and ground support equipment. In 2020, the experimental rocket engine designed by MASA for the rocket broke the student thrust record when it was tested, validating the work that the team had put into the test.

The team’s rapid progress was made possible in-part by the extensive and lightning-quick simulations using the ARC-TS Great Lakes High-Performance Computing Cluster.

The student engineers are Edward Tang, Tommy Woodbury, and Theo Rulko, and they have been part of MASA for over two years.

Tang is MASA’s aerodynamics and recovery lead and a junior studying aerospace engineering with a minor in computer science. His team is working to develop advanced in-house flight simulation software to predict how the rocket will behave during its trip to space.

“Working on the Great Lakes HPC Cluster allows us to do simulations that we can’t do anywhere else. The simulations are complicated and can be difficult to run. We have to check it, and do it again; over and over and over,” said Tang. “The previous computer we used would take as long as six hours to render simulations. It took 15 minutes on Great Lakes.”

A computer simulation of Liquid Oxygen Dome Coupled Thermal-Structural

This image shows a Liquid Oxygen Dome Coupled Thermal-Structural simulation that was created on the ARC-TS Great Lakes HPC Cluster. (Image courtesy of MASA)

Rulko, the team’s president, is a junior studying aerospace engineering with a minor in materials science and engineering.

Just like Tang, Rulko has experience using the Great Lakes cluster. “Almost every MASA subteam has benefited from access to Great Lakes. For example, the Structures team has used it for Finite Element Analysis simulations of complicated assemblies to make them as lightweight and strong as possible, and the Propulsion team has used it for Computational Fluid Dynamics simulations to optimize the flow of propellants through the engine injector. These are both key parts of what it takes to design a rocket to go to space which we just wouldn’t be able to realistically do without access to the tools provided by ARC-TS.”

Rulko’s goals for the team include focusing on developing as much hardware/software as possible in-house so that members can control and understand the entire process. He believes MASA is about more than just building rockets; his goal for the team is to teach members about custom design and fabrication and to make sure that they learn the problem-solving skills they need to tackle real-world engineering challenges. “We want to achieve what no other student team has.”

MASA has recently faced unforeseen challenges due to the COVID-19 pandemic that threaten to hurt not only the team’s timeline but also to derail the team’s cohesiveness. “Beaucase of the pandemic, the team is dispersed literally all over the world. Working with ARC-TS has benefitted the entire team. The system has helped us streamline and optimize our workflow, and has made it easy to connect to Great Lakes, which allows us to rapidly develop and iterate our simulations while working remotely from anywhere,” said Tang. “The platform has been key to allowing us to continue to make progress during these difficult times.”

Tommy Woodbury is a senior studying aerospace engineering. Throughout his time on MASA he has been able to develop many skills. “MASA is what has made my time here at Michigan a really positive experience. Having a group of highly-motivated and supportive individuals has undoubtedly been one of the biggest factors in my success transferring to Michigan.

This image depicts the Liquid Rocket Engine Injector simulation.

This image depicts the Liquid Rocket Engine Injector simulation. (Image courtesy of MASA)

ARC-TS is a division of Information and Technology Services. Great Lakes is available without charge for student teams and organizations who need HPC resources. This program aims to enable students access to high-performance computing to enhance their team’s mission.