Category

Feature

Flux transfer servers will be decommissioned on Jan. 9; use new endpoints

By | Feature, General Interest, News

To provide faster transfers of data from ARC services, ARC will be decommissioning the Flux-Xfer servers on January 9, 2023. You will need to update how you migrate your data. 

For everyone who uses ARC storage services, especially Data Den users: This message is VERY important! This change includes the use of scp from Flux-Xfer, as well as the Globus endpoint umich#flux. Any shared endpoints that you have created on umich#flux will be automatically migrated to a new Globus collection on January 9. Those who use Data Den should take special interest in Item 1 listed below. 

Action item – Use the new endpoints

  1. If you currently use globus and umich#flux to access your Locker or Data Den volume, you should use the Globus Collection ‘UMich ARC Locker Non-Sensitive Volume Collection’ for Locker, and the Globus Collection ‘UMich ARC Data Den Non-Sensitive Volume Collection’ for Data Den. 
  2. If you currently use globus and umich#flux to access your Turbo volumes, you should use the Globus Collection ‘UMich ARC Turbo Non-Sensitive Volume Collection’.
  3. If you currently use globus and umich#flux to access other storage volumes, you should use the Globus Collection ‘umich#greatlakes’.  
  4. If you currently use scp on flux-xfer to copy data to/from Turbo, you should use ‘globus-xfer1.arc-ts.umich.edu’.

User guide 

How can we help you?

For assistance or questions, please contact ARC at arc-support@umich.edu

2023 Winter Maintenance & Globus File Transfer upgrade 

By | Feature, General Interest, Great Lakes, HPC, News, Systems and Services

Winter maintenance is coming up! See the details below. Reach out to arc-support@umich.edu with questions or if you need help. 

These services will be unavailable: 

  • Great Lakes – We will be updating Great Lakes on a rolling basis throughout December and beginning of January, and if successful, there should be no downtime or impact, with the following exceptions: 
    • Single precision GPUs (SPGPU) will be down Jan. 4-5 for networking maintenance. Those nodes will return back to production when maintenance has been completed and the nodes have been reloaded.
    • Customers will be notified via email of any changes to Great Lakes maintenance that will require downtime.
    • If unsuccessful, the Great Lakes maintenance will begin on Jan. 4-5, starting at 8am.  In either case, we will email everyone with the updated maintenance status.
  • Globus on the storage transfer nodes: Jan. 17-18.

Maintenance notes:

  • No downtime for ARC storage systems maintenance (Turbo, Locker, and Data Den).
  • Open OnDemand (OOD) users will need to re-login. Any existing jobs will continue to run and can be reconnected in the OOD portal.
  • Login servers will be updated, and the maintenance should not have any effect on most users. Those who are affected will be contacted directly by ARC. 
  • Copy any data and files that may be needed during maintenance to your local drive using Globus File Transfer before maintenance begins. 
  • Slurm email will be improved, providing  more detailed information about completed jobs.

Countdown to maintenance 

For Great Lakes HPC jobs, use the command “maxwalltime” to discover the amount of time remaining until maintenance begins. 

Jobs that request more walltime than remains until maintenance will automatically be queued and start once maintenance is complete. If the plan for Great Lakes maintenance is successful, any queued jobs will be able to run as usual (except for the SPGPU nodes as discussed above). Customers will be notified via email if downtime is required for Great Lakes.

Status updates and additional information

How can we help you?

For assistance or questions, please contact ARC at arc-support@umich.edu.

XSEDE is now ACCESS 

By | Feature, HPC, News

Access: advancing innovation and decorative imageThe national HPC resource known as XSEDE has now fully transitioned to ACCESS, Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support. 

ARC staff members, Todd Raeker and Shelly Johnson, are here to help you understand what is happening and how you can take advantage of the national resources and allocations.  

ACCESS has streamlined the allocation application process. Many projects that formerly had to submit an annual allocation proposal under XSEDE may now fit within one of the new ACCESS opportunities for small or medium-sized projects. You can now make one request with a much shorter allocations proposal. You can also make these requests at any time, rather than wait for the quarterly deadlines. Allocations are awarded for the duration of a supporting grant or for 12 months, with possible extensions up to five years.

There are new tiers as well. With the new Discover tier, there are 1.5 million available credits, and is closer in line to annual allocation from XSEDE. Researchers can submit the one-page application anytime. Check the overview of the ACCESS opportunities to see where your project fits. If you find that your research activities are considered to be a large-scale project, you will need to submit a proposal for a Maximize ACCESS award. Requests for Maximize ACCESS are accepted every six months. More details are available on the Maximize ACCESS web page.

Are you a grad student researcher? You now qualify to apply as the principal investigator (PI) for an ACCESS allocation in the Explore ACCESS tier to obtain resources to help complete your dissertation. Graduate students will need to include a letter from their advisor to support the request, and advisors must be added as a co-PI.

If you are interested in assisting the national research and educational community by reviewing ACCESS allocation requests, you can sign up to join the ACCESS allocations review committee.

Need help? Contact arc-support@umich.edu, or visit the ACCESS web page on the ARC website. You can also keep up with ongoing updates on the ACCESS webpage, access-ci.org.

Attend a listening session with ARC Director Brock Palen

By | Feature, HPC, News

You’re invited! ARC Director Brock Palen would like to hear from researchers. 

Are ARC services meeting their needs? What is not working well for them? Is something technical impeding their ability to do their research? Do they like Turbo Research Storage, the HPC web interface Open OnDemand, or the no-cost allocations offered by the U-M Research Computing Package

This is an open, virtual, drop-in office hour. All are welcome. 

There are three sessions are available: 

Researchers can also email Brock Palen at brockp@umich.edu or reach out to ARC at arc-support@umich.edu.

Preserving Michigan’s musical history and culture

By | Feature, News, Research

From Kentucky bluegrass to Louisiana Zydeco to German hurdy-gurdy to East European Klezmer to Indian Manipuri dancing to Native American pow wows, and much more, these musical traditions from around the country and around the world have found their way to Michigan. Beginning in 2014, the Musical Heritage Project has been documenting Michigan’s folk music history.

Lester Monts

Lester Monts Lester Monts specializes in ethnomusicology and has been documenting Michigan’s folk cultural heritage since 2014. (Image courtesy Lester Monts)

The project is led by ethnomusicologist Dr. Lester P. Monts, Arthur F. Thurnau Professor Emeritus of Music, who began his musical journey as an orchestral trumpet player. He earned bachelor’s and master’s degrees in trumpet performance and teaching trumpet at the college level before completing the doctoral degree in ethnomusicology and embarking on a research career. In the mid-1970s, Monts began to focus his research on music and culture in Liberia and Sierra Leone in West Africa. The fourteen-year Liberian civil war thwarted his fieldwork in that region.

Noting that there has been no systematic effort to collect and archive Michigan’s rich folk music heritage, the Michigan Musical Heritage Project was launched. Monts has embraced the study of music from the cultural and social aspects of the people who make it. He notes that “music brings people together; it has the power to create community, and we witnessed this occurring throughout our many journeys around the state.”

Using his charm, passion, likeability, and keen musical knowledge to cultivate trust with his interviewees, Monts captured more than 400 hours of audio and video data over the years, amassing a total of 80 terabytes of data. He believes this to be the most extensive collection of Michigan folk music in the state and that U-M is the right place to house this collection.

The Michigan Musical Heritage Project crew.

The Michigan Musical Heritage Project crew wraps up at the end of recording session. (Image courtesy Lester Monts)

With a videography crew consisting primarily of former U-M students, Monts traveled all around the state to record performances at folk music festivals and cultural gatherings, such as the Celtic Festival (Saline), Irish Folk Music Festival (Muskegon) Hispanic Heritage Festival (Hart), Hiawatha Traditional Music Festival (Marquette), Port Sanilac Blues Festival (Port Sanilac), Africa World Festival (Detroit), Aura Jamboree (Aura), Oldtime Fiddlers Convention and Traditional Music Festival (Hillsdale).

He says, “The creative talents of the state’s outstanding musicians must be preserved, not only for my research but for that of others as well. If properly preserved, I’m confident that in the future, the ethnomusicology program and the American Cultures department will find these data provide important insights into Michigan’s diverse musical heritage.”

How technology supports this project 

Monts’ crew includes a strong partnership with Tom Bray, converging technologies consultant and adjunct assistant professor of Art and Design, Penny W. Stamps School of Art and Design. Bray has been instrumental in pairing the right technology for the long-term preservation of this collection, which includes converting older footage to digital media. 

Tom Bray

Tom Bray (image courtesy LSA)

Bray has collaborated with Monts to convert older technologies, such as VHS, 8mm, and high-8 video, to digital files. The files are both compressed and uncompressed and are very large and of high resolution.

All of this wonderful and important audio and video footage needs to be preserved somewhere. But where do you turn when you have 80 terabytes of data? Monts said, “I’ve been desperately searching for a way to archive the video data collected under the auspices of the Michigan Musical Heritage Project.” 

Enter the U-M Research Computing Package (UMRCP) and the team from Advanced Research Computing (ARC), a division of Information and Technology Services. The UMRCP offers researchers across all campuses several resources at no additional cost to researchers, including 100 terabytes of long-term storage.

Bray said, “I had to read the UMRCP email announcement twice because I couldn’t believe my eyes. I was so excited that ITS and the university are supporting researchers in this way. We jumped on this opportunity right away.” 

ARC Director Brock Palen is excited about this work, too. “This is super interesting, and not like the usual types of research ARC normally sees, like climate and genomics. We’re happy to help Dr. Monts and Mr. Bray, and anyone who needs it, anytime. The archive is intentionally built for holding large-volume, raw data such as 4k video, and we are proud to be their go-to for this important cultural preservation project.” 

Old media in Dr. Monts' office

Hours and hours of media is being converted to a digital format. (Photo by Stephanie Dascola)

ARC replicates and encrypts in two secure locations that are miles apart, so those who use ARC services will not have to worry about crashes that they might experience if they are using their own equipment. The UMRCP also includes technical expertise by talented ARC staff to further remove barriers so researchers can do what they do best.

Monts and Bray also leverage the university’s network and WiFi services to transfer the files from their studio in the Duderstadt Center to storage. The network is designed to minimize bottlenecks so that data transfers quickly and efficiently. 

Dr. Monts said, “Although the pandemic temporarily disrupted my plans to complete the video documentary, I take solace in knowing that the many hours of data we collected is in a much safer environment than we had. The UMRCP storage resource is truly a boon!”

Related links

An old reel-to-reel tape player.

A reel-to-reel tape player. (Photo by Stephanie Dascola)

Lester Monts plays footage from a special women's only dance in Iberia.

Dr. Monts shows footage from a special women-only dance in Iberia. He earned permission to record this rarely-documented group of women. (Photo by Stephanie Dascola)

U-M partners with Cavium on Big Data computing platform

By | Feature, General Interest, Happenings, HPC, News

A new partnership between the University of Michigan and Cavium Inc., a San Jose-based provider of semiconductor products, will create a powerful new Big Data computing cluster available to all U-M researchers.

The $3.5 million ThunderX computing cluster will enable U-M researchers to, for example, process massive amounts of data generated by remote sensors in distributed manufacturing environments, or by test fleets of automated and connected vehicles.

The cluster will run the Hortonworks Data Platform providing Spark, Hadoop MapReduce and other tools for large-scale data processing.

“U-M scientists are conducting groundbreaking research in Big Data already, in areas like connected and automated transportation, learning analytics, precision medicine and social science. This partnership with Cavium will accelerate the pace of data-driven research and opening up new avenues of inquiry,” said Eric Michielssen, U-M associate vice president for advanced research computing and the Louise Ganiard Johnson Professor of Engineering in the Department of Electrical Engineering and Computer Science.

“I know from experience that U-M researchers are capable of amazing discoveries. Cavium is honored to help break new ground in Big Data research at one of the top universities in the world,” said Cavium founder and CEO Syed Ali, who received a master of science in electrical engineering from U-M in 1981.

Cavium Inc. is a leading provider of semiconductor products that enable secure and intelligent processing for enterprise, data center, wired and wireless networking. The new U-M system will use dual socket servers powered by Cavium’s ThunderX ARMv8-A workload optimized processors.

The ThunderX product family is Cavium’s 64-bit ARMv8-A server processor for next generation Data Center and Cloud applications, and features high performance custom cores, single and dual socket configurations, high memory bandwidth and large memory capacity.

Alec Gallimore, the Robert J. Vlasic Dean of Engineering at U-M, said the Cavium partnership represents a milestone in the development of the College of Engineering and the university.

“It is clear that the ability to rapidly gain insights into vast amounts of data is key to the next wave of engineering and science breakthroughs. Without a doubt, the Cavium platform will allow our faculty and researchers to harness the power of Big Data, both in the classroom and in their research,” said Gallimore, who is also the Richard F. and Eleanor A. Towner Professor, an Arthur F. Thurnau Professor, and a professor both of aerospace engineering and of applied physics.

Along with applications in fields like manufacturing and transportation, the platform will enable researchers in the social, health and information sciences to more easily mine large, structured and unstructured datasets. This will eventually allow, for example, researchers to discover correlations between health outcomes and disease outbreaks with information derived from socioeconomic, geospatial and environmental data streams.

U-M and Cavium chose to run the cluster on Hortonworks Data Platform, which is based on open source Apache Hadoop. The ThunderX cluster will deliver high performance computer services for the Hadoop analytics and, ultimately, a total of three petabytes of storage space.

“Hortonworks is excited to be a part of forward-leading research at the University of Michigan exploring low-powered, high-performance computing,” said Nadeem Asghar, vice president and global head of technical alliances at Hortonworks. “We see this as a great opportunity to further expand the platform and segment enablement for Hortonworks and the ARM community.”

Building a Community of Social Scientists with Big Data Skills: The ICOS Big Data Summer Camp

By | Educational, Feature, General Interest, News

As the use of data science techniques continues to grow across disciplines, a group of University of Michigan researchers are working to build a community of social scientists with skills in Big Data through a week-long summer camp for faculty and graduate students.

Having recently completed its fourth annual session, the Big Data Summer Camp held by the Interdisciplinary Committee for Organizational Studies (ICOS) trains approximately 50 people each spring in skills and methods such as Python, SQL, and social media APIs. The camp splits up into several groups to try to answer a research question using these newly acquired skills.

Working with researchers from other fields is a key component of the camp, and of creating a Big Data social science community, said co-coordinator Todd Schifeling, a Research Fellow at the Erb Institute in the School of Natural Resources and Environment.

“Students meet from across social science disciplines who wouldn’t meet otherwise,” said Schifeling. “And every year we bring back more and more past campers to present on what they’ve been doing.”

Schifeling himself participated in the camp as a student before taking on the role of coordinator this year.

Teddy DeWitt, the other co-coordinator of the camp and a doctoral student at the Ross School of Business, added the camp presents the curriculum in a unique way relative to the rest of campus.

“This set of material does not seem to be available in other parts of the university, at least … with an applied perspective in mind,” he said. “So we’re glad we have this set of resources that is both accessible and well-received by students.”

Participants range in skill from beginning to advanced, but even a relatively advanced student like Jeff Lockhart, a doctoral student in sociology and population studies who describes himself as “super-committed to computational social science,” said that it’s hard to find classes in computational methods in social science departments.

“[The ICOS camp] doesn’t expect a lot of prior knowledge, which I think is critical,” Lockhart said.

Lockhart, DeWitt, and Dylan Nelson, also a sociology doctoral student, are working on setting up a series of workshops in Computational Social Science for fall 2016 (contact Lockhart at jwlock@umich.edu for more information). Lockhart said it’s critical that social scientists learn Big Data skills.

“If we don’t have skills like this, there’s no way for us to enter into these fields of research that are going to be more and more important,” he said.

“A lot of the skills we’ve learned are sort of the on-ramp for doing data science,” DeWitt added.

The camp is co-sponsored by Advanced Research Computing (ARC).

New on-campus data-science and computational research services available

By | Feature, General Interest, News | No Comments

Researchers across campus now have access to several new services to help them navigate the new tools and methodologies emerging for data-intensive and computational research.

As part of the U-M Data Science Initiative announced in fall 2015, Consulting for Statistics, Computing and Analytics Research (CSCAR) is offering new and expanded services, including guidance on:

  • Research methodology for data science.
  • Large scale data processing using high performance computing systems.
  • Optimization of code and use of Flux and other advanced computing systems.
  • Advanced data management.
  • Geospatial data analyses.
  • Exploratory analysis and data visualization.
  • Obtaining licensed data from commercial sources.
  • Scraping, aggregating and integrating data from public sources.
  • Analysis of restricted data.

“With Big Data and computational simulations playing an ever-larger role in research in a variety of fields, it’s increasingly important to provide researchers with a comprehensive ecosystem of support and services that address those methodologies,” said CSCAR Director Kerby Shedden.

As part of this significant expansion of its scope, the campuswide statistical consulting service CSCAR has been renamed Consulting for Statistics, Computing and Analytics Research. It was formerly known as the Center for Statistical Consultation and Research.

For more information, see the University Record article.