Map U-M Data Science Space
To be announced
Advanced Research Computing (ARC), a division of ITS, is pleased to offer a pilot called Scientific Computing and Research Consulting Services to help researchers implement data analytics and workflows within their research projects. This includes navigating technical resources like high-performance computing and storage.
The ARC Scientific Computing and Research Consulting Services team will be your guide to navigating the complex technical world: from implementing intense data projects, to teaching you how the technical systems work, to assist in identifying proper tools, to guiding you on how to hire a programmer.
The ARC Scientific Computing and Research Consulting Services team works in partnership with the Consulting for Statistics, Computing, and Analytics Research team (CSCAR), Biomedical Research Core Facilities, and others. ARC may refer or engage complimentary groups as required by the project.
Send an email to arc-consulting@umich.edu with the following information:
If you have any questions or wish to setup a consult, please contact us at arc-consulting@umich.edu. Be sure to include as much information as possible from the “Get started” section noted above.
If you have more general questions about ARC services or software please contact us at arc-support@umich.edu
Virtual office hours are also available on Tuesdays, Wednesdays, and Thursdays. Get help with machine learning, algorithms, modeling, coding, computing on a cluster, survey sampling, using records across multiple surveys, and more. Anyone doing any type of research at any skill level is welcome!
We are available to assist researchers along the entire lifecycle of the data workflow, from the conceptual stage to ingest, preprocessing, cleansing, and storage solutions. We can advise in the following areas:
The data science consulting team can assist with data analytics to support research:
Machine learning is an application of artificial intelligence (AI) that focuses on the development of computer programs to learn information from data.
We are available to consult on the following. This includes a general overview of concepts, discussion into what tools and architectures best fit your needs, or technical support on implementation.
| Language | Tools/Architectures | Models |
|---|---|---|
| Python | Python data tools (scikit, numpy, etc) | Neural networks |
| C++ | TensorFlow | Decision trees |
| Java | Jupyter notebooks | Support vector machines |
| Matlab |
We also provide consulting on programming in a variety of programming languages (including but not limited to: C++, Java, and Python) to support your data science needs. We can assist in algorithm design and implementation, as well as optimizing and parallelizing code to efficiently utilize high performance computing (HPC) resources where possible/necessary. We can help identify available commercial and open-source software packages to simplify your data analysis.
If you have any questions or wish to setup a consult please contact us at arcts-consulting@umich.edu
The Cavium/ThunderX Hadoop Cluster has been shut down.
The Cavium/ThunderX Hadoop Cluster was a next-generation Hadoop cluster available to U-M researchers. It was an on-campus resource that holds 3PB of storage for researchers to approach and analyze data science problems. The cluster consisted of 40 servers each containing 96 ARMv8 cores and 512GB of RAM per server. It was made possible through a partnership with Marvell.
What should I use instead of the Cavium/ThunderX Hadoop Cluster?
Cavium/ThunderX users are recommended to transition to the Great Lakes High Performance Computing Cluster. Visit the Great Lakes page for details and account creation instructions.
If you are not familiar with the U-M Research Computing Package, please check to see if your work is eligible for allocations of compute resources that are provided by ITS. On the UMRCP form, be sure to select the services you want (HPC, storage, sensitive, and non-sensitive), and add at least one user (lab manager, etc.).
If you used Spark or PySpark on Cavium/ThunderX, try the web-based Jupyter Notebook application on Great Lakes that provides Spark integration. Spark is available under the menu “Interactive Apps” > “Jupyter + Spark Basic.”
If you prefer running Spark batch jobs from the terminal rather than from a Jupyter Notebook, ARC has the Spark on HPC project that will demonstrate how to run a Spark cluster from a Slurm job.
The Twitter datasets have been made available on Great Lakes. Additional details about Twitter datasets are in the getting started guide.
For assistance or questions, please contact ARC at arc-support@umich.edu.
New requests are not being accepted at this time because the Cavium/ThunderX Hadoop Cluster was shut down on August 1, 2022. Contact ARC if you need assistance, arc-support@umich.edu.
Yottabyte Research Cloud (YBRC) is no longer available. The YBRC service has been migrated to the Secure Enclave Service.
YBRC was a partnership between Information and Technology Services (ITS) and Yottabyte/Verge.io from 2016 to 2022 that provided U-M researchers with high-performance, secure, and flexible computing environments enabling the analysis of sensitive data sets restricted by federal privacy laws, proprietary access agreements, or confidentiality requirements.
How can we help you?
Contact ARC at arc-support@umich.edu for assistance.
ConFlux is a cluster that seamlessly combines the computing power of HPC with the analytical power of data science. The next generation of computational physics requires HPC applications (running on external clusters) to interconnect with large data sets at run time. ConFlux provides low latency communications for in- and out- of-core data, cross-platform storage, as well as high throughput interconnects and massive memory allocations. The file-system and scheduler natively handle extreme-scale machine learning and traditional HPC modules in a tightly integrated work flow—rather than in segregated operations—leading to significantly lower latencies, fewer algorithmic barriers and less data movement.
The ConFlux cluster is built with ~58 IBM Power8 CPU two-socket “Firestone” S822LC compute nodes providing 20 cores in each. Seventeen Power8 CPU two-socket “Garrison” S822LC compute nodes provide an additional 20 cores and host four NVIDIA Pascal GPUs connected via NVIDIA’s NVLink technology to the Power8 system bus. Each GPU based node has a local high-speed NVMe flash memory for random access.
All compute and storage is connected via a 100Gb/s InfiniBand fabric. The IBM and NVLink connectivity, combined with IBM CAPI Technology provide an unprecedented data transfer throughput required for the data-driven computational physics researchers will be conducting.
ConFlux is funded by a National Science Foundation grant; the Principal Investigator is Karthik Duraisamy, Assistant Professor of Aerospace Engineering and Director of the Center for Data-Driven Computational Physics (CDDCP). ConFlux and the CDDCP are under the auspices of the Michigan Institute for Computational Discovery and Engineering.
ARC is a division of ITS
