Adaptive Resolution Imaging Sonar: Initial Exploration at the Shiawassee National Wildlife Refuge

One great example of the many services offered by the Scientific Computing and Research Consulting Services team at ARC is the support that we provided for the project, Adaptive Resolution Imaging Sonar: Initial Exploration at the Shiawassee National Wildlife Refuge, at the School for Environment and Sustainability (SEAS). ARC supported the SEAS team in various stages throughout the lifecycle of this project, such as grant proposal writing, training, and developing custom solutions to enable the team to conduct their research.

During the grant writing stage, the ARC consulting team worked closely with the SEAS team to assist in writing a successful startup allocation grant to utilize computational resources in the Jetstream cloud environment which is provided by the Extreme Science and Engineering Discovery Environment (XSEDE). We consulted the SEAS team on how to write a successful proposal, and helped the team understand technical terminology. We also calculated the computational resource requirements in order to make the proposal as strong and accurate as possible by requesting the appropriate computational resources.

In addition to the consulting support, the ARC consulting team created a virtual machine image that contained essential software, including our custom developed software, to enable their research. This greatly simplified the workflow for the SEAS team by providing a common environment in which all students could perform their work. We preconfigured this environment to require minimal setup so that the students could focus on performing their work as quickly and efficiently as possible. This significantly improved the ease of use by reducing the required technical experience for working with the system.

Along with the virtual machine image, we developed user documentation and provided hands-on training sessions on how to use Jetstream as well as our custom software within the Jetstream computing environment. Although Jetstream is a very powerful tool, performing complex workflows on large amounts of data can be a daunting task for users without a strong technical background. Therefore, the training and documentation that we provided were invaluable in equipping the SEAS team with the knowledge and skills necessary for their workflow. This training included topics such as connecting to Jetstream, working with the command line, transferring data to and from remote systems, and running our custom software to process the team’s specialized data.

The ARC consulting team also developed a custom machine learning solution to identify fish in the sonar camera feeds. Building off a ResNet-50 convolutional neural network architecture, we applied transfer learning to train the network to identify fish in sonar images. First, we used Python’s LabelImg software to label approximately 4000 images for training data. We used these labeled images as input to our ResNet-50 architecture and produced a machine learning model capable of identifying fish in images. We then applied post-processing code to track fish across multiple images and produce reliable results of tracking fish identifications through the videos. We also wrote code to determine the directionality of the fish by tracing the fish movement tracks across the frame and identifying if the tracks were moving left or right. We tested the model using validation data and determined an accuracy of 68%. 

Once we trained and validated the machine learning model, we created an interface for the team to use the model and record results. We first created a graphical user interface to allow the user to upload a video in ARIS format, convert to mp4 format, apply the machine learning model, and save the results in csv and video format. We also produced two command line tools, one to convert ARIS videos to mp4 videos, and another to apply the machine learning model to the produced mp4 videos and output the results in video and csv format. 

Future work with this project includes ongoing work with the team to improve recognition accuracy. We also plan to introduce new functionality to recognize the size of the fish, as well as improve the directionality algorithm.