The electricity goes out on the island. To find a break in an underwater power cable, a ship pulls in the entire line or sends remotely operated vehicles (ROVs) to traverse it. But what if an autonomous underwater vehicle (AUV) could draw a line and pinpoint the location of a fault that a diver could repair?
This underwater connection of humans and robots is the focus of an MIT Lincoln Laboratory project funded by the internally managed Autonomous Systems Research and Development Portfolio and implemented by Advanced Subsea Systems and Technologies Group. The project aims to leverage the strengths of humans and robots to optimize maritime missions for the U.S. military, including critical infrastructure inspection and repair, search and rescue, port entry and mine countermeasures operations.
“Divers and AUVs generally do not cooperate underwater,” says principal investigator Madeline Miller. “Underwater missions that require humans tend to do them because they involve manipulation that a robot cannot perform, such as repairing infrastructure or deactivating a mine. Even ROVs present a challenge when performing very demanding manipulation tasks underwater because the manipulators themselves are not agile enough.”
In addition to excellent dexterity, humans also excel in recognizing objects under water. However, people working underwater cannot perform complicated calculations or move very quickly, especially if they are carrying ponderous equipment; Robots have advantages over humans in terms of processing power, rapid mobility and endurance. To combine these strengths, Miller and her team are developing hardware and algorithms for underwater navigation and perception – two key capabilities for effective human-robot interaction.
As Miller explains, divers can only be guided by a compass and a fin kick counter. With few landmarks and potentially murky conditions caused by a lack of airy at depth or the presence of biological matter in the water column, they can easily become disoriented and lost. For robots to aid divers navigate, they must perceive their surroundings. However, in the presence of darkness and turbidity, optical sensors (cameras) are unable to generate images, while acoustic sensors (sonars) generate images that are devoid of color and only show the shapes and shadows of objects in the scene. The historical lack of enormous, labeled sonar image datasets has made it hard to train underwater perception algorithms. Even if data were available, a lively ocean could unknown the true nature of objects, confusing the AI. For example, a downed plane broken into many pieces or a tire covered with mussel overgrowth may no longer resemble an airplane or a tire, respectively.
“Ultimately, we want to develop solutions for navigation and perception in expeditionary environments,” says Miller. “For the missions we’re thinking about, there’s limited or no ability to map out the area in advance. For a port entry mission, you might have a satellite map, but you don’t have an underwater map, for example.”
When it comes to navigation, Miller’s team continued the work started by MIT Marine Robotics Grouphosted by John Leonardto develop algorithms for combining diver and AUV. With their navigation algorithms, Leonard’s group ran simulations under optimal conditions and conducted field tests in peaceful waters, using human-paddle kayaks as proxies for both divers and AUVs. Miller’s team then integrated these algorithms into a mission-appropriate AUV and began testing them in more realistic ocean conditions, initially with a support boat acting as a stand-in for the diver, and then with actual divers.
“We quickly learned that the diver needed greater detection capabilities given the ocean currents,” Miller explains. “With the algorithms demonstrated by MIT, the vehicle only had to calculate the distance, or range, to the diver at regular intervals to solve the optimization problem of estimating the position of both the vehicle and the diver over time. However, when faced with the actual forces of the ocean pushing everything around, the optimization problem quickly falls apart.”
On the perception side, Miller’s team is developing an AI classifier that can process both optical and sonar data during the mission and obtain human input for any objects classified with uncertainty.
“The idea is that the classifier would give the diver some information – say, a frame surrounding the image – and indicate, ‘I think it’s a tire, but I’m not sure. What do you think?’ The diver can then respond, “Yes, you understood correctly, or no, look at the photo to correct your classification,” Miller says.
This feedback loop requires an underwater acoustic modem to support diver-AUV communications. State-of-the-art data rates in underwater acoustic communications would require tens of minutes to send an uncompressed image from the AUV to the diver. Therefore, one aspect the team is working on is how to compress the information to the minimum amount to be useful, within the constraints of low bandwidth and high latency underwater communications and the small size, weight and power of the commercial off-the-shelf (COTS) equipment used. For the prototype system, the team purchased primarily COTS sensors and built a sensor payload that could be easily integrated into an AUV routinely used by the U.S. Navy to ease the transition of the technology. In addition to sonar and optical sensors, the payload contains an acoustic modem to reach the diver and several data processing and computational arrays.
Miller’s team tested the sensor-equipped AUV and algorithms on the New England coast, including in the open ocean near Portsmouth, New Hampshire, in collaboration with the University of New Hampshire (UNH) Surveyor of the Persian Gulf AND Challenger from the Persian Gulf offshore research vessels as substitutes for divers, and on the Charles River near Boston with the MIT Sailing Pavilion boat as a substitute.
“The UNH boats are well equipped and have access to realistic ocean conditions. However, pretending to be a diver in a enormous boat is hard. With the skiff, we can move slower and achieve relative movement suited to how the diver and AUV will navigate together.”
Last summer, the team began testing the equipment with divers at Michigan Technological University Great Lakes Research Center. Although the divers did not have an interface to transmit information to the AUV, each diver swam carrying the team’s prototype tube-shaped tablet, called a “tube.” The pipe-let is equipped with a pressure and depth sensor, an inertial measurement module (to track relative motion) and a distance modem – all necessary elements of navigation algorithms to solve the optimization problem.
“The challenge during testing was coordinating the movement of the diver and the vehicle because they don’t work together yet,” Miller says. “Once the divers go underwater, there is no communication with the team on the surface. So you have to plan where to place the diver and the vehicle to avoid a collision.”
The team also worked on the problem of perception. The water clarity of the Great Lakes at this time of year made it possible to take underwater photos using an optical sensor. Caroline Keenan, a Lincoln Scholars graduate student working jointly in the laboratory’s Advanced Undersea Systems and Technology Group and Leonard’s research group at MIT, took the opportunity to advance her work on transferring knowledge from optical to sonar sensors. It examines whether optical classifiers can train sonar classifiers to recognize objects for which no sonar data exists. The motivation is to reduce the operator burden of labeling sonar data and training sonar classifiers.
As the internally funded research program comes to an end, Miller’s team is now seeking external sponsorship to refine and transfer the technology to military or commercial partners.
“The state-of-the-art world relies on undersea telecommunications and power cables, which are vulnerable to attack by jammers. The undersea space is becoming increasingly controversial as more countries develop and enhance the capabilities of autonomous maritime systems. Maintaining global economic security and U.S. strategic advantage in the undersea space will require harnessing and combining the best of artificial intelligence and human capabilities,” Miller says.