 that may allow vehicles to see through murky, low-visibility waters.){kind=link}
For remotely operated underwater vehicles, gloomy and rugged waters are often unthinkable. When vehicles sink to the seabed or dig through a sand pit, they can kick up clouds of sediment that obstruct the view of on-board cameras. Often the only thing that can be done is to wait for the sea dust to settle before the vehicle can safely continue its journey.
But a modern underwater mapping technique developed by engineers at MIT and the Woods Hole Oceanographic Institute (WHOI) could allow vehicles to see through murky, low-visibility waters.
This method combines visual images from optical cameras with acoustic data from sonar sensors. This combination allows the vehicle to quickly map the general shape of its surroundings using sonar, even in waters with indigent visibility. The vehicle can move toward specific shapes in a sonar-mapped environment, getting close enough for optical cameras to recognize specific objects in detail.
The technique is similar to combining a dolphin’s echolocation with a sea turtle’s close-range vision, allowing it to see and navigate through murky water in real time.
The researchers tested this method in tank experiments where they could control how apparent the water was. Even in the cloudiest conditions, the system was able to look through the sediment to map the tank surroundings and visualize centimeter-scale details of objects in the tank.
The team continues to refine the technique, which they call Sonar-MASt3R. They predict that the mapping method could safely guide underwater vehicles through turbid environments for a range of applications, including scientific research, construction and maintenance of underwater facilities, and deep-sea mining.
“We hope this work will enable us to perform more operations in difficult, low-visibility conditions and help provide greater coverage in areas where it is currently difficult to operate,” says Amy Phung, a graduate student in MIT’s School of Aeronautics and Astronautics who led the work.
Phung introduced article detailing Sonar-MASt3R this week at the IEEE International Conference on Robotics and Automation (ICRA). The paper was co-authored by Richard Camilli, senior scientist in applied ocean physics and engineering at WHOI.
The best of both
To see underwater, scientists generally apply an either-or approach, using optical cameras or sonar sensors to guide them. Optical cameras can provide detailed images of the scene, but only in waters that are relatively clear and well-lit. In contrast, sonar sensors work equally well in clear and turbid water; by emitting acoustic waves and measuring the time and angle of their return, sonar sensors can determine the exact shape, distance and depth of objects in the environment, although the sonar map does not contain any visual details.
To make the most of both modes, scientists decided to combine them in a modern approach known as “optical-acoustic fusion.” In several previous works, research groups have combined sonar and optical data in mapping techniques that are mainly aimed at object recognition and work environment reconstruction. Most techniques require time to synchronize and process data, so they do not work in real time, and only a few can map the environment in 3D. None of them have been applied to high-resolution underwater mapping in turbid and turbid conditions.
Phung, a student in the MIT-WHOI joint program, and Camilli, her advisor, intended to develop an optical-acoustic fusion technique that could generate detailed 3D maps of underwater environments in real time and in low visibility conditions. The team was motivated in part by the challenges of safely recovering unexploded underwater mines.
“Areas where it is unsafe for ships to be present may contain old explosives, and robotics is the best way to dispose of them safely,” says Camilli. “But many of these explosives are deployed in surf zones, where visibility increases the challenge of doing so safely. This is one of many applications to which our technique can be applied.”
Shadowy, mapable
The modern Sonar-MASt3R method is based on the existing MASt3R technique developed by scientists in France. MASt3R is an image matching algorithm that is trained to take visual images of the same scene and quickly estimate the relative depth of each pixel in the scene. In this way, MASt3R can generate a 3D map of the environment in real time based on 2D images from the camera.
“The disadvantage is the lack of sense of scale,” Phung says. “It will say, ‘this pixel is five units closer than this pixel,’ but it won’t be able to tell if it’s 5 meters or 5 feet.”
Fortunately, sonar provides absolute scale measurements. The time of sonar reflections can be directly translated into the specific depth and distance of the objects from which the signals were reflected, as well as their shape and contour.
In their modern work, Phung and Camilli used sonar data to correct the scaling of MASt3R and generate precise 3D maps of underwater environments. Even in murky water, a sonar-corrected map would enable the vehicle to know the exact location of objects, and therefore the distance to which it should safely approach, for more precise inspection, which the vehicle could then perform using conventional optical cameras.
The team tested Sonar-MASt3R in experiments with a tank filled with water, sediment and various objects such as a miniature boulder, a coffee mug and a packing crate. They also placed a robotic arm inside the tank on which they mounted an underwater camera and a sonar sensor.
For each experiment, a sweep trajectory was first performed in which the robotic arm slowly moved from one side of the tank to the other to capture sonar and visual data. During the first sweep, Sonar-MASt3R quickly creates a rugged sonar-based map of the shapes and contours of the tank and its features. The rugged map is then used to capture close-up camera images of the features, which is used to improve the resolution of the map. The “keyframe” method allows you to quickly compare each modern image frame with the last keyframe. If a frame contains modern information that is not contained in the last keyframe, the image is added to the map as a modern keyframe. If it is similar, it is immediately rejected. In this way, the approach can quickly populate the map with relevant visual details in real time.
The researchers tested their modern approach underwater, testing eight different levels of turbidity, which they created by mixing sediment in a tank. Compared to other optical-acoustic fusion approaches, Sonar-MASt3R generated more exact 3D maps and resolved finer details at the centimeter scale and in cloudier conditions. In the cloudiest conditions, which the robot arm’s cameras were unable to see, its sonar sensors were able to generate a rugged map of hidden objects in the tank. This initial map allowed the arm to navigate safely through the shadowy and approach specific objects, which its underwater camera could then visualize in greater detail.
“An analogy would be walking into a china shop in the dark and trying to find a specific coffee mug without knocking anything over,” Camilli suggests. “That would allow you to do it.”
The team plans to test this approach in natural underwater conditions, where they suspect the mapping task should be simpler.
“It’s like an echo chamber in the tank,” Camilli says. “It’s like trying to do it in a mirror at an amusement park, where all the distortions and echoes and ghost images come in, which really complicates the processing. If you put it in the real world, it should be easier.”
Then, they say, Sonar-MASt3R can aid scientists explore safely in gloomy, murky and murky underwater areas.
“The real value of this effort is that we can use this technology in mission scenarios that are currently not feasible,” Phung says. “There are many missions that can’t be accomplished because we don’t have the ability to observe or perceive.”
This research was supported in part by NASA and the National Science Foundation.