A up-to-date chip developed by MIT researchers could facilitate miniature, low-power UAVs avoid obstacles while navigating tight corners inside an industrial HVAC system to check for gas leaks.
The chip allows miniature autonomous robots and other battery-limited devices to create detailed 3D maps of their environments in real time, using about as much power as a single LED. The robot could apply such a map to plan a collision-free path to reach the target.
Typically, generating such exact maps requires energy-intensive systems and enormous amounts of memory to create and store three-dimensional representations of obstacles in the robot’s environment.
MIT researchers took a different approach, combining an extremely competent mapping algorithm with specialized hardware designed to accelerate its workload, which minimizes memory and power consumption.
This system-on-chip uses only about 6 milliwatts of power, which is a fraction of the power required by other systems.
This low power consumption will make the chip well-suited for lightweight augmented reality headsets that can be worn for extended periods of time, for applications such as educational medical simulations or detailed repair and assembly work.
“This paper presents a key example of how algorithm and hardware co-design can be used to truly increase energy efficiency. While a lot of work has been done on compact 3D maps, it stands out in that it also makes the process of generating these maps as efficient as possible. Our chip allows us to store very large maps in a very small space and do it in a very energy-efficient way,” says Vivienne Sze, professor in the Department of Electrical Engineering and Computer Science (EECS), member of the Research Laboratory of Electronics (RLE) and senior author: a paper on a chip.
She is joined on the paper by co-authors and MIT graduate students Zih-Sing Fu and Peter Zhi Xuan Li, as well as Sertac Karaman, professor of aeronautics and astronautics and director of LIDS. The work was recently presented at the IEEE Very Vast-Scale Integrated Circuits Symposium.
A more compact map
For a robot, generating a 3D map containing obstacles in its environment usually requires a lot of energy because the robot must store the images captured by the camera and repeatedly process all the 3D pixels in each image.
Instead of representing the environment using 3D pixels, which are cubes called voxels, the MIT researchers used a technique to map obstacles in space using ellipsoidal patches called Gaussians.
The size, shape, and thickness of these ellipsoids can be continuously adjusted, allowing them to conform to the shape of curved objects more effectively than inflexible cube-shaped voxels.
Importantly, the map registers obstacles and free space around the robot, which together allows the robot to plan a secure, collision-free path. Mapping obstacles and free space using voxels typically consumes a lot of memory, which makes classic methods energy-hungry. Because Gaussians can flexibly adapt to geometry, a single elongated ellipsoid can represent an area that would occupy many voxels, so occupied surfaces and free space are captured much more compactly.
In their up-to-date system on a chip, called Gleanmer, the researchers used an algorithm developed by their lab called GMMap which efficiently generates a 3D map of the robot’s environment by using Gaussians to represent obstacles.
With classic approaches, the robot would have to load and process each depth image several times to adjust the size and shape of the ellipsoids. The system typically constructed a Gaussian by comparing all the pixels of the image with each other. However, the amount of memory and power needed to do this remains too high for many edge devices.
To solve this problem, MIT researchers have developed a technique that can generate highly exact Gaussians from depth images with just one pass, after which they can be discarded, so the chip never has to save the entire image at once.
Instead of comparing each pixel with every other pixel in a 3D image, their algorithm assumes that nearby pixels belong to the same Gaussian, so you just need to compare each pixel with its neighbors.
“Only a few pixels need to be stored in memory at any given time, which greatly reduces the amount of memory required by our algorithm,” Li says.
Using co-design
However, as the robot moves through space, it usually sees the same object from different points of view. When it generates a Gaussian, some of them will overlap because they represent the same object. This may result in the 3D map being too enormous to be stored on the edge device.
Combining overlapping Gaussians makes the map more compact, but this typically requires the algorithm to process many raw pixels stored in memory. Scientists have developed a novel technique to perform the fusion process directly on the overlapping Gaussians, without having to revisit the original pixels. Since Gaussians are more compact than pixels, this significantly reduces memory and energy requirements.
The algorithms are based on the same principle – most of the calculations are based directly on compact Gaussians rather than the original pixels, which enables energy efficiency.
Scientists apply this principle to design a chip that stores the Gaussians they are actively working on in miniature, rapid on-chip memory, right next to the compute units. This is only possible because the Gaussian map is so compact.
The Gaussans that the robot will need to work on next are waiting in memory units built into the chip, so they don’t have to be retrieved from more distant, power-hungry off-chip memories.
“By having dedicated memory that only stores objects you saw in the previous few frames, you can access the data much more efficiently,” explains Fu.
They tested the system-on-chip by reconstructing a variety of different pre-existing 3D environments. The chip can also reconstruct obstacles and clear space directly from live data from the iPhone’s camera.
Gleanmer generated detailed 3D maps in real time, using about 6 milliwatts of power. It required only about 2.5 percent of the power that the best existing map-building chip would require.
By reusing compact Gaussians along a planned path, the chip allows the robot to chart a secure trajectory while using only about 20 percent of the energy it would otherwise need.
“We reduce memory usage by making sure the algorithm is efficient. We then accelerate the workload performed by that efficient algorithm, so that ultimately our chip is as efficient as possible,” Li says.
Scientists plan to further improve energy efficiency by moving the chip’s processing units closer to sensors that collect environmental data. They could also explore additional applications, such as using Gaussians to represent patterns. This can facilitate AI systems formulate elaborate plans more effectively.
“Real-time 3D mapping is the missing element for small autonomous systems. A drone inspecting a pipeline or a pair of AR glasses moving around a room need to understand the surrounding space – instantly, continuously and with almost no energy costs. Gleanmer makes this possible for the first time in a chip you can hold in your fingers,” says Karaman.
This work is supported in part by the MIT-MathWorks Fellowship, Amazon, the U.S. National Science Foundation, and Intel.