Robots walking down the street surrounded by astonished onlookers are an increasingly common sight. However, these machines are not yet the all-in-one assistants you would want in a kitchen or factory, and the main bottleneck is data. Like humans, robots learn best through experience. The challenge is that physically teaching these machines to do so many things in different settings is laborious and time-consuming.
“The natural idea is to use simulation as a training ground. While there has been significant progress in the last few years in the physics engines that power robotics simulators, one of the remaining challenges is creating sufficiently rich and varied simulation content to capture the complexities of the real world,” says Russ Tedrake, professor of electrical engineering and computer science (EECS), aeronautics and astronautics, and mechanical engineering at MIT and principal investigator at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).
It turns out that AI agents, semi-autonomous programs that “think” and perform well-defined tasks, can lend a hand create the realistic virtual settings that robots need. Fresh “SceneSmith” Developed by researchers at MIT CSAIL and the Toyota Research Institute, the system uses three agents to assemble objects, walls and the overall appearance of a 3D scene. Recreated indoor spaces such as restaurants, bedrooms and hotels are more realistic and detailed than previous systems, helping robots practice skills and try out different ways of performing tasks before turning them on. In turn, engineers save time on real-world testing.
Agents have a sense of what everyday places should look like because each of them references a multimodal system called the vision-linguistic model (VLM), specifically the state-of-the-art VLM GPT-5.2. It is trained from a gigantic amount of text and images from the Internet to support more visual cues. This advanced model provides each agent with some kind of spatial knowledge: first the “designer” agent generates the elements of the scene, then the “critic” advises on whether it looks realistic, and finally the “orchestrator” manages their work back and forth, deciding when the design is completed. Once the three VLMs have completed their artistic collaboration, the scene will be ready to be loaded directly into physics simulation software.
“We found that the system could construct 3D scenes in the same way a human designer would,” says MIT EECS graduate student Nicholas Pfaff, a CSAIL researcher and lead author of the study paper with Tedrake presenting the work. “We shot over 1,300 scenes using a leading VLM that has prior web-scale solutions, and created incredibly creative and diverse arrangements. I didn’t teach this system in prompts; it just improvised.”
Talk to my agent
With VLM agents, you can ask SceneSmith to do things like “generate a garage with a car, a workbench, tires stacked in the corner, and a ladder against the wall” and get a virtual playground opulent with items for the robot to tinker with. These rooms are decorated with up to six times the number of items per scene compared to previous methods, making them perfect for helping robots learn skills such as putting a cup in the sink, arranging fruit on plates, and carrying a soda can from shelf to table.
With so many opulent virtual environments at your fingertips, you can assess whether your robot is ready for deployment without having to do a lot of trial and error in the physical world. Researchers tested different roadmaps (also called “rules”) in SceneSmith’s digital worlds, creating 100 unique spaces. A VLM agent evaluated each attempt and found that the robot’s plans were flawed and the machine often failed to perform its duties. Humans agreed with the model’s verdict more than 99 percent of the time, which can lend a hand roboticists eliminate erroneous approaches in simulations before the robot starts moving in the real world.
But how realistic are these virtual worlds? This can be complex to prove completely, so researchers have approached the question from several angles. The most telling test: the politics of a pre-trained robot were thrown into the generated environments – an AI controller trained primarily on real-world data that had never seen a SceneSmith scene. In one test, users told the system to “take an apple out of the bowl and place it on the cutting board,” and the simulated robot did exactly that. If the scenes didn’t closely resemble the actual settings from which the policy was learned, it simply wouldn’t work.
The team remotely controlled robots in virtual spaces, guiding them to open cabinets, put away bottles and move between rooms. Their experiments showed that environments withstand long-term physical interactions beyond visual inspection.
Behind the scenes
Each of the agents SceneSmith uses has a well-defined role in the generation process, completing scenes step by step. They basically create a floor plan and bring it to life.
Let’s say you want to create a scene similar to the first floor of a house. The VLM “designer” started with a general layout, which the “critic” reviewed and then the “orchestrator” signed. Agents repeat this approach at every stage: adding furniture, placing objects on walls, then ceilings, and finally dropping objects that can be manipulated by robots. For example, VLMs can add cabinets that robots can open and close – an articulated element that was often absent in previous baseline solutions.
At each stage, the second VLM ensures that the scene is practical, advising, for example, the removal of the bathtub from the living room. The third VLM ensures that a high-quality scene is generated, even reversing the design process by a few revolutions if the graphics are not up to par. Once the three VLMs have completed their artistic collaboration, the mechanics of the physical world will be added using simulation software.
With a thorough understanding of what rooms should look like, where objects should be placed, and real-world physics, SceneSmith has a noticeable advantage over previous methods. Compared to scene generation baselines such as “HSM“And”Holodeck”, SceneSmith created environments with more facilities, including a private office, a pottery shop, and even a Minecraft-inspired game room.
SceneSmith was also a favorite of over 200 users. They found that the system’s graphics were more realistic more than 90 percent of the time. They also observed that, overall, this approach followed directions more accurately than other approaches. In other words, it was best at generating virtual playgrounds that users actually wanted to see.
Multi-talent system
Realism, variety and richness are SceneSmith’s strengths, even when it comes to generating individual 3D objects. You can ask it to create a wheeled cart and it will create a 2D image, which it will then turn into a detailed model with physical properties such as mass, friction and inertia.
However, such a detailed process requires a trade-off in speed. Creating a single scene can take hours as agents create and carefully analyze each object. With more processing power, the system could see a dramatic increase in performance. CSAIL engineers also hope to expand into deformable objects (such as sponges) if extensive 3D libraries become available.
“SceneSmith represents a significant advance in this area by providing an agent-based framework for generating simulation-ready indoor environments with a simple text message,” says Jeremy Binagia, an applied scientist at Amazon Robotics who was not involved in the research. “It advances the state of the art in several ways, including pushing the boundaries of object density in a simulated environment, ensuring that all objects are physically accurate (not just visually realistic), and creating assets that are not limited to a fixed library because they can be generated using text-to-3D.”
Pfaff and Tedrake wrote the paper with Thomas Cohn SM ’24, an MIT graduate student and CSAIL researcher; and Toyota Research Institute roboticists Sergey Zakharov and Rick Cory SM ’08, PhD ’10. Their work was supported in part by Amazon, the U.S. Office of Naval Research, the Toyota Research Institute, and the U.S. National Science Foundation.
The team brought their findings into the spotlight at last week’s International Machine Learning Conference.