A better way to transform 2D designs into 3D models for rapid prototyping

Engineers often employ vision language models to create up-to-date designs, such as aircraft or car components. To simulate the performance of these components in realistic situations, they will employ proven computer-aided design (CAD) software to generate 3D models of these designs, which they can submit to virtual crash or durability tests.

Researchers at MIT and elsewhere have now developed a system that can train a vision language model to automatically convert 2D designs into CAD programs that are much more right and functional compared to other approaches, using only a fraction of the computation.

By improving the efficiency and effectiveness of AI-based CAD generation, this technique can streamline the rapid prototyping process and reduce costs. It can also assist engineers identify beneficial design choices they might otherwise overlook.

The system generates up-to-date data based on the model’s capabilities when attempting to convert a 2D image to CAD. The framework corrects model errors and incorporates them into the dataset containing successful solutions.

It uses this data to teach the model how to fix specific bugs and solve challenging problems that it would otherwise struggle with.

“We want engineers to be able to point our framework to a poorly performing CAD model, set a computational budget, and let the system take over—transforming their own model errors into better training data,” says lead author Giorgio Giannone, a research associate at MIT’s Design Computation and Digital Engineering (DeCoDE) Lab and principal scientist on Red Hat’s AI Innovation Team.

She joins him paper by Anna Claire Doris, mechanical engineering graduate from MIT; Amin Heyrani Nobari, postdoc at MIT; Kai Xu from RedHat; and co-authors Akash Srivastava, director of Core AI at IBM and principal investigator at the MIT-IBM Computing Research Lab; and Faez Ahmed, professor of mechanical engineering at MIT, head of the DeCoDE Lab and principal investigator at the MIT-IBM Computing Research Lab. The study’s results were recently presented at the International Machine Learning Conference.

“Almost every physical product around us, from airplanes to appliances, begins life as a CAD model. Industry teams crave AI that can speed up the creation of these designs, but today’s models often produce simple shapes inappropriate for practice. What excites me about this work is that it gives many models that convert image to CAD code the ability to improve themselves by learning from their mistakes rather than waiting for more human-made data, and this brings reliable AI design tools much closer to everyday engineering,” says Ahmed.

Data supporting the model

Scientists are working to build vision-linguistic models (VLM) for CAD generation. These VLMs take a 2D image and descriptive text and then generate Python code that can be executed in a CAD program to generate a 3D model of a physical object.

They analyzed the challenges of implementing existing VLMs for this task and determined that the main bottleneck limiting their capabilities was the lack of diverse, high-quality CAD datasets to train them.

To address this problem, efforts were made to create up-to-date data to train the CAD generation model, using a process known as data augmentation.

As part of data augmentation, scientists typically create up-to-date data by randomly modifying existing data to generate more samples, often adjusting the color, size, and shape of objects in the images.

Instead, MIT researchers have built a data augmentation system called GIFT (which stands for Geometric Inference Feedback Tuning), which generates data designed to improve the performance of one VLM for a specific task.

GIFT develops knowledge about the strengths and weaknesses of the model by testing it. It then uses this knowledge to generate data that can improve model performance for CAD generation problems that it has difficulty solving.

“We want to get data augmentation based on the model itself,” Giannone says.

Learning from mistakes

To do this, GIFT asks the model to generate code that solves the CAD generation problem several times in parallel. It validates these guesses to understand how well the model can solve the problem.

“With a model, generating almost correct CAD query code is not that difficult, but generating completely correct and executable code is much more difficult with standard VLM,” says Giannone.

For guesses that are almost correct, GIFT adjusts them to become effective solutions. It stores these “near misses” and successful solutions in a up-to-date dataset that can teach the model how to overcome the problems that usually tripped it up.

“If we try a model 10 times and generate 10 correct answers to the same problem, it won’t be able to learn much. We care about intermediate cases, where the model can only solve the problem 50 percent of the time,” he says.

Using these intermediate cases allows GIFT to generate data extensions that take into account both the model and the tasks. Additionally, by including multiple correct solutions to the same problem, the up-to-date data expands the model’s overall knowledge of CAD code generation.

This automatic system does not require human intervention to correct model errors.

GIFT creates data extensions from a pre-trained VLM using a process known as inference time scaling. This process allows a unchanging model that has already been trained to generate better results without the high computational cost of retraining the entire model.

Using inference time scaling, the user can determine how much computation they want to employ for GIFT, adapting it to their time and budget constraints.

GIFT outperformed several competing techniques by generating CAD programs that were more right while using only about 20 percent more computation. CAD models generated by VLM using GIFT were better suited to the shapes of the base models.

“For GIFT, we started with geometry because in engineering problems, if the geometry of the 3D shape is not correct, nothing else will be correct, but there are many other aspects to consider,” Giannone says.

In the future, researchers want to expand GIFT so that the platform can train CAD generation models that will improve the performance and manufacturability of 3D models. They also want to apply the system to larger models and more diverse CAD generation tasks.

This research was funded in part by the MIT-IBM Computing Research Lab.