Many engineering challenges come down to the same headache – too many knobs to turn and too few opportunities to test them. Whether it’s tuning the power grid or designing a safer vehicle, each assessment can be pricey and hundreds of variables can come into play.
Consider car safety design. Engineers must integrate thousands of parts, and many design choices can affect how a vehicle behaves in a collision. Classic optimization tools may start to have trouble finding the best combination.
MIT researchers have developed a novel approach that rethinks how a classic method known as Bayesian optimization can be used to solve problems with hundreds of variables. When testing realistic engineering patterns, such as power system optimization, this approach found the best solutions 10 to 100 times faster than commonly used methods.
Their technique uses a basic model trained on tabular data that automatically identifies the variables that matter most to improving performance, repeating the process to refine better and better solutions. Core models are huge artificial intelligence systems trained on huge, generic data sets. This allows them to be adapted to various applications.
The researchers’ tabular basic model does not require continuous training because it works towards a solution that increases the efficiency of the optimization process. The technique also provides greater speedup for more sophisticated problems, so it can be particularly useful in demanding applications such as materials development or drug discovery.
“Modern artificial intelligence and machine learning models have the potential to fundamentally change the way engineers and scientists create complex systems. We have developed a single algorithm that not only solves high-dimensional problems, but is also reusable, so it can be applied to many problems without having to start all over from scratch,” says Rosen Yu, a graduate student in computational sciences and engineering and lead author of the book article about this technique.
Yu is joined on the paper by Cyril Picard, a former MIT graduate student and research scientist, and Faez Ahmed, associate professor of mechanical engineering and core member of the MIT Center for Computational Science and Engineering. The research results will be presented at the International Conference on Learning Representations.
Improving a proven method
When scientists are trying to solve a multi-faceted problem but have pricey methods to evaluate success, such as crash testing cars to see how good each design is, they often operate a tried-and-true method called Bayesian optimization. This iterative method finds the best configuration for a sophisticated system by building a surrogate model that helps estimate what to investigate next, given the uncertainty of the predictions.
However, the replacement model must be trained after each iteration, which can quickly become computationally infeasible when the space of potential solutions is very enormous. Additionally, scientists must build a novel model from scratch every time they want to tackle a different scenario.
To address both shortcomings, MIT researchers used a generative artificial intelligence system, known as the tabular base model, as a replacement model in the Bayesian optimization algorithm.
“The basic tabular model is similar to ChatGPT for spreadsheets. The input and output of these models is tabular data, which is much more visible and used than language in the engineering field,” says Yu.
Like enormous language models such as ChatGPT, Claude, and Gemini, the model was pre-trained on massive amounts of tabular data. This makes it well equipped to solve a range of prediction problems. Additionally, the model can be deployed as-is without the need for retraining.
To make the system more right and proficient at optimizing, researchers used a trick that allows the model to identify features of the design space that will have the greatest impact on the solution.
“A car may have 300 design criteria, but not all of them are the main factor influencing the best design if you are trying to increase some safety parameters. Our algorithm can intelligently select the most important features to focus on,” Yu says.
It does this by using a tabular base model to estimate which variables (or combinations of variables) have the greatest impact on the outcome.
It then focuses the search on those high-impact variables rather than spending time examining everything equally. For example, if the size of the front crumple zone increased significantly and the car’s safety rating improved, this feature likely played a role in the improvement.
Bigger problems, better solutions
Yu says one of the biggest challenges was finding the best tabular foundation model for the job. They then had to combine it with a Bayesian optimization algorithm in such a way that it could identify the most crucial features of the design.
“Finding the most visible dimension is a well-known problem in mathematics and computer science, but finding a way to exploit the properties of the tabular base model was a real challenge,” says Yu.
With the algorithmic framework in place, the researchers tested their method by comparing it with five state-of-the-art optimization algorithms.
On 60 benchmark problems, including realistic situations such as power grid design and car crash tests, their method consistently found the best solution 10 to 100 times faster than other algorithms.
“When the optimization problem gets bigger and bigger, our algorithm really comes into its own,” Yu added.
However, their method did not outperform the baseline for all problems, such as path planning by robots. This likely indicates that the scenario was not well defined in the model’s training data, Yu says.
In the future, researchers want to explore methods that could improve the performance of tabular base models. They also want to apply their technique to problems with thousands or even millions of dimensions, such as warship design.
“At a higher level, this work points to a broader shift: the use of fundamental models not just for perception or language, but as algorithmic engines in science and engineering tools, enabling classical methods such as Bayesian optimization to be scaled to regimes that were previously impractical,” says Ahmed.
“The approach presented in this work, using a pre-trained foundation model combined with multivariate Bayesian optimization, is a creative and promising way to reduce the data-intensive requirements of simulation-based design. Overall, this work represents a practical and powerful step toward making advanced design optimization more accessible and easier to apply in real-world settings,” says Wei Chen, Wilson-Cook Professor of Engineering Design and chair of the Department of Mechanical Engineering at Northwestern University, who was involved in this research.