Engineers often employ vision language models to create up-to-date designs, such as aircraft or car components. To simulate the performance of these components in realistic situations, they will employ proven computer-aided design (CAD) software to generate 3D models of these designs, which they can submit to virtual crash or durability tests.

Researchers at MIT and elsewhere have now developed a system that can train a vision language model to automatically convert 2D designs into CAD programs that are much more right and functional compared to other approaches, using only a fraction of the computation.

By improving the efficiency and effectiveness of AI-based CAD generation, this technique can streamline the rapid prototyping process and reduce costs. It can also assist engineers identify beneficial design choices they might otherwise overlook.

The system generates up-to-date data based on the model’s capabilities when attempting to convert a 2D image to CAD. The framework corrects model errors and incorporates them into the dataset containing successful solutions.

It uses this data to teach the model how to fix specific bugs and solve challenging problems that it would otherwise struggle with.

“We want engineers to be able to point our framework to a poorly performing CAD model, set a computational budget, and let the system take over—transforming their own model errors into better training data,” says lead author Giorgio Giannone, a research associate at MIT’s Design Computation and Digital Engineering (DeCoDE) Lab and principal scientist on Red Hat’s AI Innovation Team.

She joins him paper by Anna Claire Doris, mechanical engineering graduate from MIT; Amin Heyrani Nobari, postdoc at MIT; Kai Xu from RedHat; and co-authors Akash Srivastava, director of Core AI at IBM and principal investigator at the MIT-IBM Computing Research Lab; and Faez Ahmed, professor of mechanical engineering at MIT, head of the DeCoDE Lab and principal investigator at the MIT-IBM Computing Research Lab. The study’s results were recently presented at the International Machine Learning Conference.

“Almost every physical product around us, from airplanes to appliances, begins life as a CAD model. Industry teams crave AI that can speed up the creation of these designs, but today’s models often produce simple shapes inappropriate for practice. What excites me about this work is that it gives many models that convert image to CAD code the ability to improve themselves by learning from their mistakes rather than waiting for more human-made data, and this brings reliable AI design tools much closer to everyday engineering,” says Ahmed.

Data supporting the model

Scientists are working to build vision-linguistic models (VLM) for CAD generation. These VLMs take a 2D image and descriptive text and then generate Python code that can be executed in a CAD program to generate a 3D model of a physical object.

They analyzed the challenges of implementing existing VLMs for this task and determined that the main bottleneck limiting their capabilities was the lack of diverse, high-quality CAD datasets to train them.

To address this problem, efforts were made to create up-to-date data to train the CAD generation model, using a process known as data augmentation.

As part of data augmentation, scientists typically create up-to-date data by randomly modifying existing data to generate more samples, often adjusting the color, size, and shape of objects in the images.

Instead, MIT researchers have built a data augmentation system called GIFT (which stands for Geometric Inference Feedback Tuning), which generates data designed to improve the performance of one VLM for a specific task.

GIFT develops knowledge about the strengths and weaknesses of the model by testing it. It then uses this knowledge to generate data that can improve model performance for CAD generation problems that it has difficulty solving.

“We want to get data augmentation based on the model itself,” Giannone says.

Learning from mistakes

To do this, GIFT asks the model to generate code that solves the CAD generation problem several times in parallel. It validates these guesses to understand how well the model can solve the problem.

“With a model, generating almost correct CAD query code is not that difficult, but generating completely correct and executable code is much more difficult with standard VLM,” says Giannone.

For guesses that are almost correct, GIFT adjusts them to become effective solutions. It stores these “near misses” and successful solutions in a up-to-date dataset that can teach the model how to overcome the problems that usually tripped it up.

“If we try a model 10 times and generate 10 correct answers to the same problem, it won’t be able to learn much. We care about intermediate cases, where the model can only solve the problem 50 percent of the time,” he says.

Using these intermediate cases allows GIFT to generate data extensions that take into account both the model and the tasks. Additionally, by including multiple correct solutions to the same problem, the up-to-date data expands the model’s overall knowledge of CAD code generation.

This automatic system does not require human intervention to correct model errors.

GIFT creates data extensions from a pre-trained VLM using a process known as inference time scaling. This process allows a unchanging model that has already been trained to generate better results without the high computational cost of retraining the entire model.

Using inference time scaling, the user can determine how much computation they want to employ for GIFT, adapting it to their time and budget constraints.

GIFT outperformed several competing techniques by generating CAD programs that were more right while using only about 20 percent more computation. CAD models generated by VLM using GIFT were better suited to the shapes of the base models.

“For GIFT, we started with geometry because in engineering problems, if the geometry of the 3D shape is not correct, nothing else will be correct, but there are many other aspects to consider,” Giannone says.

In the future, researchers want to expand GIFT so that the platform can train CAD generation models that will improve the performance and manufacturability of 3D models. They also want to apply the system to larger models and more diverse CAD generation tasks.

This research was funded in part by the MIT-IBM Computing Research Lab.

The post A better way to transform 2D designs into 3D models for rapid prototyping appeared first on AI SCKOOL.

Q: Your article introduces “neural transparency,” a way for everyday users to see into artificial intelligence neural networks before their chatbot utters a word. Can you describe how it actually works and why you focused on the design moment rather than catching problems once the chatbot is already running?

AND: Millions of people are now creating personalized chatbots and AI agents based on enormous language models, turning them into collaborators, tutors, coaches, original partners and companions with straightforward text prompts. However, most people have no idea how these prompts will affect the AI’s behavior until they start interacting with it. We wanted to change that.

“Neural transparency” means allowing people to do something like a brain scan to look for artificial intelligence. Not because the AI ​​has a human brain, but because its neural network contains internal patterns that can suggest how it might behave before it speaks. In this work, my students Anthony Baez, Sheer Karny, and I combined insights from the fields of human-AI interaction and mechanistic interpretability to make these hidden patterns accessible to everyday users.

The basic idea is straightforward. First, we choose behaviors that we care about, such as empathy, honesty, toxicity, hallucinations, or flattery. We then compare the model’s internal activations when asked to demonstrate one feature with its opposite. This difference becomes a kind of “behavior direction” within the model. When a user writes a custom system prompt—instructions that shape the personality of their chatbot before any conversation begins—we project the model’s internal activations into these directions and translate the results into an intuitive visualization. In our case, it’s a sun diagram that shows the chatbot’s likely personality traits before a user starts talking to it.

We focused on the design moment because this is where prevention is possible. Nowadays, people often discover problems only after the chatbot has behaved in an unintended way. Our goal was to move from reactive correction to anticipatory design, helping people identify potential risks as they shape AI.

Q: Your study found something quite striking: people consistently misjudge the behavior of their personalized AI, overestimating good qualities and underestimating potentially harmful ones, such as flattery. What does this tell us about the dangers of the way millions of people are currently creating AI companions, and why is it so tough to close this blind spot?

AND: I often joke that if artificial intelligence looked like the Terminator, it would be much easier for us to know what to do. The real challenge is that AI often appears as a dear friend, coach, tutor or companion. This makes it tough to recognize when something is going wrong.

Our research suggests that humans have a blind spot when designing personalized AI. People often think they know how their chatbot will behave, but in our study they incorrectly predicted its personality based on 11 of the 15 traits we measured. This highlights the need for tools that support people better understand AI before they start using it.

This matters because some behaviors that seem helpful in the moment may not be fit over time. We have documented cases in previous studies psychological harm related to interactions with AI chatbots. LLM [large language model] that constantly confirms your opinions or never challenges your thinking can reinforce harmful decisions, unhealthy beliefs, or emotional dependence. Psychology has long shown that humans are naturally drawn to affirmation, so designing artificial intelligence is not only a technical challenge, but also a psychological one.

The deeper problem is that today’s AI systems remain largely black boxes: even experts can’t always predict how a system message will affect the AI’s behavior over the course of a long conversation. As AI companions become part of everyday life, we need tools that support people understand what they are building before they start using them. Artificial intelligence should be supportive but not blindly conciliatory, personalize without manipulation, and be crystal clear enough for people to make informed choices.

Q: One of the most fascinating findings is that visualization significantly increased user trust, but it didn’t actually change the way people designed their chatbots. What will it take to close this gap, and where do you see such tools heading as AI companions become more deeply embedded in people’s everyday lives?

AND: I think this is one of the most fascinating conclusions of the article because it shows that transparency alone is not enough. Users appreciated the opportunity to have insight into the model and reported greater trust in the system, but simply presenting the information did not fundamentally change the way they designed their AI companions.

In our further work that is currently underway available as a preprintwe examine how the model’s internal neural representation changes over the course of a multi-turn conversation, rather than remaining constant from the initial prompt. We are already seeing promising results. Visualizing how these internal representations change over time allows people to be much better at recognizing and predicting changes in AI behavior, and less likely to become overconfident in their understanding of the chatbot. AI companions are energetic systems that evolve as they interact with us, so understanding these internal changes is an critical next step. Nevertheless, this is still a very juvenile area of ​​research.

Looking ahead, I believe these types of transparency tools could become as common as food nutrition labels. As AI becomes deeply woven into education, healthcare, work and personal relationships, people should be able to understand not only what AI can do, but also its impact on their thinking, emotions and behavior. This kind of transparency is indispensable if we want AI to actually support people thrive.

The post 3 questions: Neural transparency and the future of AI design appeared first on AI SCKOOL.

With the growing popularity of generative AI, many open-source models are now available on the Internet for anyone to adapt to their tasks, such as generating product renderings in a specific artistic style.

However, these models also fall into the hands of nefarious actors who can optimize them to create illegal content such as hate speech or child sexual abuse material (CSAM). This is a growing problem – National Center for Missing and Exploited Children received over 1.5 million reports AI-generated CSAM in 2025, up from 67,000 in 2024.

Engineers typically test AI for malicious capabilities by monitoring the model and checking its results, but with CSAM this is impossible because in the US it is illegal to generate such content, regardless of intent.

To avoid this dilemma and improve AI security, a team of MIT researchers, led by graduate student Vinith Suriyakumar and associate professors Ashia Wilson and Marzyeh Ghassemi, teamed up with researchers from Thorn develop a recent audit approach that determines whether a model can generate CSAM without prompting it. Thorn is a nonprofit child safety organization with a mission to transform how children are protected from sexual exploitation and abuse in the digital age.

Their technique examines how the inner workings of the model have been adapted, but never produces results. By examining hidden representations, it can be reliably inferred whether a model has been specialized to create malicious images.

During testing, the audit procedure identified model variations that specialized in generating CSAM with 100% accuracy. The hosting platform can exploit this technique to flag unsafe models and remove them quickly or prevent them from being uploaded in the first place.

“This opens up new opportunities for platforms supporting open-source models, and for law enforcement to actually test whether a model can generate CSAM. Previously, we had no way to measure this. It was a huge blind spot that some took advantage of. Now we can solve an AI security problem that has serious negative consequences,” says Vinith Suriyakumar, an electrical engineering and computer science (EECS) graduate student at MIT and lead author of a paper on the technique.

Suriyakamura and Wilson, the Lister Borthers Career Development Professor at EECS and principal investigator at the Laboratory for Information and Decision Systems (LIDS), are joined on the paper by Lena Stempfle, a postdoc at MIT; Ghassemi, associate professor at EECS and member of the Institute of Medical Engineering Sciences (IMES) and LIDS; and others at Boston University and Thorn. The paper was presented in the spotlight at the Trustworthy AI for Good workshop at the International Machine Learning Conference.

Adaptation audit

Recent techniques have made it easier for users to specialize a generative AI model for their tasks through a process known as tuning.

Instead of retraining the entire model on a task-specific dataset, individuals can exploit an algorithm called low-rank adaptation (LoRA) to specialize the model in a more competent way.

This has led to a wave of recent variants of generative AI models for various purposes, such as creating watercolor paintings that mimic artistic movement. However, it also allowed malicious actors to create models that can generate high-quality CSAM and other malicious images.

To audit a model, engineers typically query it about malicious content and check the results, but this manual auditing procedure is not scalable. Furthermore, the repeated generation of disgusting images may have a negative psychological impact on evaluators.

This evaluation method quickly falls apart when testing CSAM, which is illegal to generate for any purpose in the US and many other international jurisdictions.

“We are in a very difficult situation where, by law alone, we cannot apply de facto assessment measures. We have had to throw away the entire toolkit and take a different approach,” says Suriyakumar.

After learning about this puzzle, researchers teamed up with Thorn to solve the problem.

Non-generative solution

Instead of focusing on the results, researchers focused on the modifications made by the LoRA algorithm during tuning.

Their technique examines these modifications, called LoRA adapters, to determine whether the model has been specialized for malicious features, without generating results.

Using a technique called Gaussian probing, researchers feed the model a set of random data points and analyze how it manipulates that data within its multi-layered internal structure.

“We never run the model all the way or prompt it, so we never generate images,” explains Suriyakumar.

Scientists capture these modifications at multiple time points of the model’s internal structure and average them to summarize how the LoRA adapter changed the model’s calculations. They found that these responses provided a sturdy signal of model specialization.

They tested their method on variations of three types of models, comparing the results with real data from LoRA adapters known to generate CSAM, other malicious images, and safe and sound content.

Their method was 100% exact in identifying models suited to generating CSAM.

“There are a lot of child safety concerns around AI, and these are real concerns that need to be addressed. Many children are harmed by AI fakes. We’ve shown that Gaussian probing can be a very useful tool, and we hope that the research community will really pay more attention to this issue,” Wilson says.

Importantly, their technique is scalable and would be relatively inexpensive to implement. With thousands of model variations published on the Internet every month, scalability is crucial because it helps auditors remove harmful adaptations before they are widely distributed.

Gaussian probing is also more stalwart than other auditing techniques because a nefarious actor would have to carefully alter the inner workings of the underlying model to avoid detection.

In the future, the researchers want to evaluate their technique on a larger set of model variations and investigate whether Gaussian probing can detect malicious features in base models before they are adapted.

“We now have a technological approach that can partially solve this problem. So much effort has gone into this collaboration, which has allowed us to tackle a really difficult problem that is harming so many children in this country and around the world. Hopefully, we can have a transformative impact in this area,” says Ghassemi.

This work was supported in part by a Bridgewater AIA Labs Research Fellowship.

The post The recent method aims to protect children from illegal content generated by artificial intelligence appeared first on AI SCKOOL.

Most people think of the waterfront as the edge of the city. The MIT research team sees it as a energetic construction site, reminiscent of Lego bricks.

Their recent system, called “Floating form” is a swarm of small, square, robotic boats that come together into larger structures on the water, break apart, and reassemble into something new, all with minimal human guidance.

About the size of a dinner plate and 21 square centimeters in size, each robot is a self-contained ship with its own thrusters, sensors and magnetic latches. Together they point to a future in which floating infrastructure could become more adaptive: a temporary platform in the event of an emergency, a market on a canal, or a stage that appears at a festival and disappears when the crowd returns home.

“Our FloatForm designs envision a future in which the waterfront becomes a programmable extension of the city, where autonomous boats can self-organize into bridges, platforms, and other useful structures on demand,” says Daniela Rus, professor of electrical engineering and computer science at MIT and director of MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). “This type of distributed robotics opens up new possibilities in mobility, emergency response, public spaces and on-water infrastructure.”

“With FloatForm, we essentially transform static water surfaces into dynamic, programmable spaces,” says Wei Wang, lead author of the new article about the project and a former MIT scientist who now directs the Marine Robotics Laboratory at the University of Wisconsin, Madison. “Imagine an urban environment in which public space is not fixed but can autonomously expand, contract, or reconfigure on demand.”

“We see it as creating infrastructure on the water using a modular system to create one larger system,” says Alejandro Gonzalez-Garcia, a former researcher at MIT CSAIL and Senseable City Lab. “In an emergency, you can create a recent bridge to ease traffic in the city. You can also create floating markets and floating stages. If you want to make the city more livable, you also want to operate the water.”

public works, published today in comes from the labs of Rusa and Carlo Ratti, professors of the practice of urban technology and planning at MIT and director of the Senseable City Lab, and comes from Roboat, their joint project with the Amsterdam Institute for Advanced Metropolitan Solutions that aims to place full-scale autonomous ships on Amsterdam’s canals. These canals were once used to transport city goods; today they mainly transport tourists.

“We investigated whether canals could be used for waste collection or transportation to transfer some of the stresses that occur on roads back to the water,” says Niklas Hagemann, an MIT architecture graduate, CSAIL division and former Senseable City Lab researcher who worked on the project from its early stages. “Urban areas are becoming denser, so can public space be expanded to include water that is currently underutilized?”

FloatForm reduces this vision to the tabletop scale to answer a more difficult question: How did dozens, and eventually thousands, of floating robots organize themselves?

Lessons from a raft of ants

The team found the answer in biology. Fire ants are famous for surviving floods by combining their bodies into living rafts, and no leader supervised the choreography of the gathering. Each ant follows simple local rules and a resistant structure is created.

“Each ant is an independent agent,” says Gonzalez-Garcia. “We wanted each robot to have its own capabilities, just like an ant colony forms a raft.”

Most existing self-assembling robotic systems on water and elsewhere rely on a central computer dictating every movement. This approach is susceptible to single points of failure and scales poorly: the scheduling math balloons as robots are added, and the swarm must assemble sequentially, with most robots idling, waiting for their turn. FloatForm reverses the balance. A lightweight central planner steps in sparingly, assigning each robot a final position for mesh refinement, a level of geometric precision that distributed methods alone can hardly guarantee. Everything else, including navigating towards a target shape, avoiding collisions and adapting to disturbances, works on the robots themselves, which coordinate actions by exchanging positions with their immediate neighbors. The entire swarm moves at once.

This parallelism is what makes this work stand out. The scheduling complexity of the FloatForms approach depends only on the robot’s local neighbors, not on the overall size of the swarm. “We try to keep central intervention to a minimum and get them all moving together at the same time,” says Gonzalez-Garcia.

In experiments at MIT, a fleet of eight robots repeatedly moved into random positions, assumed a target shape, snapped into a rigid structure, broke apart on command, reassembled into a new configuration, and then traversed the pool as a single craft, with each run lasting four to eight minutes. In this latter mode, called mass transit, the planner determines the trajectory of the entire structure and each robot calculates its own contribution. “Every robot becomes an actuator,” explains Gonzalez-Garcia. Simulations showed a smooth scaling of the structure to swarms after 64.

“The beauty of this largely decentralized approach is that the computations don’t get bogged down as the swarm grows,” Wang says. “Whether you’re working with eight or 80 boats, the entire fleet coordinates and moves simultaneously. Because overall assembly time is generally not significantly increased, the system remains highly scalable.”

Sticking together also has physical benefits. “Our boats become more stable, coming together like a raft of ants if there are waves or currents,” Hagemann says.

Origami handshake

The robots connect via a latch mechanism hidden entirely inside each hull. A single servo motor positioned in the center drives an origami-inspired auxetic structure whose geometry contracts evenly in all directions simultaneously, pulling permanent magnets on all four sides inward for release or pushing them outward to grab a neighbor in 10- to 15-centimeter increments. The magnets are arranged with variable polarity, allowing the boats to reliably connect to clean square grids.

The elegant part is what the mechanism doesn’t do: use (a lot of) power. The 3D printed gear keeps the latch in any state with the engine turned off. “It uses energy to latch and unlock, but it doesn’t use any energy in between,” says Hagemann. This is important for infrastructure that can store configuration for many hours. “Because the robots are so small, you can only have a battery that big,” adds Gonzalez-Garcia. “If they use less energy snapping, they can use more energy calculating or actually moving.”

It took humility and engineering to get there. Four miniature thrusters arranged in an “X” provide each robot with omnidirectional motion, including rotation in place, but they exert large forces compared to the robot’s low inertia, which made early prototypes twitchy and prone to aggressive spins at low speeds. The team added stabilizing ribs to increase hydrodynamic resistance and tuned the controllers to maintain the strength of the robots, which at this scale are never completely identical. The magnets presented their own problem: They held on so well that detaching sometimes required the robots to wriggle away.

From the reservoir to the canal

In 10 trials, the system performed its tasks without human intervention 90 percent with four robots and 70 percent with eight. When something went wrong, the architecture showed its resilience: a robot that became momentarily disoriented could rejoin the structure on its own without stopping the entire swarm, and robots stuck in formation stalemates learned to shake themselves off and try again.

Moving from a controlled internal reservoir to a real canal or port will require more than just confidence. “There is always a relationship between the size of a boat and the amount of disturbance it can withstand,” says Gonzalez-Garcia. “These boats are very small, so they can’t work in very rough water.” Scaling up will mean strengthening latches, potentially with mechanical locks like the full-size Roboat used, and replacing indoor ultrasonic lab positioning with GPS or vision sensors. Interestingly, the coordination algorithm is designed to be sensor-agnostic: swap sensors, keep logic.

The team envisions applications beyond urban channels, from creating temporary platforms for inspection and maintenance at sea, to adaptive sensor networks for studying migratory species, to reconfigurable docking stations for emergency response in hard-to-reach areas. There is also the potential to conduct activities at sea and at a distance, from temporary construction platforms to environmental monitoring and scientific expeditions.

And the geography is wide open. “Venice, the Netherlands, Belgium, the fjords and lakes of Norway, basically any city with a river can benefit from this,” says Gonzalez-Garcia. “The design takes advantage of spaces where water is already important, but it also raises the question: Where else can water be used for something more?”

“This is an exciting step forward in the study of distributed collective behavior on water,” says University of Michigan assistant professor Steven Ceron, who was not involved in the research. “Assembly, self-reconfiguration, and collective movement are tough enough in a arid environment, but achieving these behaviors in a predominantly distributed manner over water poses significant additional challenges, and this team has plausibly managed to overcome them. By offloading the computational burden to the robots themselves, they have built a more resilient system that could, in the near future, enable the deployment of such robot collectives in open water environments for search operations, environmental monitoring, and reconfigurable marine infrastructure.”

Gonzalez-Garcia, Hagemann and Wang wrote the paper with senior authors Ratti, who is also a professor at the Politecnico di Milano and Rus. Gonzalez-Garcia is additionally associated with the MECO Research Team at KU Leuven. The research was supported by a grant from the Amsterdam Institute for Advanced Metropolitan Solutions and additional support from the University of Wisconsin at Madison. The team thanks MIT Sea Grant and Professor Michael Triantafyllou for providing the test tank.

The post Petite robot boats build floating structures appeared first on AI SCKOOL.

From the basement of MIT Building 35 to Monterrey, Mexico, and now beyond. Such is the journey of FrED, a low-cost, stationary (Fr)(E)(D) fiber extrusion device designed and assembled by students at the MIT Education Factory.

This factory is changing the way manufacturing is taught, replacing textbook learning with hands-on experience in a space where tinkering and a constant flow of information are encouraged. Thanks to a collaboration between MIT and Tecnológico de Monterrey (Tec), managed by MIT.nano, FrED has been refined in dozens of master’s theses and undergraduate research stays. It is used to study production systems in academic and professional courses and in FrED factories, first established at MIT and now at Tec campuses in Monterrey and Mexico.

“What does it mean to bring the factory closer to the student?” asked Brian W. Anthony, deputy director of MIT.nano and principal scientist in MIT’s Department of Mechanical Engineering (MechE), during the second annual FrED Summit in Mexico. “We have FrED as a process that makes the fiber, we also have an FrED factory, which is an education and practice factory where we make the real product. It’s not just a learning factory where we tear the product when we’re done. We’re really sending FrED to our online students, educators at MIT and Tec, and soon to new partners around the world.”

Designed from the ground up to scale multi-node communities, FrED and the FrED factory have created a booming collaborative ecosystem for current and future manufacturing engineers. The next step is the global expansion of this ecosystem. Announced at the FrED Summit by Tec Professor Pedro Ponce Cruz, the modern FrED plant on the Tec Saltillo campus will open in the next academic year. The team then plans to expand to other campuses in the United States and Mexico.

“Together, we are helping to build a global pipeline of engineering talent,” says Adriana Vargas Martinez, executive director of research strategy at Tec. “Thanks to the FrED and FrED factories initiative, nearly 500 students have already been trained in advanced manufacturing automation, moving from Tec classrooms to research labs and projects in collaboration with MIT.”

Discussing the research impact of FrED and the FrED factory, he notes 25 publications and 7 articles in preparation. “International mobility was also an important dimension of this partnership,” he says.

Moving towards newfangled topics related to deep technology

FrED’s expansion comes as manufacturing at MIT and across the industry moves toward intelligent manufacturing, or Industry 4.0, integrating automation, machine learning and artificial intelligence. One of MIT’s strategic priorities is the MIT Fresh Manufacturing Initiative (INM), is working to support modern manufacturing research, develop modern courses and employee training, and build collaborative facilities to pilot production lines and immersive manufacturing experiences. FrED and the FrED factory are already designed to support these efforts, and on an international scale.

“I think FrED and the FrED factory really solve at least one problem: how do we provide real, physically meaningful physical context and data at the manufacturing level, manufacturing-level problems in academia that can be directly transferred to the knowledge needed on the factory floor,” Anthony says. He adds that it is complex to obtain data from a real factory; what FrED offers is a physical context crossed with data science, providing an open platform and open data for learning and experimentation.

FrED naturally generates the multimodal data required for digital twins, analytics, and AI-driven process improvement, turning the abstract integration of AI and manufacturing into practical practice. The next set of research goals at the FrED factory will focus on developing a realistic and interactive digital twin of the factory, an immersive technology for collaborative learning that integrates agent-based controllers. These will include modern downstream manufacturing processes and machines using FrED fiber as input – all with the aim of improving intelligent manufacturing education.

Students from MIT and Tecnológico de Monterrey will work on these goals as part of the FrED research stay at the factory. In this program, Tec students study at MIT to work side by side with MIT students – not observing, but fully integrating with the research team. Students then take what they learn to Mexico to improve FrED factories at their home institution.

“Beyond the technical side, FrED provided me with memories, friendships, and much more confidence than I thought I would,” says Naomi Najera, an undergraduate Tec student who completed a research internship at MIT in 2025. “It also gave me the space to make mistakes and learn from them. And to realize how much I could accomplish with my team. The human side of this project really changed my entire experience.”

A recent result of this exchange, announced June 23 by the American Society for Engineering Education (ASEE), a paper titled “Hands-On Predictive Maintenance Kit for Manufacturing Education: An Accessible Experiential Learning Approach,” written by Tec and MIT students, received the 2026 ASEE Manufacturing Division Best Paper Award.

Transferring classroom learning to factory operations

On the MIT campus in Cambridge, Massachusetts, passersby can peer into the basement of Building 35 and see the constant flow of activity, materials and knowledge at the MIT FrED plant. In Mexico, seven cohorts of students over four years designed a custom version of FrED and built and operated an automated FrED factory production line. Indeed, FrED has restructured the way Tec teaches mechatronics and manufacturing systems. “This collaboration integrates research directly into education,” says Vargas Martinez, “connecting learning factories and our manufacturing environments with student-centered research.”

Tec students’ enthusiasm led to the launch of an Undergraduate Research Opportunities program in Mexico (BOX: Factory Research for All in Mechatronics Education), in which first-year students work alongside graduate students in a FrED factory.

“Joining FrED as a first-semester university student was an incredible opportunity for me to gain hands-on experience on real-world projects in areas such as coding, manufacturing and robotics,” says Katherine Lucia McLean. “It helped me grow a lot as an engineering student.”

The FrED factory model enforces true leadership behaviors: coordinating multi-station systems, managing bottlenecks, embedding maintenance logic into the student experience, enforcing quality measurement, and iterating system design year after year. As each class ends and a modern one begins, knowledge is passed on, some of it is lost, and most of it is built upon. In this way, FrED never becomes obsolete as each cohort reinvents technologies and production systems to create a smarter, more productive factory.

FrED and the FrED factory are gaining momentum. Last year, Anthony taught a global capstone course at the Monterrey campus, and in 2027 he will expand to teach at all five of Tec’s international campuses. The FrED Factory Conference will be held at MIT in 2027.

The post The MIT-designed learning factory embraces newfangled manufacturing appeared first on AI SCKOOL.

Q: What inspired the “Beyond Data-Driven Aesthetics” project and what questions does it raise?

AND: The conceptual origins of “data-transcending aesthetics” emerged from three intersecting lines of research.

First, while completing my PhD in Design and Computation at MIT’s School of Architecture around 2022, I watched in real time as advances in data-driven machine learning – systems like ChatGPT and Stable Diffusion – were rapidly entering public discussions about creativity, aesthetic judgments, design, and even high-profile art auctions.

At the same time, my own research was already focusing on aesthetic judgment and evaluation, and it became increasingly clear to me that many of the questions presented to the public as “new” regarding artificial intelligence actually have a much longer history, dating back to the 20th century. For example, in the 1956 Dartmouth Summer Research Project, the fundamental event in artificial intelligence, creation, and evaluation processes was identified as one of seven key dimensions of human intelligence that future artificial intelligence research should address.

Second, the exhibition was influenced by research on design computation and shape grammar, which explored the relationship between human insight and computation using rule-based methods rather than solely data-driven learning. More recent interpretive studies of aesthetic theories – drawing on figures such as Samuel Taylor Coleridge, Oscar Wilde, and even John von Neumann – have been particularly vital to me. This research examines whether theories of aesthetic value and comparisons articulated in philosophical and literary texts can reveal possibilities or limitations in contemporary models of digital computing and artificial intelligence in architecture and design.

Finally, the theme of the exhibition was the operate of data design, production and visualization as methods for interpreting mathematical concepts, algorithms and “black box” machine learning systems. Across disciplines, researchers are increasingly using reconstruction and visualization techniques to make computational systems more actual and interpretable – from neural network visualization in computing to software reconstruction and digital fabrication in architecture and curatorial practice.

Q: How to translate computational and aesthetic research into an exhibition?

AND: The exhibition’s approach is to ask what exactly in a particular research article or book captures the most salient idea, and then operate design to interpret that idea in a visual, spatial and experiential format. Drawing on design techniques such as software reconstruction, physical creation, and data visualization, the exhibition takes written sources that are luxurious in algorithmic ideas, abstract concepts, and mathematical formulas, and translates them into stories in space that include interaction, material form, and digital visualization.

The exhibition itself is organized around five thematic areas: Aesthetic Measure, Aesthetic Guidelines, Algorithmic Aesthetics, Aesthetic Appropriation and Aesthetic Novelty. Each topic functions as a selective “window” into a distinct computational approach to aesthetic evaluation drawn from a specific publication – a book or research article. The titles of these topics are derived from concepts central to each publication. For example, “measure” refers to mathematician George Birkhoff’s work in the 1930s to mathematically determine aesthetic value, while “novelty” examines how the AICAN machine learning system evaluates how images are generated according to the theory of cognitive aesthetics, which balances familiarity with and deviation from familiar artistic styles.

In all five cases, the key insight is that design itself can function as a method of interpretive translation—a way of making observable, actual, and experientially what time-honored academic scholarship in technical fields typically communicates only through words and word-like representational devices such as scientific diagrams and tables.

Q: What questions do you hope to explore next?

AND: “Beyond Data-Driven Aesthetics” is conceived as both a research exhibition and a indefinite platform for examining how computational systems participate in processes of aesthetic appreciation, generation and transformation in architecture and the applied arts.

One of the central questions of the exhibition—which is increasingly the focus of architecture, design, and engineering scholars—is computational evaluation beyond purely performative or functional requirements. This applies to many different design spaces, whether buildings, structural forms or everyday products. The case studies in the exhibition suggest that many of these questions arose long before the current interest in computer science and artificial intelligence, and have been approached through a range of computational and theoretical assessment models since at least the early 20th century.

At the same time, I am becoming increasingly interested in how these ideas can be transferred to broader built environment applications. In particular, I am interested in how research related to “Moving Beyond Data-Driven Aesthetics” can facilitate designers and engineers better understand how computation – whether rule-based or data-driven – can inform us about what positively impacts human experiences of the spaces and objects in which people live and operate.

Finally, a direction that I continue to explore is the methodological role of the project itself as an interpretive tool. Through software reconstruction, visualization, and physical creation, the exhibition uses design to translate cloudy computational systems into more readable, actual, and experiential artifacts. More broadly, this raises questions not only about the mechanization of “beauty” or “taste” (the time-honored preoccupation with aesthetic formalism in the 20th century), but also about how time-honored forms of scientific inquiry and communication can evolve through spatial, visual, and public-facing formats.

The post 3 questions: Beyond data-driven aesthetics appeared first on AI SCKOOL.

MIT News

Q: What is agent-based AI and how does it differ from generative AI models such as ChatGPT and Claude?

AND: Agentic AI is artificial intelligence that takes action in the world. These actions can be physical, such as manipulating a robot, or digital, such as booking a flight. On the other hand, we think of generative AI as inventing stories, poems, works of art and images, rather than taking actions for us.

The word “agent” is just a brand name. This usually means artificial intelligence that will aid people interact with an app, website or the physical world. Most of the agents we encounter today are digital agents, such as customer service agents, who you can talk to about product complaints.

Most agent companies operate the same few AI models that give them the ability to take action and remember what happened. The agent starts with a basic, generative AI system like Claude. Companies then place different packaging around this base model of their product or application. These wrappers can be specific tools that an agent can operate, and these tools are application-specific. Perhaps the agent has access to a calculator so he can solve math problems, or maybe he has access to a more complicated demanding drive and operating system so he can remember company financials and past business negotiations.

The biggest challenge in developing agentic AI is the lack of training data. If I want to create a system that can go online and book a flight for me, it seems pretty elementary. But we don’t have much data that explains exactly how to do it – where to move the mouse, which buttons to click, what to do if something goes wrong, or how to call someone and negotiate the price of an airline ticket. One way to train such a system is to ask an AI agent to visit airline websites, try different solutions and see what works and what doesn’t. These environments are complex to model, so the agent often has to learn through trial and error.

Q: What are the promising applications of agentic AI?

AND: I think the area where we had the most success was coding agents. This is something that evolved from generative artificial intelligence. People trained language models on code and then could predict what a human would do to solve a coding problem. Additionally, the agent can learn this by going through a feedback loop where it tries different solutions and checks whether it has given the correct answer. As long as the AI ​​agent can check the answer, it can conduct a trial-and-error loop until it finds a good strategy.

However, there is always a balance between automating decision-making and simply helping and informing people. Analytical artificial intelligence methods, such as systems that aid predict possible decision outcomes, are not agentic in nature, but provide a wealth of information for human decision-makers. In cases that are high or safety-critical, such as medical, security, high-level business policy, etc., the technology may not be ready for AI to completely automate these processes, or we may not even be comfortable with it.

Q: Are there risks we should be aware of when using AI agents?

AND: One massive area of ​​risk comes from the fact that it is often very effortless to hire agents to do certain types of work for you. With coding agents, you can “vibe code” and simply have an agent create the code for you so you don’t have to do the demanding work yourself. There’s a massive risk that because it’s so elementary, people won’t put enough effort into making sure it’s doing it right. Errors will appear, private data will be leaked – this is already happening.

Agents are not perfect in the sense that they can make mistakes because they are not well trained and do not know what to do. But even if they are very competent, if a human does not operate them appropriately or gives instructions that are too vague, the AI ​​agent may make a mistake because the human made a mistake. I think if people are less committed to thinking about all the consequences, we will be more susceptible to making these mistakes.

An additional aspect is the risk of losing qualifications. It’s unclear how far this will go, but as we rely on agents to do homework, program, and perform math, we may lose the ability to do it ourselves, and we may lose that ability too soon because the technology is not yet ready to fully automate these processes.

Q: What does the future hold for agentic artificial intelligence?

But then again, maybe an exceptionally good coding model could act as a puppeteer for communicating with sensors, actuators, and web APIs? Perhaps once you have a superintelligent reasoning system that understands math, language, and code, you can give it a camera and a keyboard and it will figure out what to do in the spatial domain. Will the next wave of AI just be Claude with sensors, actuators and tools, or will it be something built from the ground up in a novel way? This is the most essential question that many people working in the field of artificial intelligence are struggling with today.

The post Questions and answers: What is agentic artificial intelligence today and what do we want it to be? appeared first on AI SCKOOL.

The MIT Music Technology and Computing (MTC) Graduate Program. — launched in fall 2024 as part of a collaboration between Section of Music and Theater Art In School of Humanities, Arts and Social Sciences (SHASS) i Engineering School (SoE) – On May 13, MIT presented its inaugural Music Technology Research Showcase. The event took place in the Thomas Tull Concert Hall in the Edward and Joyce Linde Music Building and featured diverse and captivating research presentations and musical performances.

The ceremonial occasion was attended by the first five MTC participants (all of whom were previously MIT students), as well as several graduate students and faculty. Each scholar presented inspiring examples of inventive engineering that reflected the broader and growing music technology scene at MIT.

The 90-minute event showcased a wide range of research projects, including real-time visualization of what an AI co-improvising agent would play on the piano; sound and art installation based on cacophonous network communication; a hip-hop dance club in which music is created from dance; and using electroencephalogram (EEG) signals to identify the musical melodies our brains imagine.

“A new space of exploration and insights”

The presentation, a combination of a technical presentation and a live performance, began with remarks from SHASS Dean and Professor of Philosophy Agustín Rayo, SOE Dean and Professor of Chemical Engineering Paula Hammond, and MTC Director and Professor of Musical Practice Eran Egozy.

Rayo began, “The goal of this program is simple – for MIT to lead the world in the theory and application of music technology,” adding, “it’s not just about making music with technology; it’s also about working across disciplines to help better shape the future of expression in an AI-powered world, all while reflecting MIT at its best.”

Rayo noted that the graduate program was made possible in part by the opening of the Edward and Joyce Linde Music Building in 2025, which added fresh classrooms, studios, rehearsal spaces and a dedicated music technology laboratory. He also thanked the MIT Schwarzman College of Computing for its support of the graduate program.

Hammond continued: “As those in this room already know, music and engineering have some common roots. Both are based on mathematical precision and draw on specific structures, rhythms, and frequencies. Both require hard work and technical knowledge combined with inspiration and imagination to create something completely new. Given these parallels, it is no surprise that so many of MIT’s faculty, students, and staff are also accomplished musicians and artists.”

She continued, “Our music program is a gem. Only at MIT have we been able to bring together the best technologists and the best musicians to create unique opportunities for collaboration. Here we have brought together faculty and students who identify strongly with both music and engineering to create a new space of exploration and insight. This is a powerful example of the collaborative culture that defines the Institute.”

Egozy called the event “a harmonious hybrid of a concert and a symposium” and recalls, “it’s a bit astonishing what our students have accomplished in just one short and fast year. While we initially debated the trade-offs between a one-year and two-year graduate program, I think this cohort has really shown us that we can make huge strides in our learning and research skills in a concentrated period of time.”

Student research on display

One such student is Claire Southard ’25, SM ’26, who developed a machine learning model to identify notes hidden in EEG signals.

Southard explains: “Every year, musicians are diagnosed with movement disorders such as Parkinson’s disease and dystonia, or experience injuries that prevent them from controlling their hands and bodies in the way needed to play their instruments. Because of this, too many musicians are forced to stop doing what they love. In my work, I am exploring one strategy to help such musicians perform again by translating the music they are trying to play directly into their brain activity – completely bypassing the need for motor control. To this end, I have trained learning models machine to predict what music a person imagines based on brain activity measured by EEG, and many of the predicted songs turned out to be recognizable representations of the user’s imagination. By designing a system that allows musicians to create music regardless of their physical abilities, I hope this work will help bring reality closer to a more accessible future of music performance.

Before joining MTC, Southard was initially unaware of the breadth, scope and magnitude of what the program could offer to further develop and pursue her interests. “MIT’s music technology and computing graduate program taught me a lot about the possibilities at the intersection of STEM and the arts,” he says. “When I first started the program, I honestly wasn’t sure what counted as ‘music technology.’ Through classes, research, and conversations with faculty, guest speakers, and colleagues, I learned that this field is much broader and more fascinating than I could have previously imagined.”

He continues, “Coming from a neuro-computer science background, many of my undergraduate projects were completed entirely on devices. However, this program allowed me to gain more hands-on experience, from conducting audio recordings to building electronic musical instruments from scratch.”

Fellow MTC alumnus and 2026 SHASS Graduate Ceremony speaker, Mariano Salcedo ’25, SM ’26, demonstrated a custom web application that allows anyone to create unique, emergent visual effects powered by real-time music streaming. To achieve this effect, Salcedo has built algorithms that leverage the complex visual behavior of self-organizing systems as a means to achieve an aesthetically synergistic goal.

In his graduation speech, Salcedo expressed his gratitude and admiration for the passionate people he met not only at MTC, but at MIT in general. In an appropriately compassionate way, he expressed his empathetic opinion: “I think in times like this, the call from us is to be a leader in human-centered technology, which means that we not only ask what we can build, but we also ask who will be impacted and who will not be affected? Who will benefit from it?”

Music technology thrives at MIT

Associate Professor Anna Huang SM ’08 MTA and the Faculty of Electrical Engineering and Computer Science (EECS, to SCC), graduate of the Faculty MIT Media Laband one of the world’s leading researchers on collaborative human-AI music-making, echoed Southard and Salcedo’s views in her keynote presentation titled “In Search of Resonance in Human-AI Interaction.” Her speech was fascinating and intimately conversational. Her speech emphasized the importance of centering the human musician in everything that happens with artificial intelligence, while making efforts to include all the music in the world in the discourse at every opportunity.

Huang reflected that many of her family members were in the audience: “I have the honor of working at both MIT Music and EECS – in an interdisciplinary, shared space. What does it mean to build music technology in this context? We are surrounded by incredibly talented musicians, so we take a co-design approach: we collaborate with these musicians, go into the studio and try something every week. And the technology grows with the artistic process. We are always trying to push both of these things forward, and it is always on on the edge. This is the place where I feel most at home.

Huang also explained how this internship sets the stage for a fresh music technology major, which she will co-taught in the fall with recently appointed theater arts professor Grisha Coleman. Class 21M.369/569 (Tuning Attention: Inventive Practices in Motion, Sound, and Artificial Intelligence) proposes that the study of sound and motion practices can inform how we build and imagine computational systems, focusing particularly on our relationship with artificial intelligence. It will introduce students to a range of improvisational and somatic musical practices using motion capture technologies, critical interaction design, generative modeling, and interpretation and learning algorithms through human feedback.

Overall, the future is glowing for MIT’s music technology and computing graduate program. Egozy says MTC has admitted 10 master’s students out of more than 100 applicants for the 2026-2027 academic year. Unlike this year’s class, next year’s students will include not only recent MIT undergraduate graduates, but also fresh faces on campus.

“Expanding the pool to include graduates from other schools and institutions will bring an extraordinary wealth of perspectives and experiences to the program. Additionally, all three faculty members shared by MTA and EECS—including Mark Rau, Paris Smaragdis SM ’97, PhD ’01, and Huang—invite new graduate students in music technology to their labs within EECS,” says Egozy.

In fulfilling its mission, MTC proves to be a lively, multidisciplinary program that attracts many students from diverse backgrounds with a variety of career goals.

“Despite their diversity, all of our students have one thing in common,” says Egozy, “not only a shared love of music, but also a deep desire to pursue this passion through technology in a very warm and humane way.”

The post The inaugural music technology research showcase celebrates the work of the first students of the fresh graduate program appeared first on AI SCKOOL.

Imagine that you will be working in a warehouse or office in the near future and you will be asked to support a up-to-date intern learn the basics of his job. The catch: it’s a robot. To teach them this, you can play a game of “show and tell”, which is physically showing you how to do something in several different ways while explaining what you are doing.

Let’s say you asked a robot to put coffee on your desk without disturbing you during a Zoom call. You prefer that the robot does not get too close to you and the laptop so as not to disrupt the meeting. To enable this behavior, the robot must be trained on data that clearly shows the execution of the entire task. Computer scientists have tried to explain manipulation tasks to robots by recording many physical demonstrations or writing lengthy instructions. However, if you don’t have both, the machine will likely misunderstand what it needs to do.

Showing and telling all this is laborious for humans, so researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) automated the robot’s training process, automatically explaining instructions and using almost five times less demo data. Their Masked Inverse Reinforcement Learning (Masked IRL) approach uses the Gigantic Language Model (LLM) to develop ambiguous prompts based on data collected from the user demo. The next LLM then narrows down the details the algorithm should include in its motion plan so the robot can safely perform work in homes, offices and factories.

“Our approach can be useful when a human is interacting with a robot but doesn’t want to describe all the details of the task,” says MIT graduate student and CSAIL researcher Minyoung Hwang, who is the lead author paper project presentation. “We minimize human effort by enabling machines to get to the heart of what users really want.”

According to Hwang, Masked IRL can support robots safely maneuver in places where there are elements that a human might not describe in a tooltip but are nevertheless crucial. For example, a machine grabbing a snack from the kitchen may not know how to avoid colliding with a laptop. Similarly, a factory robot placing products into different boxes must carefully navigate the shelves.

To learn up-to-date tasks in such situations, Masked IRL uses the robot’s sensors to capture information about its surroundings. These components also record every movement as part of kinesthetic demonstration, a training approach in which a human physically moves the robot to perform a specific action. It’s a bit like being a physical therapist operating a machine and bending your joints in a specific direction to show the robot how to grab, move and place objects.

The MIT system then calls on LLM to compare this sequence of movements (called a trajectory) with the shortest possible path. The model also clarifies what may be unclear in the prompt by turning a request like “stay close” into “stay close to the table surface.” Using trajectory comparison and explained cues, the LLM begins to understand why the movements he has been trained in are vital to the task.

The second LLM then evaluates details of the environment, such as the location of obstacles and the shape of the robot’s target object. During this process, it “masks” (in other words, ignores) items that it considers irrelevant to the task at hand, giving each of them a rating of “1” (vital) or “0” (not very much). For example, whether or not the user was leaning on a table during the demonstration will have a value of “0”, meaning it doesn’t matter. Any detail that is considered a “1” is taken into account by the algorithm in the final action plan.

These masks gave Masked IRL a key advantage over comparable baselines in both 3D and real-world demonstrations by teaching the robot which information to prioritize. Thanks to the researchers’ system, both virtual and real robots were able to skillfully maneuver objects around obstacles, for example moving a coffee mug around a laptop to different places on the table. In these tasks, masked IRL correctly identified users’ preferences that they did not explicitly express in their prompts as much as 15 percent more often than comparable baselines.

During simulation experiments, CSAIL researchers also found that Masked IRL learned quickly. Understanding how to move the cup required fewer demonstrations than the base version. They also found that the robots performed better when the LLM explained the instructions, rather than forcing the machine to follow unclear requests.

This more focused approach also translated well to a real robotic arm executing commands the system didn’t see in the training phase. After being trained in 50 kinesthetic demonstrations, the robot carefully moved the cup toward the human, avoiding colliding with the user’s computer – an obstacle it learned to avoid by developing a more general “stay away” request. He also wiped the table while “staying close” and handed the user a bag of chips while “staying away” from both the man and the table.

Masked IRL senses and clarifies what users leave unsaid, but can soon “see” it too. CSAIL scientists plan to make their approach more energetic by equipping it with cameras that enable the robot to take photos of its surroundings. It could then highlight and focus on specific items nearby. For example, if you ask the machine to pick up a toy, it may see some bananas nearby and ignore them before moving on to the target object.

Hwang wrote the paper with three CSAIL colleagues: graduate student Alexandra Forsey-Smerek ’20, SM ’22; postdoc Nathaniel Dennler; and MIT assistant professor Andreea Bobu, who is a member of the Department of Aeronautics and Astronautics and CSAIL. Their work was supported in part by the Tata Group through an MIT Generative AI Impact Consortium Award and the Department of Defense. They will present the project at the 2026 IEEE International Conference on Robotics and Automation in June.

The post LLMs support robots understand unclear instructions and focus on key details appeared first on AI SCKOOL.

Agentic workflows are artificial intelligence-based software systems that combine multiple models and external tools to solve convoluted tasks, such as analyzing video and answering questions about it.

However, the way these highly fragmented systems are designed and implemented often creates inefficiencies that can lead to wasted processing power, energy and costs.

To improve efficiency, researchers from MIT and Microsoft have developed an clever system that streamlines the process of designing agentic workflows and automatically optimizes how they are implemented.

With this recent method, a developer can describe what he or she expects from an agentic workflow in plain language, without having to specify all the details of the application in advance.

The system automatically determines the best models and tools to operate, as well as the ideal hardware configuration and computational resource allocation when the workflow is executed by the cloud provider.

It adjusts these configurations on the fly based on each user’s priorities, such as minimizing costs or maximizing speed.

When tested on several agent-based workloads, this recent system reduced the number of compute units needed for deployment, significantly reducing power requirements and costs compared to time-honored approaches, without compromising performance.

“Agent workflows are becoming very complex and are quickly becoming the backbone of cloud providers’ businesses. Energy consumption is a huge issue, so we have to be very careful about the efficiency of these workflows. It’s very easy to overspend on resources, wasting energy and money. Enabling the cloud provider to intelligently make these workflows more resource-optimal is a win-win for all parties involved,” says Gohar Chaudhry, an electrical engineering and computer science (EECS) graduate student and lead author of the book article about this system.

He is joined in this article by Adam Belay, EECS associate professor and member of MIT’s Computer Science and Artificial Intelligence Laboratory; senior author Ricardo Bianchini, technical fellow and corporate vice president at Microsoft Azure; and others at Microsoft Azure. The paper will be presented at the USENIX Symposium on Operating System Design and Implementation.

Configuration puzzle

An agentic workflow is a system composed of several autonomous AI agents that collectively operate different models and tools, such as databases or Python programs, to dynamically perform a multi-step task, such as data processing or code generation.

These workflows can serve as behind-the-scenes processes that power user-facing applications.

Typically, developers must code all technical choices in advance. They must determine what AI agents, models and tools to operate and in what order to operate them. They also need to determine the hardware that supports the workflow and how to balance tradeoffs such as speed and cost.

This is particularly challenging because agentic workflows combine multiple black box models and a variety of tools, each with their own configuration options that may be offered by different companies.

If a recent AI model is released that improves the accuracy or performance of an application, the developer will have to start from scratch to implement it.

“Even if you want to do it all manually, it’s unlikely you’ll be able to optimally configure the workflow because the space of possible configurations is so large,” says Chaudhry.

Additionally, the cloud data center that deploys the application to customers lacks visibility into the workflow to allocate hardware resources in the most capable manner when a user requests them.

With this recent system called Murakkab (an Urdu word meaning putting things together), researchers sought to optimize the entire agent workflow process.

Active decision making

First, Murakkab enables developers to create an agentic workflow by describing their intentions for the application in high-level terms, rather than detailing how the multiple components of that workflow should be connected.

For example, a developer might describe a video Q&A application that extracts key frames, generates a transcript, and then answers users’ questions about the video.

“There are many ways to do this, and all these different models and tools influence how quickly the application can complete the task,” he says.

Murakkab uses straightforward developer specifications and automatically identifies the best existing models and tools that can be applied to your workflow.

It also determines which components must run sequentially and which can be run in parallel to improve performance.

“The platform makes configuration decisions dynamically over time, so if a new model or graphics accelerator comes out tomorrow, the developer doesn’t have to worry about it,” he says.

When a cloud service provider deploys this application to a customer, Murakkab optimizes the workflow by configuring its components to meet user constraints, such as prioritizing accuracy while meeting latency requirements.

Adaptively identifies ideal hardware allocation and deployment schedules to maximize performance in real time, then generates a workflow that is ready for execution by the cloud provider.

“Our system also gives cloud service providers visibility across multiple workloads so they can divide compute resources in the most efficient way while meeting user constraints,” he says.

Tested on various agentic video Q&A and code generation workflows, Murakkab met user requirements by using only about 35 percent of the computations required by other methods. It used only about 27 percent more energy for less than 25 percent of the cost.

The animated nature of Murakkab also allows users to balance trade-offs. In one case, the system reduced energy consumption in an agent-based workflow by more than an order of magnitude, with only about a 2 percent decrease in accuracy for the client.

The system also managed to identify an unexpectedly ideal configuration for the video frame selection model, optimizing video question and answer performance. Chaudhry says this type of optimization would be almost impossible for a programmer to do manually.

Next, researchers plan to expand their system to more convoluted workflows and larger computing clusters, while also exploring opportunities to optimize recent agent applications.

“There is a lot of potential to make these workflows more resource-optimal, so they use much less energy, but we need to think about it at the scale of the major cloud platforms,” Chaudhry says.

This research was supported in part by the Semiconductor Research Corporation and the United States Defense Advanced Research Projects Agency.

The post Improving the speed and energy efficiency of AI agents appeared first on AI SCKOOL.