Join the event trusted by corporate leaders for almost two decades. VB Transforma connects people building AI Real Enterprise. Learn more
Companies throw AI agents for production – and many of them will fail. But the plaintiff has nothing to do with their AI models.
On the second day VB Transform 2025Industry leaders shared demanding -exhausted lessons in the implementation of AI agents on a enormous scale. Moderated panel by Joanne Chen, COMPANY The capital of the foundation, It included Shawn Malhotra, CTO W Rocket companieswho uses agents on the journey of the ownership of the house from mortgage insurance to clients’ chat; Shailelesh Nalawadi, product head in Birdwhich builds agency experience with customer service for companies in many industries; and Thys Waanders, SVP transformation AI in Cogninigywhose platform automates customer experiences for enormous contact centers of the company.
Their joint discovery: companies that build evaluation and orchestration infrastructure are successful while people spending production with powerful models fail on a enormous scale.
>> See all our transform 2025 coverage HERERoi reality: Apart from a straightforward cutting of costs
The key part of the AI agent engineering for success is to understand the return on investment (ROI). Early implementation of AI agents focused on reducing costs. Although this remains a key element, corporate leaders now report more sophisticated ROI patterns that require various technical architecture.
Cost reduction wins
Malhotra divided the most dramatic example of costs from rocket companies. “We had an engineer [who] Within about two days of work, he was able to build a simple agent to deal with a very niche problem called “transfer tax calculations” in the insurance part. And these two days of effort saved us a million dollars a year, “he said.
In the case of Cogninigy, Waanders noticed that the cost of connection is a key measure. He said that if AI agents are used to automate some of these connections, it is possible to shorten the average service service.
Methods of generating revenues
Saving is one thing; Making more revenues is another. Malhotra announced that his team had recorded conversions: because customers receive answers to their questions faster and have good experience, they convert at higher rates.
Proactive possibilities of revenues
Nalawadi emphasized completely novel revenue options through a proactive range. His team enables proactive customer service, reaching before customers realize that they have a problem.
An example of the delivery of food is perfectly illustrated. “They know when the order is late, and instead of waiting for the customer to get angry and call, they realize that there is a possibility of overtaking,” he said.
Why AI agents are bursting in production
Although there are solid possibilities of roi for enterprises that are implemented by Agentic AI, there are also some challenges in the implementation of production.
Nalawadi identified the basic technical failure: companies are building AI agents without evaluation infrastructure.
“Before you start building it, you should have an evaluation infrastructure,” said Nalawadi. “We were all software engineers. Nobody is implemented in production without starting unit tests. And I think that a very simplified way of thinking about oces is that this is a unit test for the AI agents system.”
Conventional software testing approaches do not work for AI agents. He noticed that you just can’t predict any possible contribution or write comprehensive testing cases in natural language. The Nalawadi team learned this by implementing customer service in retail, providing food and financial services. Standard approach to quality ensuring the omitted cases that appeared in production.
AI AI testing: Up-to-date quality assurance paradigm
Given the complexity of AI tests, what should organizations do? Waanders solved the test problem through simulation.
“We have a function that we release soon, concerns the simulation of potential conversations,” Waanders explained. “So basically AI agents test AI agents.”
Testing is not only testing the quality of the conversation, its enormous -scale behavioral analysis. Can he aid understand how the agent reacts to enraged customers? How does it support many languages? What happens when customers employ slang?
“The biggest challenge is that you don’t know what you don’t know,” Waanders said. “How does he react to everything that someone could come up with? You only learn by simulating conversations, really pushing them in thousands of different scenarios.”
The approach is tested by demographic changes, emotional states and cases of advantage that human teams cannot include comprehensively.
The upcoming explosion of complexity
Current AI agents support individual tasks independently. Enterprise leaders must prepare for another reality: hundreds of agents learn from each other for organization.
Infrastructure implications are huge. When agents provide data and cooperate, the failure modes multiply exponentially. Conventional monitoring systems cannot track these interactions.
Companies must now have an architect to this complexity. The infrastructure of modernization of multi -stage systems costs much more than the correct building from the very beginning.
“If you fall forward in what theoretically possible, there may be hundreds of them in the organization, and maybe they learn from each other,” said Chen. “The number of things that can happen simply explodes. Complexity explodes.”