When AI agents enter into real implementation, organizations are under pressure to define where they belong, how to build them effectively and how to operate them on a huge scale. In VentureBeat’s Transform 2025Technology leaders gathered to talk about how they transform their activities with agents: Joanne Chen, a general partner at the Foundation; Shailelesh Nalawadi, Vice President of Project Management with Sendbird; Thys Waanders, SVP AI transformation in Cognigy; and Shawn Malhotra, CTO, Rocket.
Some of the best cases of using AI AI
“The initial attraction of any of these implementations for AI agents is usually saving human capital – mathematics is quite simple,” said Nalawadi. “However, this emphasizes the transformation ability that you receive with AI agents.”
At Rocket, AI agents proved to be powerful tools in increasing websites.
“We discovered that thanks to our experience based on agents, conversational experience on the website customers will convert this channel three times more often,” said Malhotra.
But it’s just scratching the surface. For example, a rocket engineer built an agent in just two days to automate a highly specialized task: calculating taxes on transferring during mortgage insurance.
“These two days of effort saved us a million dollars a year,” said Malhotra. “In 2024, we saved over a million hours of team members, mainly outside our AI solutions. This is not only saving expenses. This also allows members of our team to focus their time on people who are often the largest financial transaction of their lives.”
Agents generally recharge individual team members. This million hours of saved is not quite someone replicated many times. These are fractions of the work that employees do not like to do or have not added value to the client. And this one saved a million hours gives the rocket the ability to deal with more activities.
“Some members of our team were able to cope last year by 50% more customers than a year ago,” Malhotra added. “This means that we can have a higher bandwidth, increase business and we see higher conversion rates again because they spend time understanding the client’s needs compared to the much more Rote that AI can now do.”
Agent complexity
“Part of the journey for our engineering teams is a departure from the way of thinking of software engineering – write once and test it, and he works and gives the same answer 1000 times – to a more probabilistic approach in which you ask about the same with LLM and it gives other answers through some probability,” said Nalawadi. “Many of them bring people. Not only software engineers, but product managers and UX designers.”
Waanders said LLM had a long way. If they built something 18 months or two years ago, they really had to choose the right model or the agent would not work as expected. Now, he says, we are now at the stage where most mainstream models behave very well. They are more predictable. But today the challenge is to combine models, provide reaction, organize appropriate models in the right sequence and weave the relevant data.
“We have customers who push tens of millions of conversations a year,” said Waanders. “If you automate, let’s say, 30 million conversations during the year, how is this scale in the world of LLM? That’s all we had to discover, uncomplicated things, even from obtaining the availability of the model with cloud suppliers. Having, for example, a sufficient number of amounts with the chatgpt model. There are all teachings that we had to go, as well as our clients.
Malhotra said that the LLM -organizing layer would organize a network of agents. The conversation experience has a network of agents under the hood, and the orchestrator decides to which the agent will risk a request from the available ones.
“If you play forward and think about hundreds or thousands of agents who are capable of different things, you have really interesting technical problems,” he said. “This becomes a bigger problem because the delay and time. This agent’s routing will be a very interesting problem to solve in the coming years.”
Using the accounts of suppliers
Until then, the first step for most agentic AI starting companies was building internal, because specialized tools have not yet existed. But you cannot differentiate and create values by building the general LLM infrastructure or AI infrastructure, and you need specialist specialist knowledge to go beyond the initial compilation and debate, iterate and improve what has been built, as well as maintain infrastructure.
“We often find the most successful conversations with potential customers, who are usually someone who has already built something internal,” said Nalawadi. “They quickly realize that reaching 1.0 is fine, but as the world evolutions and in accordance with the evolution of infrastructure, and because they have to mention technology for something new, they are not able to organize all these things.”
Preparation for agentic ai complexity
Theoretically, Agentic AI will only grow with complexity – the number of agents in the organization will augment and start learning from themselves, and the number of cases of apply will explode. How can organizations prepare for a challenge?
“This means that controls and balances in your system are becoming more stressed,” said Malhotra. “In the case of something that has the regulatory process, you have a man in the loop to make sure someone is signing on it. In the case of critical internal processes or access to data, do you have observation? Do you have the right warning and monitoring, so that something goes wrong, you know that it is bad? It doubles in detection, understanding where you need a man in a loop, and then trust it. do it.
How can you be sure that the AI agent will behave reliably as evolution?
“This part is really difficult if you didn’t think about it at the beginning,” said Nalawadi. “A short answer is: before you start building it, you should have an assessment infrastructure. Make sure you have a strict environment in which you know what you look like from the AI agent and that you have this set of tests. Refer to it when you introduce improvements. A very simplified way of thinking about assessment is that these are unit tests for your aggressive system.”
The problem is that it is indefinite, added Waanders. Individual testing is crucial, but the biggest challenge is that you do not know what you do not know – what incorrect behavior an agent can display, how he can react in any situation.
“You can only learn this by simulating large -scale conversations, pushing them in thousands of different scenarios, and then analyzing how it persists and how he reacts,” Waanders said.