Opinion
AI that usually works is not good enough for business
"Without trust," writes Arato CEO Shahar Erez, "customers return to humans and companies withhold the access that makes AI valuable."
In just a few months, the enterprise AI conversation has raced through several milestones: building AI, putting it into production and showing that it works with real users. I now see companies with systems they believe work, but without enough evidence to trust them in critical business processes, let alone measure their true impact. Getting AI into production was an achievement. Getting customers to rely on it repeatedly is the test that will decide whether adoption continues.
People who use ChatGPT or Claude every day still abandon the AI agent their bank, insurer or airline offers. The difference is the stakes. A weak answer from a general-purpose chatbot is often easy to work around. A wrong answer about a refund, claim or booking leaves the customer paying for the mistake. Research shows that two or three disappointing experiences are enough for a user to never return. The question for enterprises is no longer whether AI can do the job. It's whether anyone trusts it enough to let it.
Trust is the price of access, and access is where the value is. We give a human personal assistant access to our calendar, accounts and private information because we trust how that person will use it. AI agents need that confidence too. Completing real tasks requires sensitive data and access to business systems. A company that cannot trust its agent with those resources keeps it answering FAQs, while the work that justified the investment remains with people.
The doubt starts inside the company. Leaders who do not feel in control of their AI's behavior hesitate to put the brand behind it. They hide it from customers, hedge its role or restrict what it can complete. Customers have little reason to use an agent that explains a process and then sends them elsewhere to finish it. Low usage seems to confirm that the AI does not work (or at least not as we expected it to). The company has helped create the result it was afraid of.
A chief AI officer at a bank recently told me that he calls his bank's voice assistant every morning on his way to work, just to see how it is behaving. Usually, it works well. The day before we spoke, however, the AI accepted a check that it should not have. He did not know how often the same thing was happening. His morning calls gave him impressions of performance; they did not tell him whether this was an isolated mistake or a recurring failure. That is a poor basis for deciding whether customers should rely on the system.
AI needs a more useful definition of “working.” The same application can behave differently across users, conversations and repeated attempts. A successful interaction alone says little about how it handles the next customer's language, assumptions or route through the process. Each variation expands the space in which mistakes occur. For a critical business process, the relevant evidence is how reliably customers reach the correct outcome, where the system crosses a boundary and how often it fails.
In one simulation my team ran, an enterprise AI supplied a $400 price and later acknowledged that it had made the number up. A person checking a draft has a chance to catch that error. A customer relying on the quoted price has already been given a reason to doubt the business. Even a fluent, apparently helpful interaction can leave the customer with information the company cannot stand behind.
Testing a few familiar questions will miss much of that behavior. Validation has to follow complete customer journeys, including people who are confused, impatient, vague or who provide information in an unexpected order. Repeating those journeys reveals whether a successful interaction is dependable and whether failures cluster around particular situations. Companies often invest heavily in getting the system live while giving too little attention to producing an excellent experience repeatedly. Customers experience that gap directly, regardless of how impressive the launch was.
An overall accuracy score is a poor substitute for knowing which failures customers encounter. A system that handles routine questions well may still fail precisely where a customer needs it to complete a claim or resolve a payment issue. The acceptable failure rate depends on the task and the cost of getting it wrong. Leadership needs evidence about those failures, their frequency and their consequences to decide what the agent can be trusted to do and set honest expectations with customers, regulators and the board. Access expands as the evidence earns it.
The pressure to move fast is real. Companies see AI as an existential race, and standing still has a cost. But a prompt tweak, model upgrade or new data source can change behavior that was previously reliable. Confidence earned in one release does not automatically survive the next. If customers keep encountering new mistakes, each release asks them to take another chance on a system that has already let them down.
Continuous validation belongs in that release cycle, before and after changes reach production. Representative customer journeys have to be rerun, failures understood and fixes checked for new problems elsewhere in the process. The winners will be the companies that can keep improving their AI at speed and stand behind it in real business processes. Companies that keep asking customers to tolerate another disappointing release will be left with an expensive system that people have stopped using.
Shahar Erez is co-founder and CEO of Arato.
