Opinion
Trust has to be earned and re-earned: How to scale AI autonomy without losing control
"Businesses that keep their agents accountable and continuously checked will earn enough trust from customers and regulators to hand them bigger decisions," writes Oren Winter, Head of R&D Salesforce Israel. "Trust, unlike permission, has to be earned and re-earned, one monitored decision at a time."
AI agents aren’t set and forget. One of the biggest risks in enterprise AI deployment is assuming launch is the finish line and treating testing as a one-off milestone rather than an ongoing discipline. That approach misunderstands how these systems actually work.
Companies that want agents to behave accurately and consistently over time must do five things: keep the model’s thinking separate from its actions, expand autonomy gradually, ground it in how the business really works, set up guardrails and contingency plans, and monitor it continuously after launch.
AI agents are probabilistic. They generate answers based on likelihood that what they are suggesting is correct rather than fixed rules. Tomorrow’s answer might differ from today’s, and over time it could drift into inaccuracy. The data shifts, customer inquiries evolve, and small changes elsewhere in the business can derail an agent’s behavior. For example, an agent that handled refunds well at launch could be quietly getting them wrong six months later. Without monitoring, no one would notice.
Separate thinking from doing
The model is excellent at reading a customer's history, weighing the options, and recommending a next step. But it shouldn't be issuing the refund or changing the account on its own. Businesses need to pair the model with deterministic logic: business software that applies their rules, permissions, and workflows to every action. When an agent proposes a refund, that software checks it against policy and approval limits before any money moves.
Scale autonomy incrementally
The most successful deployments treat autonomy as a dial, not a light switch. A new agent might start with narrow, low-stakes permissions, such as drafting replies for a person to approve. As we continue to monitor its behavior and build higher trust, it can take on small refunds, then larger ones. Equally critical is requiring the agent to identify itself clearly as AI. Customers as well as IT systems in the organization should always know who, or what, they’re dealing with.
Ground the agent in how the business really works
Business logic governs what an agent may do. Grounding shapes what it knows. A general-purpose model wrapped in a few instructions fills gaps with generic assumptions, so it answers similar cases differently each time. An agent grounded in a retailer's actual returns policy, product catalog, and customer history, including exceptions such as extended returns for loyalty members, gives the same answer to its hundredth case as to its first.
Set up guardrails and a contingency plan
Agents must strictly operate under the principle of least privilege, granted only the specific accesses and capabilities required to perform their intended function. Beyond standard operational limits, robust safety guardrails should be put in place to prevent actions that fall outside acceptable parameters. Finally, organizations need a well-defined contingency plan outlining immediate mitigation protocols, human intervention pathways, and fallback mechanisms for when an agent encounters unexpected edge cases or fails.
Monitor continuously
Pre-launch testing shows how an agent handles the scenarios you thought to test. It says nothing about what turns up months later, such as a new product, a policy change, or a surge in a type of query that nobody anticipated. Continuous observability and monitoring means regularly reviewing an agent's live decisions and actions, assessing them against the original standards and goals and logging them for auditability. If its refund decisions start to drift, you catch it before it turns into a wave of complaints and you have the papertrail to troubleshoot and fix the problem.
Trust has to be earned and re-earned.
These guardrails let a business move quickly with confidence. Keeping a human in the loop means people set the limits, review how agents perform, and step in when something looks off, so every decision can be traced, explained, and, if necessary, reversed. Their role changes as an agent earns trust, but it never disappears.
Businesses that keep their agents accountable and continuously checked will earn enough trust from customers and regulators to hand them bigger decisions. Trust, unlike permission, has to be earned and re-earned, one monitored decision at a time.
Oren Winter is Head of R&D at Salesforce Israel.
