When Products Act on Our Behalf: Designing for Trust in AI

A user asks an AI assistant to “make sure everyone has the latest document”.

The assistant finds a file, updates access permissions and sends it to a distribution list.

The task is complete. But was that what the user meant?

Perhaps they wanted a list of people still using an older version. Perhaps the document needed approval before distribution. Perhaps “everyone” meant the project team, excluding external suppliers.

The system may have executed its chosen workflow perfectly while exceeding the authority the user intended to give it.

This is where designing for trust in AI becomes a different product challenge.

We are asking people to trust software with interpretation, judgement and action. A clear interface and a reassuring tone cannot resolve the questions that follow: What have I authorised? What will this system decide for itself? What happens if it gets my intention wrong?

Trust becomes a question of authority

Consider three requests:

“Find an available meeting time.”

“Suggest the best meeting time.”

“Choose a time and send the invitations.”

They concern the same task, but each grants a different degree of authority. Finding requires access to information. Suggesting introduces judgement. Sending creates consequences for other people.

A product can obscure these differences behind a single conversational interface.

For a design team, the important question is what the user has actually delegated. Access to a calendar does not automatically mean permission to rearrange it. Access to a customer database does not imply authority to contact every customer.

We need to design the boundaries of delegation as carefully as we design the interaction itself.

The goal is appropriate trust

“Make users trust the AI” is an incomplete design brief.

Someone can trust a system too much. They can accept an unsupported answer because it is written confidently, or authorise an action because the product makes the decision look routine.

That experience may feel effortless while leaving them exposed to consequences they do not understand.

The goal should be appropriate trust: helping people judge when to rely on the product, when to inspect its work and when to take over.

Communicating what a system can do and how well it can do it. These are separate design responsibilities.

A feature that can summarise a document has not necessarily established that it can reliably identify every exception in that document. Our interfaces should preserve that distinction.

Make uncertainty useful

“AI can make mistakes” tells users very little about the decision in front of them.

What might be wrong? What information is missing? What should they check?

Imagine a product recommending a supplier. It might explain that the recommendation uses price and delivery history, but excludes contractual restrictions because those records were unavailable.

Now the user knows what the recommendation covers and what remains unresolved.

The same principle applies to action. An assistant preparing to distribute a document could identify that it found two versions and cannot establish which one is approved.

A useful admission of uncertainty changes what happens next. It may prompt clarification, narrow the action or direct attention to a specific check.

We should also question confidence displays. A percentage without a clear meaning or sound basis can add an appearance of precision without helping someone decide.

Make delegation explicit without making it exhausting

Asking for approval at every step can make an assistant burdensome. Asking too rarely can allow it to exceed the user’s intentions.

The design challenge is to establish useful boundaries.

A user might authorise an assistant to organise incoming documents automatically, while requiring approval before anything is shared externally. They might permit routine purchases within a budget, with exceptions for new suppliers.

Those boundaries should reflect the consequences, ambiguity and reversibility of the work.

An agent should pause when the next action falls outside its authority, when essential information is missing or when a decision has consequences the user has not accepted.

Anthropic’s work on trustworthy agents identifies misinterpreted intent and unintended actions as important risks as systems operate with less human oversight. For product teams, that makes deciding when to stop part of the product’s competence.

The interface should help users understand the agreement they are making: what the assistant may do independently, what it must bring back and how that agreement can be changed.

Make verification worth the effort

We often tell people to review AI output. But what does that review require?

If checking a summary means rereading the entire document, some of the work has simply moved. If checking an automated workflow requires opening five systems, the apparent time saving may disappear.

Verification is part of the experience.

A contract summary could connect material claims to the relevant clauses. A recommendation could expose the assumptions that would change the decision. A completed workflow could show which records changed, which actions failed and which items remain untouched.

Sources need to support the claims beside them. Activity records need to distinguish an attempted action from a completed one.

For product leaders, the value calculation should include the time people spend checking, correcting and supervising the work.

A fast answer is only one part of a useful outcome.

Design for intervention and consequences

Control becomes more demanding when work continues beyond a single response.

Can the user see what is happening? Can they interrupt it? What happens to actions already completed?

A stop button should make its scope clear. It may prevent further messages from being sent while leaving earlier messages delivered. A reversal may restore a database value while being unable to undo someone’s decision based on that value.

The recovery experience needs to explain those limits.

It should also establish ownership. If an assistant makes an incorrect change, who helps resolve it? Can a user reach someone with enough context to act, or must they reconstruct the entire incident?

“The AI did it” offers the customer no useful path forward. The product organisation still owns the experience it has provided.

Let users correct what the product believes about them

Personalisation creates another form of delegation: allowing a product to use past information to interpret future requests.

That can be useful, but remembered context can become outdated or travel beyond its intended scope.
A preference for cheaper flights does not necessarily apply to an urgent business trip. A decision made for one project should not silently become a rule for every project.

Users should be able to inspect, correct and remove information that materially shapes the product’s behaviour. They should also understand when a preference has become an instruction governing future actions.

Memory should help the product understand the person without trapping them in an earlier version of themselves.

Measure whether reliance is justified

Satisfaction and repeat usage cannot tell us whether people are delegating appropriately.

A user may be delighted because they have not noticed an error. Another may be dissatisfied because the product correctly paused at a consequential ambiguity.

Research should examine whether people understand the system’s authority, recognise unsupported conclusions and intervene when necessary.

Evaluation should also test the product’s behaviour. Does it stay within agreed permissions? Does it stop when essential context is missing? Does it report partial completion accurately? Can users recover from its mistakes?

Trust is a shared responsibility across design, product, engineering and operations. The interface can explain the boundaries, but the system must honour them.

As products take on more responsibility, the quality of the experience depends increasingly on how well they manage the authority we give them.

The question for product teams is therefore: what has our product earned the right to do on someone’s behalf?