Human-in-the-loop: how to keep control when AI answers your customers

TL;DR: Human-in-the-loop means a person stays in the decision path when AI talks to your customers. There are three workable levels: approve every reply before it sends, let the AI reply but hand off flagged conversations, or let it reply freely on a narrow scope and review a sample. Whichever level you choose, control comes from four mechanisms: a written scope, gated actions, a real handoff into a shared inbox, and an audit log of everything said and done. Start strict, loosen by evidence.

What human-in-the-loop actually means

Human-in-the-loop is a design choice that keeps a person in the decision path when an AI assistant answers customers. It is not a feature you switch on; it is a set of rules about what the assistant may say on its own, what it must ask permission for, and what it must hand to a person. Done well, it lets a UAE business automate routine WhatsApp and web enquiries without ever losing the ability to say "we checked that before it went out".

The reason this matters is that a language model is not deterministic. It can be right on a question a thousand times and wrong on the thousand-and-first, with the same confident tone. No amount of prompt-writing removes that property. What removes the risk is the system around the model: what it is grounded in, what it is allowed to do, when it must stop, and what gets recorded. This article is about that system.

The three levels of control

Most businesses picture one of two extremes: a chatbot that answers everything alone, or a human reading every message. The useful design space sits between them, and it is worth naming the levels explicitly because you will move between them over time.

Level	What the AI does	What the human does	Fits when
Draft and approve	Reads the message, drafts a reply, waits	Edits or approves every reply before it sends	A new assistant, a sensitive sector, or a team that wants to build trust first
Auto-reply with handoff	Replies on its own inside a defined scope; hands off anything outside it	Takes over flagged conversations in the inbox; reviews handoffs daily	Routine enquiries such as opening hours, availability, prices from a list, booking
Auto-reply with sampling	Replies freely on a narrow, well-tested scope	Reviews a sample of conversations and all complaints	Mature assistants on low-risk topics with months of evidence

The mistake is not choosing the wrong level. It is choosing a level and never revisiting it. A well-run assistant starts at draft-and-approve, collects the evidence of which questions it handles well, releases those to auto-reply, and keeps everything else with people. The evidence is the log, which is why the log is not optional.

How do you define what the AI is allowed to handle?

Scope is the first control and the one most often skipped. "Answer customer questions" is not a scope. A scope is a list: the topics the assistant may answer, the topics it must hand off, and the sources it may answer from.

• Allowed topics. Opening hours, locations, services offered, prices from an approved list, availability from a calendar, order status from a system, how to book, what to bring. Each one should map to a document or a system the assistant reads from.
• Handoff topics. Complaints, refunds, anything involving money beyond quoting a listed price, medical or legal questions, negotiation, requests to speak to a person, and anything the customer says is urgent. In a clinic, anything clinical. In a brokerage, anything about whether a price is fair.
• Sources. The assistant answers from your documents and systems, not from general knowledge. If the answer is not in the sources, the correct behaviour is "I will get someone to confirm that", not an educated guess. This is the grounding step, and it removes most invented answers before they happen.

Write the scope down in plain language, in Arabic and English if your customers write in both, and treat it as a document the team owns. When the assistant handles something it should not have, the fix is usually a line in the scope, not a rewrite of the system.

How do you gate actions with consequences?

Answering a question is low-risk: the worst case is a wrong sentence a person can correct. Taking an action is different. Booking an appointment, creating a CRM record, sending a quotation, changing an order, issuing a refund: each of these has a consequence in the real world, and the model should not be the last thing that decides.

The engineering pattern is simple. The model proposes an action as structured data. Deterministic code checks it: is the slot actually free, is the price from the approved list, is the refund within the limit, is the customer who they say they are. Only then does the action run. Above a threshold you set, the action waits for a person to approve it in the inbox. Below the threshold, it runs and is logged. The model never calls the booking system directly; it asks a function that validates first.

A practical way to think about it is least privilege. Reading and drafting need no permission. Writing to a calendar or CRM needs validation. Moving money or changing a customer's contract needs a person. If a vendor cannot describe which actions their assistant can take and what checks sit in front of each, that is a gap to close before launch.

What does a real handoff look like?

Handoff is where most assistants fail in practice, not because the rule is missing but because the receiving end is not designed. "Escalate to a human" only works if a human can see the conversation, knows why it was escalated, and can reply in the same channel without the customer noticing a seam.

The inbox

Every conversation, automated or not, should live in one shared inbox that staff can open on a phone or desktop. On WhatsApp, this is the inbox that sits on top of the WhatsApp Business API, whether from a provider, a CRM with a WhatsApp channel, or one built for you. When the assistant hands off, the conversation is assigned to a person or a team, marked with the reason, and the assistant stops replying until a person releases it back.

The handoff message

The customer should be told what is happening in their language: "I am passing this to a colleague who will reply here shortly." Outside working hours, the message should say when to expect a reply. The assistant should not keep talking after handoff, and it should not pretend the colleague has arrived before they have.

The notification

Someone has to know. The handoff should notify the assigned person on WhatsApp, email or the inbox itself, with the summary and the reason. If nobody picks it up within a period you choose, it escalates to a second person. This timer is the difference between a handoff and a black hole.

The return

When the person has resolved the matter, they release the conversation back to the assistant or close it. Either way, the log records who said what, and the next time this customer writes in, the assistant can see the history.

What goes in the audit log?

The log is what makes the other controls real. Without it you cannot investigate a complaint, measure the assistant, or safely widen its scope. At minimum it should contain:

• Every inbound and outbound message, with timestamps and the channel.
• For each automated reply, which documents or system data the answer was based on. This lets you trace a wrong answer to a wrong or missing source and fix it at the source.
• Whether each reply was auto-sent, approved by a person, or edited before sending, and by whom.
• Every action the assistant proposed, whether it passed validation, whether it ran, and who approved it if approval was required.
• Every handoff, with the trigger and the time it took a person to respond.

Review the log on a schedule. In the first weeks, daily: read every handoff and a sample of auto-replies. Later, weekly. The output of each review is a short list of changes: a document to correct, a topic to add to the handoff list, a question that is now safe to auto-reply. This loop is what makes the assistant better, and it is the part that self-service chatbot tools rarely support.

A checklist before you let the AI reply on its own

1. The scope is written down, with allowed topics, handoff topics, and the sources for each allowed topic.
2. The assistant answers only from your documents and systems, and says so when the answer is not there.
3. Every action with a consequence goes through a deterministic check, and the ones above your threshold wait for a person.
4. There is one shared inbox, and staff have used it in a drill.
5. Handoff notifies a named person, with a timer and a second person if the first does not respond.
6. The handoff message exists in Arabic and English and tells the customer what to expect.
7. The audit log captures messages, sources, approvals, actions and handoffs.
8. Someone owns the review and the scope document, by name.
9. There is a kill switch: one action that puts the assistant back into draft-and-approve mode, or pauses it entirely, that a non-engineer can use.

How Soluvide approaches this

We build every customer-facing assistant with human-in-the-loop as the default, not an option. New assistants launch in draft-and-approve mode inside a shared inbox, with a written scope, gated actions and a full log, and they earn auto-reply topic by topic from the evidence. Handoff and escalation rules are part of what we deliver under AI chatbots and agents, and the grounding that keeps answers tied to your own documents is the same private knowledge layer we describe under private knowledge bases and RAG.

For businesses whose customers write in Gulf Arabic as often as English, the scope, handoff messages and review all need to work in both, which is why our Arabic and English support agents are tested by people who read both. We scope the assistant first and send a fixed-fee proposal after a free conversation.

If you already have an assistant and are not sure what it is allowed to do or who sees its mistakes, that is worth a conversation on its own. Message us on WhatsApp and describe how customer messages reach your team today.