Why 'I don't know' is the most important sentence in an AI system

The value of an AI answer is determined by what the system refuses to say, not by what it does say.

Most AI systems can't say that they don't know the answer. After all, they've been trained to be as helpful as possible. And that can sometimes have serious consequences.

I was recently confronted with a situation where a customer had asked the chatbot a very specific question. The chatbot, in turn, assured the customer that this was definitely possible. The customer confidently started experimenting with the software, only to eventually discover that the software couldn't do what had been promised at all. They then went back to the chatbot and pointed this out. They simply got the same firm confirmation that it would definitely work. But it couldn't be done at all — the chatbot had simply made its answer up.

We call that a hallucination. The AI agent fabricates its answer because it wants to be helpful. That, after all, is what it was taught during the training phase. Chatbots are often sold on the basis of their resolution rate: the share of questions the AI can answer on its own without involving a human. A figure like that looks good on a dashboard. It just doesn't say whether the answers the chatbot gave were actually correct. A chatbot that answers 95% of questions simply sounds far more impressive than one that answers 'only' 75%.

But the question isn't how many questions has my chatbot answered, but rather: how many customers or users were genuinely helped further by the chatbot? That's a far trickier figure to put on a product page, because then you'd have to honestly admit that your chatbot doesn't solve everything. Anyone who takes 'I don't know' seriously as a possible answer automatically designs a path to a human.

For me it was a condition for Sarrai that the chatbot could weigh in on anything — as long as it had to do with the software. Writing little poems about the quality of your software, or inventing a delivery time that's offered nowhere, that isn't serious. And that isn't quality. So Sarrai works differently. An answer without a source is no answer with us. Sarrai isn't allowed to say anything that isn't in the database. If the knowledge base doesn't contain the information, then Sarrai says so too. At that point the escalation to a human operator is automatic. And when that person answers the question correctly, Sarrai's knowledge base is updated with the right information. That way your knowledge base keeps getting better, without you having to put an enormous amount of time into it yourself.

An AI that always answers seems more impressive than an AI that sometimes stays silent. But you don't want to unleash an 'impressive' chatbot on your customers — your customer communication is too precious to you for that. You want to get close to the human experience, and people? When they don't know something, they just say so honestly, and then go and find out. That's exactly what Sarrai does too.