Latest news

Difficult Conversations 1: When Voice Recognition goes wrong

No comments

Have you thought about what your chatbot will do when the conversation goes wrong?

The patterns for handling problems in chatbots turn out to be surprisingly different than in other forms of interfaces. We need to be very careful how we handle errors, since in a CUI it will feel to the user not that the application just isn’t working properly, but that they are not being listened to. That makes poor error handling here not just frustrating, but also uncomfortable for the user.

I started writing this as one article, and quickly realised that even at a high level there was way too much I wanted to cover, so I have decided to split it into 3 separate articles:

  • Part (this) 1 – Voice Recognition Errors: The one where we end up shouting at a computer
  • Part 2 – Conversational Errors: The one where it gets really interesting
  • Part 3 – Technical Faults: The one we want to pretend will never happen (or ‘Help! My chatbot has fallen over and can’t get up’)

This is the first part, and the second and third parts will be coming soon.

Voice Recognition Errors

Although Automatic Speech Recognition (ASR) has been improving rapidly it is still very far from perfect. Background noise (marching band arriving in your living room, one single toddler) or regional accents can easily confuse it.

Unfortunately errors in voice recognition are the most excruciating and embarrassing for user. Yup, I said embarrassing. Think about a time that a person asked you to repeat yourself. And then asked you to repeat yourself again— it’s not a nice feeling. You feel like the fault is yours, that you can’t speak clearly enough to be understood. This is why we need to handle these very carefully.

Now, most of you will not be handling your own speech to text service, the channel you are surfacing your bot through will do that for you. However, there are still things you need to be aware of and consider on your side, and you do need to be aware of how the channel you are using handles voice recognition errors. Is it a good experience for the user? Does it fit with your bot and they way it interacts with the user?

Complete Failure in Speech to Text Transcription

This is the only time you should use “can you repeat that?”.
Let me repeat that.
This is the only time you should use “can you repeat that?”.

I have seen text chatbots ask me to repeat what I just typed when they didn’t understand. If they didn’t get it the first time, retyping the same words isn’t going to help. However, if it’s a voice system, saying the same thing again very well might work.

If you are handling this part yourself make sure you are checking the level of confidence the service you are using has in the accuracy of the transcription. When it’s too low ask the user to repeat — don’t send gobbledegook back to your chatbot.

However, be mindful of how many times you want to ask the user to repeat. As I said earlier, this gets embarrassing for the user really quickly. If you can, offer the if user a different way to complete their action after a couple failed attempts. If you are passing over to a human, and it is at all possible, bring some of the context on the handover so the person handling the call can tell the user has come from the chatbot, and whatever information they did already manage to give, since they are likely to start the passed-over conversation already frustrated.

Mistakes in the Transcription

These can be difficult to catch as there is no immediately obvious flag to your system that there has been an error. It believes that it is passing on to the chatbot the words that the user said. However, if it isn’t what the user said, the likelihood of bringing the right answer back is low.

So how will I know this is happening?

Here are some options that can alert you that you are having this issue:

  1. Review feedback from your users — are they regularly complaining about problems with being understood, or the bot giving the wrong answer?
  2. Review logs of what the user said to your bot regularly. It may be hard to spot when a transcription error has occurred — or it may be very obvious! If they are happening regularly then you may need to consider if you need to move to a different service, or consider the audience that is using your bot — are you targeting a particular set of users that would have a hard time articulating, or have very strong accents? Is there a better channel you could use?
  3. Try to catch follow-on sentiment or phrases that would indicate that a mis-transcription has occurred. For instance “That’s not what I said”. Make sure you are logging these and the preceding phrases somewhere you can review them, and that you respond to the user with an apology and ideally an action they can take (leave feedback, talk to a person instead). If you pass them to a human, as before, bring the context of the conversation (transcripts of what the bot did catch, length of time trying to use chatbot, etc) so that it is a smooth transition and not like completely starting over again.

To demonstrate this last one I did try the phrase “That’s not what I said” with a few different bots to observe their particular user experiences.

Both Alexa and Google Assistant recognised the phrase as the user calling out that something went wrong, but both appeared to just fall back to general error handling. Alexa said “Sorry, I know a lot, but I don’t know everything”. Google Assistant apologised and offered a way to give feedback, which was a nice touch as it didn’t leave you feeling like there was nothing you could do.

Siri was the only one that recognised and responded to the specific error that I thought it hadn’t heard me properly (sorry, Siri, you did hear what I said perfectly, I was just messing with you) and offered me some options to correct its transcription. This is a nice touch, especially as I have noticed that it does learn to recognise better in the future what you said.

Cortana… well, it went its own way. It offered me a number of search links for “that’s not what I said” — which had I really been having that conversation would not have made me feel less frustrated! I think this demonstrates really nicely a user experience you want to avoid. I didn’t achieve what I wanted to the first time, and then when I expressed frustration I ended up further from my original goal.

In Part 2 I’ll talk more about Conversational Errors, and the need to have a really solid set of general error responses for scenarios where the chatbot can’t understand the input, as well as how to adjust when your bot is not correctly responding to user input. This will all be applicable to handing errors when they occur in transcription as well.

Be aware of the feedback the user is able to get about what has gone wrong. If there is still a visual interface they may see the text transcription (or lack thereof) onscreen and immediately be aware that something has gone wrong. If it is truly a voice-only interface it may be harder for them to understand. Some channels do offer the ability to go to a visual interface and see what they chatbot thought they said. Ideally, like Siri, you can actually correct it so it can improve its speech recognition overall, but at the very least allowing the user to send feedback to you is really important.


I hope you can already see that there is a lot to think about when building a really great conversational user experience, and how good error handling can really differentiate your chatbot — and we’ve only scratched the surface so far! Part 2 will cover working with Natural Language Understanding and the very interesting challenges that it brings — the trickiest part of error handling in a CUI!

If you found this article useful or interesting, please click below to let me know. Follow me here or on twitter @virtualgill to make sure you don’t miss the other parts of this series.

Difficult Conversations 1: When Voice Recognition goes wrong was originally published in Chatbots Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source: Chatbots Magazine

magnoliaDifficult Conversations 1: When Voice Recognition goes wrong

Related Posts

Leave a Reply