You Shall Not Speak
Benchmarking Famous Speech Recognition APIs for Chatbots
Unless you are an English native speaker with a perfect British accent or a direct descendant of Uncle Sam, the future is not yet here for you. Sorry, but you cannot talk to your chatbot. You’ll have to continue typing for a few more months (fingers crossed) until Google, IBM or Microsoft nail it. Yep. We share the disappointment.
So let’s throw some figures in. There’s this magic number everyone keeps referencing: 0.85. I don’t know if it’s researchers or popular science gurus, but we keep hearing that in order for a speech-to-text machine to be performing it should have a precision higher than 0.85. Google scratches this threshold and gets a “pass” score. No “summa cum laude”. Watson in English is behind. And it ends there. Watson in French is not even close (0.6) and Microsoft is in the twilight zone (somewhere between 0.7 and 0.8). Amazon, Apple, Nuance are not in the match. APIs not available, no doc, or too restrictive technical constraints.
We guess that at Recast.AI, we’ll have to go with (and pay for!) Google for our collaborative bot platform. To be honest, we had better expectations before we started this exercise. Let’s hope that in 6 months time, when we’ll redo it, we’ll encounter way better results. Until then, our 15,000 developer community will have to excuse us and wait.
This article presents results for three speech to text providers. Others are not present for several reasons detailed below.
The following services were tested in this benchmark:
- Google Cloud speech API
- IBM Watson
- Microsoft Speech Bing API
Regarding Microsoft, the Bing API was used instead of the Custom Speech Service because it’s only available in English for now. The Bing API allows you to create custom speech models depending on your usecase and vocabulary.
Nuance, Houndify and Amazon services are not present in this benchmark because of some audio format limitations. They accept only some specific sampling rates, which doesn’t make them production-ready for us. As we receive non-constrained audio data from our users, we cannot limit ourselves to a certain sampling rate.
For more information about frequencies and audio processing, we published an article about how developers can work with audio data.
This benchmark uses audio data and transcription from the Voxforge corpus (standard dataset highly referenced in research papers). We have used 1000 samples (from 3 to 11 seconds) for both English and French benchmarks. The whole dataset is available here. The sentences are all audible, even if the quality isn’t consistent.
- Google speech API is consumed through the official Google cloud SDK
- IBM Watson is consumed through the official IBM Watson SDK
- Microsoft speech API is consumed through the HTTP API
As we use speech recognition in a real time context, we will analyze both accuracy and speed of each provider.
Accuracy is measured with two metrics:
- Percentage of exact recognized sentences
- WER (Word Error Rate): Ratio between insertions, deletions and substitution over the size of the sentence
These two metrics are computed on lowercase sentences, without punctuation. The speed is measured in milliseconds.
This metric is a binary value on all samples, whether the prediction is the same as the expected output or not. Every single letter must be the same.
Example: “This sentence will be recognized” doesn’t match with “This sentence will be recognize”.
The first thing to note is that all three providers are significantly better in English than in French.
Google is outperforming IBM and Microsoft and has the lowest difference between the two languages. Microsoft and IBM are more than twice (!) as good in English than in French.
Provider Exact Match (English) Exact Match (French) Google 40.2% 35.1% IBM Watson 32.5% 8.8% Microsoft 25.8% 12.0%
At a higher level, the complexity of French conjugation gives rise to a lot of mistakes that make the sentences inaccurate.
Here is an example of the output the different APIs provided for the sentence “lls étaient quatre réveillés son commandant lui et deux techniciens”.
On the phonetic aspect, the three outputs are quite good. Only IBM Watson got a perfect phonetic score not missing the link between “ils” and “étaient”! (the ‘s’ IS pronounced). Nevertheless, none of them managed to construct a grammatically valid sentence. Note that although the error of replacing “deux” (=two) par “de” (=of) doesn’t make the sentence completely unreadable, it leads to a loss of quite valuable information.
Word Error Rate
This metric describes the percentage of errors on word recognition. An error can either be a substitution, a deletion or an addition.
Once again, Google is the best performing and the most consistent while IBM and Microsoft are acceptable in English but not in French.
Google and Microsoft record a disappointing 11% and 31% error increase respectively while IBM’s error is increased by 140% (!). We could attribute the error increase between English and French to the high level of complexity of French grammar.
API Response time
While Google and Microsoft have a very acceptable response time, IBM is quite slow (twice as slow in English, three times as slow in French).
Real life test
At Recast.AI, we use speech recognition to help our users build chatbots. Therefore, we conducted a real life experiment of speech to text applied to bots!
In this test, we’re comparing how the bot and NLP react to both text and voice-translated text.
The use case is a simple chatbot for customer service: it provides information about insurance records. To do that, it needs to collect the customer’s name and their reference number.
The dialog is very simple and is made of 6 interactions:
- The user asks for the status of their insurance claim concerning a water damage.
- The bot asks for the customer’s name.
- The customer gives their name
- The bot asks for the reference number
- The customer gives the reference number
- The bot gives the claim status to the customer
Here is the text input in English:
- User — Hi, I’ve made a water damage claim last Tuesday and would like to know its status
- Bot — I’d be glad to help you with that. Can you please give me your full name?
- User — Jeffrey Lebowski
- Bot — Very well. What is your record number?
- User — AEFD1267
- Bot — Thank you. The status of your claim is “In Examination”
And in French:
- User — Bonjour, j’ai déclaré un dégât des eaux mardi dernier dans mon appartement. Je voudrais savoir où en est mon dossier.
- Bot — J’aurais besoin de votre prénom et de votre nom.
- User — Jeffrey Lebowski
- Bot — Très bien, pouvez-vous me fournir votre numéro de dossier?
- User — AEFD1267
- Bot — Merci, l’état de votre dossier est actuellement: En attente de validation.
For the experiment, we recorded speech files with the content above, fed them to all APIs in both languages and used the outputs to conduct the bot conversation.
The first sentence is well recognized and translated. However, the last name Debowsky and the reference number are in the correct format and accurately detected but not spelled properly. Therefore, the insurance’s information systems can’t match the two pieces of information to provide the correct update. Conversation fails.
That’s going well! Minus a conjugation error, the first sentence is detected and understood correctly, the name and ID record are well understood and transcribed, so the bot can go to the end of the conversation. Well done!
With Watson, there are a few mistakes in the first sentence, but the overall meaning is clearly conveyed. The name is correctly translated and recognized, but the reference number is not accurate, even if it is in the right format (4 letters and 4 digits). Therefore, the bot cannot move forward.
Note that Recast.AI easily detected the reference number entity, even if Watson transcribed the values in letters and not numbers!
It gets trickier!
In the first sentence, there is a translation error that puts a name in the sentence when there isn’t one: au nom de Doucet (in the name of Doucet). The bot detects this name, assumes it’s the client name, and follows up directly by asking for the reference number. The bot doesn’t need a very strict “problem -> name -> id” structure to function properly! If it detects a name, it has what it needs and moves on to the next piece of information required: in this case, skipping the second step of our test.
Unfortunately, the ID record is not understood in a valid format (4 letters and 4 digits) as the first number is understood as “en” instead of “un (1)”. Conversation has failed.
With Microsoft, the first sentence is understood correctly while both the name and reference number are detected in the correct format but inaccurately translated. The user can’t get an update on his claim.
In French, the reference number is correctly detected and translated, but the name isn’t! Therefore, the bot can’t match them with an existing claim.
As you can see, through 6 conversations, only one has been able to correctly finish the 3-sentence exchange. Strangely, the most successful is in French! Maybe that comes from the fact that we are native French speakers and have a slight French accent in English.
In the end, speech recognition doesn’t look as human-like as what we might expect!
Google Cloud Speech seems to be the better solution as they have the most accurate and consistent results, but it is the most expensive. IBM Watson suffers a big drop in performance when working with French speech. Moreover, the response time is way too high for real-time usage. Microsoft is in the middle with a good response time.
Bottom line, if you are the head of IBM Watson or Microsoft Speech Bing API and you are reading this, speaking at least on behalf of our community, your users would truly appreciate an earth-shaking performance boost. As for Google Cloud speech API, keep working, make it free and we’ll all say Amen 🙂
Originally published at Recast.AI Blog.
You Shall Not Speak: Benchmarking Famous Speech Recognition APIs for Chatbots was originally published in Chatbots Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source: Chatbots Magazine