In this post we are going to look at how and why we are building voice bots in the Python programming language. At Orion Labs, we run a voice communications platform which allows teams to talk to each other using our wearable push-to-talk device, Onyx.
Every day, our customers speak to each other over our network. Some of them work at big companies, some work at small ones, and many speak different languages.
As we grow our business and develop our products, we are focusing on improving communication and providing voice integrations and solutions for customer workflows. One of our first offerings in this area is a translation bot which allows our users to translate between English and Spanish.
Needed bot functionality
Before we can build our bot, we need to decide what exactly we want it to do. Our translation bot needs to do the following tasks:
- Translate between English and Spanish
- Accept voice commands for the following:
- Swap language between English/Spanish: ‘switch my language’
- Explicitly change to a language: ‘change my language to Spanish’
- Tell the user what commands can be said: ‘what can I do here?’
Mapping functionality to services
To build out the above functions we need the following supporting services and components which let us translate speech.
First, we need a data store to remember the user’s current spoken language. We also need a translation service to convert native speech to speech in a new language.
We also need services which let us accept voice commands.
First, we need a speech to text (STT) engine to go from audio in to text in the native language. Second, we need a natural language understanding (NLU) engine to detect intents, commands to act on. Third, we need a text to speech (TTS) engine to speak to the user and tell them if the command succeeded or failed.
So we need the following: a data store, a STT engine, a TTS engine, a NLU engine, and a translation service.
Picking the right tool for the job
Now is an exciting time to be working with cognitive services and voice data! Many different solutions are available for text to speech, speech to text, natural language understanding, and translation. Some of them are:
- Watson NLU
- Watson Speech to Text (STT)
- Watson Text to Speech (TTS)
- Watson Language Translator (text only)
- Amazon Polly (TTS) 100 request/second limit, 24 languages
- Facebook wit.ai (NLU), 50 languages
- eSpeak NG (TTS) GPLv3 open source, 99 languages
The pricing for the above services is quite reasonable and we are focused more on ease of integration and useful features. Microsoft is great because it offers all of the services that we need in one place. The speech-to-speech translation feature in the Microsoft Translator Speech API gives it an advantage over its competitors, who only offer text-to-text translation.
Which programming language?
We wrote our voice bot in Python for a number of reasons. We want to be able to make a Python SDK, and writing our translation bot in Python further develops our Python code base.
There are also great Python libraries already out there which make interfacing with third party services and making http requests easy. Some of our favorites are the libraries speech_recognition and requests.
Another reason we chose Python is that it is very easy to read and understand, and we want our bots to be easily understood by our internal developers and future public bot developers.
When building our first bots, we decided that they should be modular programs that can be added to Orion communication groups. That way, they provide additional functionality to groups, the bot complexity is self contained, and bot code remains separate from our server/infrastructure code bases.
We also wanted to make using the voice data as easy as possible, so we built a voice layer module which includes all our TTS, STT, Translation, and NLU services in one place. This lets us easily change providers for our services, and also alias our calls into understandable method names. So we can write
And finally, we design for the best of times and for the worst of times. Since we’re relying on third-party services, we need ways for our bot to degrade gracefully in case any of those services temporarily become unavailable.
Firstly, we focus on the user experience. If any of the services are not working the bot verbally lets the user know that there’s been a problem, and the bot emails and slack messages engineers to fix the problem.
- Capture Exceptions: wrap all method calls in try.. except to capture any exceptions so the bot does not crash
- Include Canned Audio: include audio files that will be said by the bot for standard responses for success and failure
- Implement Useful Logging + Reporting: we record how long it takes the bot to respond back to a user and how much of our translation capacity we are using
Putting it all together
So bringing it all together, we write a bot. When we turn it on it logs on to our network and joins the group that it lives in. In its home group, it waits for users to talk to it and say commands and phrases for it to translate. If a user says is a command, like ‘Change my language to Spanish’ their language is changed to Spanish, and a confirmation message is said to them in Spanish. If they say something that isn’t a command, their utterance is translated into another language, and played back to them. With a bit of work programming, going through code reviews, and with quality approval, we have a bot! You can watch our demo video below to learn more about our translation bot.
Source: Chatbots Magazine