Latest news

Rose in the Loebner Chatbot Competition; An Interview with Bruce Wilcox

No comments

(Part of the Bot Master Builders Series)

Bruce Wilcox is a legend in the chatbot community. His bots have come in 1st or 2nd more than anyone else for the Loebner Competition in the last 7 years, a competition that sees if an AI intelligence can pass the infamous Turing Test (where a panel of human judges cannot tell a bot apart from a human). He was an early expert in creating a Go program and has created AI avatars for companies like 3DO and Fujitsu Labs.

The most impressive thing about Bruce is that he’s thought more deeply about conversational interfaces than almost any other engineer and scientist in the world, and has built some really neat ones. He even went so far to create his own scripting language and engine, ChatScript. He and his wife Sue run the natural language processing company Brillig Understanding to consult on projects and promote ChatScript. Here is a great talk Bruce gave at Google on how he thinks about chatbots and here is a paper he wrote on Winning Chatbot Design for building great bots for the Loebner.

What is your current design vision for Rose? Just winning the Loebner Prize, or something else?

Rose is a private bot we made to win the Loebner, further our research, and publicize my company Brillig, which does bot development.

Tell me about your team — how did your team for Rose come together, and what are the roles?

It’s my wife Sue and I — she is the writer, I am the programmer. That’s it.

How do you measure success; what are your metrics?

The Loebner has two parts; a first qualifying round to make it to the top 4, then a second round where you have to pass the human judges.

The qualifiers are a test of human knowledge, your bot is asked a question or asked to remember something. It uses all past Loebner questions and all the questions from Chatterbox Challenge, Chatbot Battles.

The judges can do anything, either interrogation or conversations. I use an automated regression test to test Rose myself.

What are successful interactions? What are failed interactions?

It’s mostly a matter of going and looking at the logs — a human has to do it. There are two things we want to happen: minimize the obvious blunders (bogus answer, totally fail), create an “aha” moment (a brilliant, spot on answer, where we understood the intent and gave a useful response).

How do you get to the “aha” moment?

ChatScript uses mostly IF/THEN rules — all rules are organized into topics. Topics have a list of keywords suggestive of the topic. If you can’t succeed, then the bot goes to another topic. If you cannot recognize within a topic, the bot is better off switching.

How do you make scripts better?

How long should a script be? We started with paragraphs and now have code that you can limit us to 140 characters. We change the size of a message to be channel appropriate — the original Suzette bot [a prior version or ancestor of Rose] would look at reply length and agreement/disagreement; if a user asked too many questions or gave too few answers, we could run a quick module and then output a variable. We would then try to push a re-engagement conversation. “I have to go” -> “do you have to?” OR “I’d like to talk a little longer.” For insulting or sexually abusive users, we would check an IP address and blacklist them.

What have you learned about developing a viable personality.

Go to a movie or book — authors have created a character with a consistent of set of information about themselves, a worldview. They can predict what the character will do. If the character says “I hate fish”, they can’t fish later or must explain themselves. The problem with CleverBot it that it’s a random mishmash. So you need to write consistent personalities. TalkingAngela is a great bot; she’s an 18 year old female cat with strong fashion views — people bond with her on those views — when she disses me and I diss her back, she gets back.

What did you mean in this statement: “Have a lot of persistence if you intend to compete.”

Multiple levels of persistence are important — large amounts of engine introspection ability. What rule causes me to give this particular output? Bot developers can see and modify what the engine’s behavior will be. Most bots don’t have much user persistence — you have to pass your own data in and out — there’s no built-in solution. The right model for a bot platform is to have persistence — conversations are inherently persistent.

Also what about persistence with respect to pronoun resolution?

There’s no ideal way to do this. Two approaches are often taken: 1) look at your own statement to figure out the pronoun, pre-path referencing. 2) post-path referencing. ChatScript can access the things it will say to the user and find its own pronoun referencing to bind a noun to its pronoun. There can be any number of sentences in a volley — then the ChatScript bot replies. The first thing is execute a pre-path script. The main pass is once for every sentence, it takes and interprets the script. Post-pass is after all inputs have been processed — it knows the set of answers it wants to say. We can set expectations on what you expect from the user — initialize and set up for the user reply. This is all in the ChatScript docs.

Tell me more about topics and the idea of conversational volleys or topics, which limit where conversations goes and simplify pattern matching.

A ChatScript topic is a name, a collection of keywords, and a collection of rules. A ChatScript bot can go out of a topic quickly and get back fast — it’s a list of keywords or phrases to get into a topic — as a human would go into the topic. “I” or “action” are too generic for topics, but “cinema” or “fire station” could work. “Death” and “funerals” will have an overlap in keywords — how well does this sentence match — we’re using a simple COUNT function.

Said again, topics have keywords so the ChatScript engine can automatically find the most relevant topic for an input sentence by looking at the number of matching keywords it has and how big the keywords are. The system will try rules in the most likely topic first, and if they all fail, it will try lesser matching topics. Topics that don’t match don’t get tried unless you explicitly tell the system to try them (you can completely control how processing is done).

Explain gambits to our audience, and why bot developers should use them.

Topics have a type of rule called a gambit. It’s something the chatbot can say when it has control. Even if the system cannot find a direct response to your input, if your input suggests we are talking about baseball, the chatbot can offer you a relevant gambit. Or the bot can initiate a topic and say a gambit. Gambits allow the chatbot to tell a story in the topic. If the user metaphorically nods his head in acknowledgement after a gambit, the system is free to issue the next one in sequence. If the user asks a question or makes his own statement, the system can try rejoinders or other responders to reply.

One of the implied rules of a conversation is maintaining a balance of intimacy. If I ask you a question, I am expected to share my answer to it as well. So I often author a topic in a style of gambits asking a question and then volunteering the chatbot’s answer, as is seen in the baseball topic. Gambits do not require patterns, but they can have them. Typically this is done to test conditions unrelated to the user’s input.

Most bots are a stimulus and response. They follow the REST protocol of the web. ChatScript is not that, all users have their own file memory. If the user says something not understandable, Rose takes charge of the conversation, and then the flows can follow and the user can respond in a semi-predictable way. “What is your favorite natural disaster?” Can recognize a range of them, then have a scripted response. Ready of the answer to “typhoon”, look for expected rejoinders.

Most bots are purely reactive — no sense of trying to lead a conversation or steer back to a topic — one of the things that Rose does is say something, then wait, and then re-prompt.

Can you expound on long term memory?

Rose can re-answer a standard question many times — some topics never erase themselves. For example, “what is your eye color” — same answer all the time. This will erase all gambits as you use them up, will usually erase all questions (it will only keep personal data information — an information topic is different than a conversation topic). In ChatScript, it is two-fold — you know if every rule is used up, per an enumeration table. Also you have a “short-term” memory of the last 20 things the user said and can also choose to record information about the users. “My eyes are blue” can be put in a JSON structure. It’s mostly personal information about the user for Rose. For commercial projects, other tasks like “Book me a flight to PHX tmrw”, it could be the date, destination, etc. All user conservations can do a specified memory.

What are the most common things users ask outside of the main function. Do they ask for jokes or other advice?

Outside the main function it’s mostly trolling and jokes, sexual innuendos and non-innuendos — or they’re trying to trip it with Loebner questions. Other people just chat.

How do you deal with mangled language?

We weakly deal with this. ChatScript has a built in spelling corrector — it has to make a choice and has access to a sentence in multiple formats: what the user typed; the statement after corrections to the query; the canonical representation of what he said (lemma values). Example: dogs, lemma is singular form dog. Infinitive for verbs, digits for numbers, so nine -> 9. We will also lemmatize pronouns and so can use I/me interchangeably. The pattern engine can choose a corrected or lemma version. Memory — personal information — have access to original.

Can you describe nuances of conversations that bot developers should know about?

Chat is self-extinguishing. You don’t want to repeat yourself. If you ask me what my job is and I tell you I work for the phone company, I shouldn’t later volunteer that same information. By default ChatScript both marks rules when they get used, to avoid using them again, and looks up its current output to see if it has already said it recently. In either case, the current rule would fail and the system would move on to find another matching rule. This means that I often write rules that share data using a reuse function.

Thoughts on ontologies and building entities? Find any good ones?

ChatScript comes with some 1400 predefined concepts, you can define your own, and it has WordNet’s ontology. Their noun ontology is often good, but other times it is not what I would want, and their non-noun ontologies are poor. Concepts and hierarchies work great — WordNet only works for nouns, not verbs or adverbs. It only has a single hierarchy — for dog would have canine to mammal, but not “other household pets.”

What is your Knowledge Base (KB) like? Do you use fact triples and a graph database to store them? What have you learned?

When I define a concept, it’s a set of words and phrases. Canine~1 would be all dogs that come there — can inherit from WordNet. For representation, we use fact triples, subject-verb-object (SVO), fixed verb and ordering of fact. ChatScript can do graph queries on simple structures: Find me what dog is a member of, what is the capital of france. Talking about concepts sets — hierarchy is what matters, everything is subset of sets. When doing pattern matching, we can match a set as fast as a word, ~dog run, can match with the set of any dogs (bulldog, poodle, etc). Patterns as representations of meaning. With ~own, set of words and phrases that deal with ownership.

Any thoughts on using neural nets to represent meaning in more sophisticated ways — have you played with Google’s TensorFlow or Random Decision Forests?

The problem with machine learning is you have to learn an intent — it can be a thousand user says expressions to train it. The problem with most people building bots is that they don’t have user data — they only use 50 sentences, but the user comes along and says something different. You get an error log and re-train. Small companies don’t have the data. So it’s much faster to write a pattern to cover a range of sentences, include all the synonyms. We can use slot filling and entity matching to do this; can use an enumeration set of pizza types. For an infinite set, patterns are better. Training times for ML is too long.

Brute force bots versus hand-scripted: “Cleverbot approach will eventually follow in the AI tradition of brute force winning out.” Have your recent ones worked out?

Machine learning bots to date are not good enough — Google is not good enough. Ultimately with data you can do it [create great bots].

What can you tell us about your tech stack? What decisions have you made doing NLP in-house, what external services do you like?

Everything is built-in house — we use a text editor. We are production quality system — good for enterprises. ChatScript now has third-party software, like Curl for communication, MongoDB and PostgresDB. Duck as JS compiler. All the underlying NL is part of the ChatScript engine.

Where do you get chat transcripts and user data? Any suggested data sets?

Get a large body of testers, don’t just have two testers. Disperse your testing on a large body of users with different thinking and expectations, different ways of saying things. You just need a bunch of people. More is better, a few dozen is good, a few hundred is unmanageable (Amazon can do it with a massive team).

What other bots have you looked to for inspiration? Who else out there is making a decent bot? Are there other use cases you’ve thought were simply brilliant?

ChatScript was inspired by personality forge, facade, other early loebner engines. A good chatbot would be Mitsuku — but only for stimulus / response. We have different goals between AIML and ChatScript — AIML is simple and easy to pick up, using XML to script. Once ChatScript rule can do 50 AIML rules. AIML is not a standard, not a complete solution. Pandorabots is not AIML, it has extensions. AIML 2.0 didn’t support JavaScript.

Who are other smart people in the bot world you’ve met, whether on the tech stack, UX, scripting, or even financing sides?

Steve Worswick is smart — he has done miraculous things in the context of AIML — it’s amazing he can make the bear dance. But he’s stuck in the wrong pond. I’m biased toward teams using ChatScript. Mark Meadows at Botanic.IO uses ChatScript to create chatbots for business — multi-model world of gesture recognition and sentiment analysis. David Colline at Sapientx — the Hilary versus Trump website — you can chat with the images. YourMD uses ChatScript as a client, it’s top rated for diagnostic applications. They remap the weird way people say stuff to the standard NIH terms and can also chat with health users.


Rose in the Loebner Chatbot Competition; An Interview with Bruce Wilcox was originally published in Chatbots Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source: Chatbots Magazine

magnoliaRose in the Loebner Chatbot Competition; An Interview with Bruce Wilcox

Related Posts

Leave a Reply