Intelligent Conversational Bot 2017
TVBot
C.-S. Chen, K.-L. Lo, F.-Y. Sun, H.-T. Yeh & Y.-T. Yeh
https://facebook.com/theTVBot
Overview
We are able to:
-
find show info given other info.
-
find character name given person’s real name, or the opposite.
-
find the person related to a show given her/his job.
-
accept English and Chinese.
-
do voice recognition.
Ontology and Database
Source
Trakt.tv: it provides a unified interface to popular movie / tv show sources (IMdb, etc.).
Intent List
Fine-grained intents over vague intents. Why? Reduce possibility for wrong slot filling.\
e.g. Who House of cards? v.s. How many are there in Game of Thrones?
-
request_show_title
-
request_show_first_aired
-
request_show_air_info
-
request_show_runtime
-
request_show_network
-
request_show_country
-
request_show_status
-
request_show_rating
-
request_show_genres
-
request_show_aired_episodes
-
request_show_season_count
-
request_cast_character
-
request_person_name
-
request_person_birthday
-
request_pershon_birthplace
-
request_person_death
-
request_crew_sound
-
request_crew_camera
-
request_crew_directing
-
request_crew_writing
-
request_crew_visual_effects
-
request_crew_production
-
request_crew_lighting
-
request_crew_editing
-
request_crew_art
-
request_crew_costume
-
inform
-
confirm_yes
-
confirm_no
-
deny
-
thanks
Value Mapping
The slot value from user NL might not match actual slot.
-
country: Japan, Japanese -> jp
-
year: 1970, the 70s -> 1970
-
rating: nine, 9 -> 9.0
-
time: 16:00, 4p.m., 4 o’clock -> 16:00
Language Understanding
RNN-NLU
RNN-NLU is able to classify intent and fill slots at the same time, jointly optimized.
Tend to over-fill. e.g. I would like to watch a show that airs
Friday.
request_show_title(show.air_day: ’Friday’, show.country: ’to’)
api.ai
Provided by an external service. Model unknown.
Tend to under-fill. e.g. I would like to find a show produced by .
request_show_title()
Comparison
RNN-NLU | api.ai | |
---|---|---|
Good | New model, powerful if fine tuned. | Online training. |
Bad | Hard to train (days every time). | Black box, can’t tune or apply tricks. |
Data
Mostly generated by hands. We generate almost 1000 templates, and fill in the db values as the training sentences.
Dialogue Management
State Tracker
For both user
and agent
, the state tracker tracks the following
information in vector form
<a1, a2, …, an> a_i in {0, 1}
-
(latest_)inform_slots
-
(latest_)inform_values
-
(latest_)request_slots
-
(latest_)confirm_slots
-
(latest_)confirm_values
-
latest_action
RL Agent (DQN@Mnih+2013)
Ruled-based Agent
- If user confirms a slot, make sure the slot is what we confirmed previously. If not, confirm again.
- If the slot value user informs mismatch with our state, confirm the slot.
- If user informs without any slots in NLU, confirm what the user want actually.
- If we have show cadidates few enough, inform user or raise a multiple choice directly.
- If candidates are too many, some heuristic (cast, crew first) and database information (how many possible values in a slot) helps to decide which slot to request.
- If all the slots have been requested or there’re no candidates at all, then confirm a slot according to previous confirm/request slots which is stored in state.
1 ~ 3 is responsible for handling NLU error. If none of it is triggered, then it means that NLU did it quite well.
User Simulator
-
Goal: contains target slot and information known by user.
-
Success: after episode ends, check if agent reply correct slot.
-
Error model: to simulate NLU error, a parameter controlls the possibility of wrong NLU.
Ruled-based | DQN | |
---|---|---|
w/ error | 0.66 | 0.53 |
w/o error | 1.0 | 0.93 |
Reward function
Our reward function definition is
- -1 * max_turn, when dialog fails
- 2 * max_turn, when dialog success
- -1, otherwise
Performance
Natural Language Generation
We separated the sentence into 7 categories:
-
thanks
-
inform
-
request
-
confirm_again
-
confirm_conflict
-
confirm_no_conflict
-
multiple_choice
BLEU | |
---|---|
Training | 1.0 |
Testing | 0.4425 |
Rule-based NLG
Fill the slot values into predifined template.
NN-NLG
The model is a seq2seq. Trained on several templates for each categories.
Miscellaneous
Speech API
Integrated with Bing Speech-to-text API. If audio gotten, call the api.
Translation API
The db is built in English. To support other language, sent to Bing Translation API if other language detected.