Question & Answering Service Template
Question Answering Service
is available in Lens Studio Asset Library. Import the asset to your project, create new Orthographic camera and place the prefab under it.
The dialog module uses a Machine Learning Module to find the answers to questions in a given text.
In your Scene Hierarchy
panel, you’ll see an object named ConversationHandler [EDIT_ME]
. Click on this to show its input window in the Inspector. For “context”, insert whatever you want your source to be, e.g., a Wikipedia article about the moon.
This is what Dialog will use to answer the users’ questions.
Debugging Tools
To make sure your lens is working the way you want, add a few questions you would like your lens to be able to answer to the list of Sample Questions, such as “How far away is the moon?” Then, set the Debug Mode to, “Every Question has an Answer”. If any of your questions are not answered by the module, you will see an error message in the Logger. We recommend adding around eight questions but no more than 15.
You can ask up to 15 questions per Lens session.
Example Errors
The Debug tools will also check if your context exceeds the 5000 character limit for you. This is not a hard limit, but beyond that point, the DialogML’s performance tends to decrease.
Make sure you don’t delete the dialogUtility SceneObject, which preprocesses your context. Without it, your Lens may still run, but cannot be approved for publishing.
How the text is displayed + Hint Asset
You can plug in your own Text and Image components that will be displayed with the transcript and answer. After the user has asked a question, the transcript and response will disappear after a few seconds. You can change how long it stays up with the Leave Up Text Time
variable.
The hint and listening indicator are displayed when the user hasn’t said anything yet as well as after a transcript and response are cleared from the screen.
QuestionsAnsweredEvent
The conversation script manages listening to the user and answering questions. Whenever it answers a user’s question, it calls a QuestionsAnsweredEvent that you can listen to from another script if you want to do something with that question-answer combo.
Make sure that the script doing this is below the Conversation script in the Scene Hierarchy
panel. Scripts that are bound to the same event will run in the order of their location in the Scene Hierarchy
panel, so you can’t add a callback to QuestionAnsweredEvent before this event is initialized in Conversation.js.
global.QuestionAnsweredEvent.add(callbackFunction);
The callback function will automatically have the parameters, which are two strings;
- The transcript of what the user said.
- What the DialogML responds.
callbackFunction(transcript, response);
In the template, this callback is used to check if any of the words the user said were “elephant” and if so to have a little elephant appear on the side of the screen before swooping away again.
Custom Triggers
There are three built-in custom triggers that you can use to trigger behaviors without running any code. These will be automatically sent when an answer comes back with one of these states, so all you have to do is put your behavior script with your trigger response in the corresponding array.
In the template, there are two behaviors bound to the unknown event, meaning they’re called when a question is answered with “I don’t know.” One behavior turns on a head binding image and the other turns it off with a two second delay. The result is that every time a question is answered with “I don’t know”, the user will have a question mark spinning near their head for two seconds.
How to Write Good Context
In order for your model to take in information well, your context should ideally be made of factual, simple sentences. Too many relative clauses may end up being too convoluted for DialogML.
The context should be less than 5000 characters. After that, it tends to decrease in accuracy.
Background knowledge
People may have a lot of base knowledge that they assume that other people have. However, DialogML only knows what it has been told in your context. If your context is “Elephants live in South Asia”, the model will not be able to answer whether elephants live in Asia, because it does not know that South Asia is a region in Asia.
While this might seem obvious, consider the example context “I live in West Virginia.” In this case, we would not want the model to assume you live in Virginia, since Virginia and West Virginia are different locations.
Similarly, if you tell the model “Elephants live only in Asia and Africa,” the answer to the question “Do elephants live in Europe?” will be unknown. The model doesn’t know what Europe is.
How does the scripting API work?
There are two ways to access DialogML: directly or via VoiceML Events. We recommend using dialogModule.askQuestions() for debugging or if the user is using a non-speech kind of input such as text. If you just want the model to answer questions the user asked out loud, then the VoiceML approach is probably preferable.
How does DialogML work?
The dialog module is an asset in your Resources tab, that you can pass into a script component using the following code.
//@input Asset.DialogModule dialogModule
It only has one method which has four parameters:
/**
* check if answer has a valid value and if not throw an error
* @param {String} context the text the model will use to answer the questions
* @param {String[]} questions the questions you want to have the model answer
* @param {function} answerCompleteHandler the function that will be called when the model has figured out the answer to your questions
* @param {String[]} errorHandler the function that is called if there’s an error
*/
dialogModule.askQuestions(
context,
questions,
answerCompleteHandler,
errorHandler
);
As stated above, answerCompleteHandler
and errorHandler
are functions that DialogML will call for you. Unless you want it to do something special, errorHandler should probably just throw the error:
var onQuestionsAnswerErrorHandler = function (error, description) {
throw new Error(description);
};
answerCompleteHandler
will handle what to do with the answers when you receive them. Its argument is a list of Answer objects which have three properties:
Class Answer
- questionId is the index of the question this is answering in the array questions.
- answer is a string with the answer to your question.
- status is an enum: AnswerResponseStatusCode which can take 4 values;
UNSET_UNSET
STATUS_OK
NOT_A_QUESTION
NO_ANSWER_FOUND
It does not have a context ID, since you have to plug in a specific context into askQuestions()
.
Thus, your answerCompleteHandler could look something like this:
var onQuestionsAnswerCompleteHandler = function (answers) {
for (var i = 0; i < answers.length; i++) {
var a = answers[i];
if ((a.status = AnswerResponseStatusCode.STATUS_OK)) {
print(
'The answer to question number ' +
a.questionId +
" is '" +
a.answer +
"'"
);
}
}
};
This would just print your answer to the logger, but you can just as easily add this text to a screen text component.
How do I use the dialogModule together with Speech Recognition
While the dialog module only has one method, there is a bit more capability built directly into VoiceML, so that you can have them automatically detect questions and feed them into DialogML.
To this end, we first have to let our VoiceML know what context it should use to answer questions, which we do via the ListeningOptions:
var options = VoiceML.ListeningOptions.create();
var qa = VoiceML.QnaAction.create(context);
options.postProcessingActions = [qa];
Again, context is the text you want to be used to answer questions.
Don’t forget to add these listening options to your VoiceML:
var onListeningEnabledHandler = function () {
script.vmlModule.startListening(options);
};
Don’t forget to bind your functions to the voiceML’s events:
//bind event callbacks
script.vmlModule.onListeningUpdate.add(onUpdateListeningEventHandler);
script.vmlModule.onListeningError.add(onListeningErrorHandler);
script.vmlModule.onListeningEnabled.add(onListeningEnabledHandler);
script.vmlModule.onListeningDisabled.add(onListeningDisabledHandler);
From there on, the VoiceModule handles it and reports back to you in the following snippet;
var onUpdateListeningEventHandler = function(eventArgs)
Where you can access your answer using the following;
var answers = eventArgs.getQnaResponses();
The variable this returns is another list of QnaResponse objects, which are similar to Answers but not the same.
Class QnaResponse
- answer = the textual answer
- answerStatusCode - the answer status which is one of:
- OK = 1
- NOT_A_QUESTION = 2
- NO_ANSWER_FOUND = 3
The methods with and without VoiceML do almost the same thing. Using the VoiceML methods simply cuts out some of the steps
Limits of DialogML
Since this is the very first version of DialogML, there are some known issues that will be addressed in later versions. We hope you’ll still enjoy making Lenses with it and look forward to what DialogML can still become.
If you find any bugs or issues not addressed here, please contact voiceml-mobile@snapchat.com with a description of the issue. Screenshots, complete error messages or even the project file itself are appreciated to help us figure out what went wrong.
Overconfidence
The model can be somewhat overconfident in its ability to answer a question. As a result, it may answer yes or no questions it doesn’t know the answer to with a random yes or no answer. Similarly, if you give it a context like “I have two arms” and ask it how many legs you have, it may answer “two”.
This can cause some unexpected behavior, but can be ameliorated by making your context more comprehensive.
Comparisons
Currently the model cannot handle comparisons very well. This means if your context was “The moon is 239 miles away. Mars is 103 million miles away”, your model would not know which is farther away. It also cannot answer if the moon is farther than 100 miles away or not.