Question Answering

Question Answering example allows you to find the answers to questions in a given text using the Dialog Module powered by DialogML.

Question Answering asset is available in the Lens Studio Asset Library. Import the asset to your project, create a new Orthographic camera and place the prefab under it.

Customize source text

In your Scene Hierarchy panel, you’ll see an object named ConversationHandler [EDIT_ME]. Click on this to show script component inputs in the Inspector panel.

For Context, insert whatever you want your source text to be, for example a Wikipedia article about the moon. This is what Dialog Module will use to answer the users’ questions.

Debugging Tools

To make sure your lens is working the way you want, add a few questions you would like your lens to be able to answer to the list of Sample Questions, such as “How far away is the moon?”. Then, set the Debug Mode to, “Every Question has an Answer”. If any of your questions are not answered by the module, you will see an error message in the Logger. We recommend adding around eight questions but no more than 15.

You can ask up to 15 questions per Lens session.

Example Errors

The Debug tools will also check if your context exceeds the 5000 character limit to ensure efficiency. This is not a hard limit, but beyond that point, the DialogML’s performance tends to decrease.

Make sure you don’t delete the Dialog Utility scene object, which preprocesses your context. Without it, your Lens may still run, but cannot be approved for publishing.

Customizing Text, Image and Hint

You can plug in your own Text and Image components that will be displayed with the transcript and answer. After the user has asked a question, the transcript and response will disappear after a few seconds. You can change how long it stays up with the Leave Up Text Time variable.

The hint and listening indicator are displayed when the user hasn’t said anything yet as well as after a transcript and response are cleared from the screen.

Question Answered Event

The Conversation.js script manages listening to the user and answering questions. Whenever it answers a user’s question, it calls a QuestionsAnsweredEvent that you can add callback to from another script if you want to do something with that question-answer combo.

global.QuestionAnsweredEvent.add(callbackFunction);

Make sure that the script doing this is below the Conversation.js script in the Scene Hierarchy panel. Scripts that are bound to the same event will run in the order of their location in the Scene Hierarchy panel, so you can’t add a callback to QuestionAnsweredEvent before this event is initialized in Conversation.js.

The callback function will receive two string arguments:

• The transcript of what the user said.
• A response.

function callbackFunction(transcript, response) {
  //...
}

In the template, this callback is used to check if any of the words the user said were elephant and if so to have a little elephant appear on the side of the screen before swooping away again. (See QuestionAnsweredEventHandler.js)

Custom Triggers

The example can also be set up in combination with Behavior and used to trigger instances of Behavior script directly.

There are three built-in custom triggers that you can use to trigger Behaviors without running any code. These triggers will be automatically sent when an answer comes back with one of these states. All you have to do is place your Behavior script with the trigger response in the corresponding array.

In the template, there are two behaviors bound to the unknown event, meaning they’re called when a question is answered with “I don’t know.” One behavior turns on a head binding image and the other turns it off with a two second delay. The result is that every time a question is answered with “I don’t know”, the user will have a question mark spinning near their head for two seconds.

How to Write Good Context

In order for your model to take in information well, the context should ideally consist of factual, simple sentences. Too many relative clauses may end up being too convoluted for DialogML.

The context should be less than 5000 characters. After that, it tends to decrease in accuracy.

Background Knowledge

People may have a lot of base knowledge that they assume that other people have. However, DialogML only knows what it has been told in your context. If your context is “Elephants live in South Asia”, the model will not be able to answer whether elephants live in Asia, because it does not know that South Asia is a region in Asia.

While this might seem obvious, consider the example context “I live in West Virginia.” In this case, we would not want the model to assume you live in Virginia, since Virginia and West Virginia are different locations.

Similarly, if you tell the model “Elephants live only in Asia and Africa,” the answer to the question “Do elephants live in Europe?” will be unknown. The model doesn’t know what Europe is.

How does the scripting API work?

There are two ways to access DialogML: directly or via VoiceML Events. We recommend using dialogModule.askQuestions() for debugging or if the user is using a non-speech kind of input such as text. If you just want the model to answer questions the user asked out loud, then the VoiceML approach is probably preferable.

How does DialogML work?

The dialog module is an asset in your Resources tab, that you can pass into a script component using the following code.

//@input Asset.DialogModule dialogModule

It only has one method which has four parameters:

/**
 * check if answer has a valid value and if not throw an error
 * @param {String} context the text the model will use to answer the questions
 * @param {String[]} questions the questions you want to have the model answer
 * @param {function} answerCompleteHandler the function that will be called when the model has figured out the answer to your questions
 * @param {String[]} errorHandler the function that is called if there’s an error
 */
dialogModule.askQuestions(
  context,
  questions,
  answerCompleteHandler,
  errorHandler
);

As stated above, answerCompleteHandler and errorHandler are functions that DialogML will call for you. Unless you want it to do something special, errorHandler should probably just throw the error:

var onQuestionsAnswerErrorHandler = function (error, description) {
  throw new Error(description);
};

answerCompleteHandler will handle what to do with the answers when you receive them. Its argument is a list of Answer objects which have three properties:

Class Answer

• questionId is the index of the question this is answering in the array questions.
• answer is a string with the answer to your question.
• status is an enum: AnswerResponseStatusCode which can take 4 values;
  • UNSET_UNSET
  • STATUS_OK
  • NOT_A_QUESTION
  • NO_ANSWER_FOUND

It does not have a context ID, since you have to plug in a specific context into askQuestions().

Thus, your answerCompleteHandler could look something like this:

var onQuestionsAnswerCompleteHandler = function (answers) {
  for (var i = 0; i < answers.length; i++) {
    var a = answers[i];
    if (a.status == AnswerResponseStatusCode.STATUS_OK) {
      print(
        'The answer to question number ' +
          a.questionId +
          " is '" +
          a.answer +
          "'"
      );
    }
  }
};

This would just print your answer to the logger, but you can just as easily add this text to a screen text component.

Using Dialog Module with Speech Recognition

While the dialog module only has one method, there is a bit more capability built directly into VoiceML, so that you can have them automatically detect questions and feed them into DialogML.

To this end, we first have to let our VoiceML know what context it should use to answer questions, which we do via the ListeningOptions:

var options = VoiceML.ListeningOptions.create();

var qa = VoiceML.QnaAction.create(context);
options.postProcessingActions = [qa];

Where context is the text you want to be used to answer questions.

Pass in these listening options to your Voice ML Module:

var onListeningEnabledHandler = function () {
  script.vmlModule.startListening(options);
};

Don’t forget to bind your functions to the VoiceML’s events

//bind event callbacks
script.vmlModule.onListeningUpdate.add(onUpdateListeningEventHandler);
script.vmlModule.onListeningError.add(onListeningErrorHandler);
script.vmlModule.onListeningEnabled.add(onListeningEnabledHandler);
script.vmlModule.onListeningDisabled.add(onListeningDisabledHandler);

From there on, the VoiceModule handles it and reports back to you in the following snippet;

var onUpdateListeningEventHandler = function(eventArgs)

Where you can access your answer using the following:

var answers = eventArgs.getQnaResponses();

The variable this returns is another list of QnaResponse objects, which are similar to Answers but not the same.

Class QnaResponse

• answer = the textual answer
• answerStatusCode - the answer status which is one of:
  • OK = 1
  • NOT_A_QUESTION = 2
  • NO_ANSWER_FOUND = 3

The methods with and without VoiceML do almost the same thing. Using the VoiceML methods simply cuts out some of the steps

Limitations

There are some known issues in current version of DialogML.

Overconfidence

The model used by Dialog Module can be somewhat overconfident in its ability to answer a question. As a result, it may answer yes or no questions it doesn’t know the answer to with a random yes or no answer. Similarly, if you give it a context like I have two arms and ask it how many legs you have, it may answer “two”.

This can cause some unexpected behavior, but can be ameliorated by making your context more comprehensive.

Comparisons

Currently the model cannot handle comparisons very well. This means if your context was “The moon is 239 miles away. Mars is 103 million miles away”, your model would not know which is farther away. It also cannot answer if the moon is farther than 100 miles away or not.