Using Microsoft Orchestrator for Intent Recognition and Dispatch to Luis

A common challenge for voice assistants is correctly recognizing the intent in user utterances. This can be especially challenging for general-purpose, multi-domain voice assistants. For example, imagine an assistant that is designed to be a friendly companion, in addition to being a useful source of content, and a helpful interface for smart home systems. Such an assistant will need to be able to classify a wide range of utterances, like:

• what’s the weather tomorrow?
• do you know any martian jokes?
• turn on the living room lights
• i’m feeling lonely
• play the latest song by taylor swift
• it’s great to see you
• good morning, buddy
• have a good night

Language understanding systems need to extract at least two kinds of information from the utterance: 1) the “intent” of the utterance — the general meaning of the utterance and 2) any discrete pieces of information in the utterance. These discrete pieces of information are often called “entities”.

An utterance like “turn on the living room lights” might be classified something like:
- intent: turnOnLights
- entities: { target: “livingRoom” }

With this information, a simple program can be invoked to turn on the specified lights.

A language understanding system like Microsoft Luis is powerful because it can be used to extract both intents and entities using one model — in one shot. However, it tends to be impractical to use one, big Luis model to classify a large number of intents. As the size of a model increases, it becomes harder to make changes without introducing errors. Adding training data to make the model better at recognizing one intent can inadvertently make it worse at recognizing other intents. Technologies like Luis can be mysterious. When they produce the intended results, all is good. When they don’t, it can be difficult to figure out what went wrong. In general, troubleshooting is easier with smaller models.

Dispatch

A common way to address the problem is to create a “dispatch” model that determines the general class of the user utterance as a first step. Then the utterance can be sent to a simpler, class-specific model to determine the intent and entities.

Not long ago the best practice was to use an ML technology like Luis to build a big dispatch-only model trained with example utterances from a collection of smaller models. This makes sense and it can work well. But Luis models often get confused by similarly-worded examples that should map to different intents. For example: In a large, complex model, “play auld lang syne” (Music) and “play space invaders” (Games) could easily be confused and misclassified because the structure of the utterances is so similar. As a result, using Luis for Dispatch has been deprecated in favor of using a different kind of model technology— namely deep neural nets or DNNs.

Orchestrator

In early 2021, Microsoft began previewing a technology called “Orchestrator” as the new best practice for dispatch scenarios. (See: Intent recognition with Orchestrator in Composer ) Orchestrator was first made available as part of the Microsoft Bot Framework to help chatbots choose the most appropriate Luis model based on the chat context. Orchestrator makes use of pre-trained DNN models to classify utterances. [oversimplification ahead] It does this by locating utterances in a vast, multi-dimensional vector space defined by all of the data in the DNN training set.

The typical way to use Orchestrator with Luis is to first calculate location vectors for example utterances extracted from a set of domain-specific Luis models — remembering the model (or class) from which each example originated. Then at runtime, a location vector is calculated for each new user utterance and compared to the examples. The example that is mathematically closest to the user utterance (in the vast multi-dimensional space of the DNN) is used to determine which domain-specific Luis model to use next — i.e. which Luis model to “dispatch” to.

Like the ML technology used by Luis, deep neural nets (DNNs) are also mysterious black boxes. But the way the Orchestrator DNN locates utterances in the vector space seems to be less dependent on specific words and word order than Luis. So it is well-suited to the dispatch task. For example, Orchestrator can recognize that “Where am I?” is semantically similar to “What town is this?” even though the examples do not share any words or grammatical structure.

Understanding Orchestrator

To better understand and explore the potential of Orchestrator I created an demonstration project that includes several examples of building and testing a dispatch classifier. The examples use Orchestrator in three different ways:

1. Via the Bot Framework cli tool, bf
2. Via JavaScript in a Jupyter Notebook
3. Wrapped in a node microservice, optionally deployed with Docker

All of these examples are available in this github repo: https://github.com/wwlib/orchestrator-microservice

Example 1: Using Orchestrator via the Bot Framework Cli Tool

First, clone the examples repo and then…

cd orchestrator-microservice
npm install
cd orchestrator-files

To see the available DNN models…

npx bf orchestrator:basemodel:list

This will return a list of available models, including:

• pretrained.20200924.microsoft.dte.00.06.en.onnx
• pretrained.20210205.microsoft.dte.00.06.unicoder_multilingual.onnx
• pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx

Download (get) a model and save it to the ./model folder…

npx bf orchestrator:basemodel:get — versionId=pretrained.20210205.microsoft.dte.00.06.unicoder_multilingual.onnx — out=./model

The next step is to generate a snapshot (.blu) file. To get started you can use the example Luis (.lu) files in orchestrator-files/lu/Dispatch. These incomplete example models represent the kinds of distinct domains that a general-purpose voice assistant might need to handle, including:

• Fitness.lu — a model for recognizing utterances about fitness and exercise
• Games.lu — a model for recognizing utterances about games
• Music.lu — a model for recognizing utterances about music
• SmallTalk.lu — a model for recognizing general small talk
• Time.lu — a model for recognizing utterances about the time, dates, holidays, etc.

cd [project-root]/orchestrator-files
npx bf orchestrator:create --hierarchical --in ./lu/Dispatch --out ./blu/Dispatch --model ./model

Dispatch.blu will contain lines like:

LABEL       EXAMPLE                   CLASSIFICATION_VECTOR
Fitness     teach me how to do yoga   FA16EBDB45...
SmallTalk   are you able to be wrong  D1B2ED9AC7...
SmallTalk   can you be happy          F330E5DA4D...
SmallTalk   do you ever feel happy    711065DA82...
Time        tell me the time          791301F206...

Each example from the supplied .lu files will be labeled and assigned a location vector.

Again, for each new user utterance, a vector is calculated and compared to the examples’ vectors.

Now you can test the snapshot with the bf cli:

• cd [project-root]/orchestrator-files
• npx bf orchestrator:query --in=./blu/Dispatch.blu --query="what time is it" --model=./model

Output for the query, “what time is it”

[
  {
    "label": {
      "name": "Time",
      "label_type": 1,
      "span": {
        "offset": 0,
        "length": 15
      }
    },
    "score": 0.9997169968564052,
    "closest_text": "what time is it"
  },
  ...
]

The output is a ranked list of results, one for each model used to define the Dispatch.blu snapshot

The results are ranked by how close the user utterances is to the example utterances. Each result’s label is the name of the model where the example came from — in this case, the “Time” model.

Example 2: Using Orchestrator in a JavaScript App via Jupyter Notebook

The following Jupyter Notebook makes use of the Microsoft orchestrator-core npm module to utilize Orchestrator from JavaScript.

https://github.com/wwlib/orchestrator-microservice/blob/main/notebooks/orchestrator-example.ipynb

Note: The code for harnessing Orchestrator via JavaScript is based on this class from the Bot Framework SDK: https://github.com/microsoft/botbuilder-js/blob/main/libraries/botbuilder-ai-orchestrator/src/orchestratorRecognizer.ts

If you have python, jupyter, node and npm installed you can execute the JavaScript code in the the notebook by also installing ijavascript:

npm install -g ijavascript
ijsinstall
cd [project-root]/notebooks
jupyter notebook

Note: You don’t need to run the notebooks to see the examples and documentation.

The code from the notebook is also provided as script in the repo: https://github.com/wwlib/orchestrator-microservice/blob/main/orchestrator-files/test-example.js

In addition to loading the required npm modules, the notebook defines two variables:

modelFolder = '../orchestrator-files/model';
snapshotFile = '../orchestrator-files/blu/Dispatch.blu';

The initializeResolver() function instantiates an orchestrator instance and loads the model found at path specified by modelFolder. Then it instantiates a resolver instance using the snapshot file (.blu) specified by snapshotFile.

Once initialized, the resolver can be used to score utterances. For example, the utterance, “I need to work out”, can be scored by calling resolver.score(utterance, 1)

results = resolver.score(utterance, 1);

The resolver returns a ranked list of results indicating which which of the snapshot labels are “closest” to the utterance. In this case, the top ranked result indicates that the utterance is most like the examples from the Fitness model.

{
  closest_text: 'want to work out',
  score: 0.892648646722145,
  label: { name: 'Fitness', label_type: 1, span: [Object] }
}

Example 3: Wrapping Orchestrator in a Microservice

The examples repo also includes an example of Orchestrator wrapped in a node-based microservice. To install and build the microservice:

cd [project-root]
npm install
npm run build

To download the model, generate the snapshot file and start the service:

npm run get model
npm run generate
npm run start

Then the service can be tested with curl:

curl --location --request POST 'http://localhost:8000/intent' \
     --header 'Content-Type: application/json' \
     --data-raw '{
       "utterance": "what do you want to do today",
       "resolverName": "Dispatch"
     }'

The results will look like:

[
    {
        "closest_text": "what plans do you have for today",
        "score": 0.7299530143026409,
        "label": {
            "name": "SmallTalk",
            "label_type": 1,
            "span": {
                "length": 28,
                "offset": 0
            }
        },
        "processingTime": 422,
        "resolver": "Dispatch"
    },
...
]

Or with Postman:

Or with the included socket client:

cd [project-root]/tools
node socket-cli.js

Finally, the microservice implements a simple Web UI. Choose the Resolver from a pulldown and type an utterance to test, then hit RETURN (or click the Submit button)

There are three resolver options:

• Dispatch — uses the Dispatch.blu snapshot made from the three example Luis models: Fitness.lu, Games.lu, Music.lu, SmallTalk.lu and Time.lu
• SmallTalk — uses the SmallTalk.blu snapshot made from a set of Luis models extracted from the SmallTalk.lu model — one .lu file per intent.
• 2-Stage — uses 2 instances of Orchestrator to accomplish a 2-stage dispatch: Stage1 determines the model label and then if the model label is “SmallTalk” Stage 2 determines the SmallTalk-specific intent.

The goal with Orchestrator is to dispatch utterances to the most appropriate, domain-specific Luis model which will then determine the intent and entities. In this example, the Fitness.lu, Games.lu, Music.lu, and Time.lu models are relatively simple with only a handful of intents each. The SmallTalk.lu model is a little more complex and has ~15 intents of its own so to demonstrate a 2-stage dispatch, the SmallTalk.lu model has been broken into ~15 smaller .lu files — one per intent. Then a SmallTalk-specific snapshot (.blu) has been generated and used to initialize the SmallTalk resolver.

Get Andrew Rapo’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Note: A more realistic SmallTalk model might have hundreds of intents. That is when a 2-Stage dispatch would really be necessary.

Using the microservice Console UI (http://localhost:8000/console) you can test the three Resolver options with an example utterance.

The results for “what do you want to do today” are:

Dispatch resolver

{
  "resovler": "Dispatch",
  "label": "SmallTalk",
  "closest_text": "what do you want to do this weekend",
  "score": 0.9354338696921423,
  "processingTime": "18 ms"
}

The utterance is correctly labeled, “SmallTalk”

SmallTalk resolver

{
  "resovler": "SmallTalk",
  "label": "doesHavePlansForEvent",
  "closest_text": "what do you want to do this weekend",
  "score": 0.9053338150950618,
  "processingTime": "20 ms"
}

The utterance is correctly labeled with the SmallTalk intent, “doesHavePlansForEvent”

2-Stage resolver

{
  "resolver": "Dispatch -> SmallTalk",
  "label": "SmallTalk -> doesHavePlansForEvent",
  "closest_text": "what do you want to do this weekend",
  "score": 0.9053338150950618,
  "processingTime": "37 ms"
}

The 2-stage dispatch uses the Dispatch resolver to correctly return the SmallTalk model label and then automatically uses the SmallTalk resolver to correctly determine the intent, “doesHavePlansForEvent”

Note: Results may vary if a different base DNN model is used. The model used for these examples is: pretrained.20210205.microsoft.dte.00.06.unicoder_multilingual.onnx

A Workflow for Developing Orchestrator Dispatch Solutions

Developing an Orchestrator dispatch solution is an iterative process. Every time the input examples (.lu files) change the outputs can change in unexpected ways. New errors can emerge. It is helpful to have a workflow that includes at least some sanity-check tests.

Sanity-check Tests

The orchestrator-example notebook in the project repo includes a simple testing script. The script loads tests from a json test manifest file and then makes calls to the microservice for each test.

The SmallTalk-manifest.json file includes tests that look like:

{
    "utterance": "when is he arriving",
    "label": "Time"
},
{
    "utterance": "I need some exercise",
    "label": "Fitness"
},
{
    "utterance": "did you see the game",
    "label": "SmallTalk:didWatchEvent"
},

For each test utterance the expected Label is provided.

The script described in this notebook can be run via: npm run test-microservice

$ npm run test:microservice
> orchestrator-microservice@0.0.1 test:microservice /[...]/orchestrator-microservice
> node test/test-microservice.js
manifestPath: ./orchestrator-files/test-manifests/SmallTalk-manifest.json
resolverName: 2-Stage
37 tests completed in 952ms. Average test time: 25.72972972972973ms
0 errors.
$

Splitting the SmallTalk.lu File Into Per-intent .lu Files

The experimental 2-Stage dispatch approach described above requires that the SmallTalk.lu file be split into a set of .lu files — one per intent.

The intents-to-lu-files notebook in the project repo describes a script for splitting the SmallTalk.lu file.

The script described in this notebook can be run via: npm run intents-to-lu-files

$ npm run intents-to-lu-files
> orchestrator-microservice@0.0.1 intents-to-lu-files /[...]/orchestrator-microservice
> cd orchestrator-files; node intentsToLuFiles.js
Splitting ./lu/Dispatch/SmallTalk.lu into individual .lu files...
Writing: ./lu/SmallTalk/didWatchEvent.lu
Writing: ./lu/SmallTalk/doesBelieveInThing.lu
Writing: ./lu/SmallTalk/doesCareAboutEvent.lu
Writing: ./lu/SmallTalk/doesFeelEmotionAboutThing.lu
Writing: ./lu/SmallTalk/doesHaveFavorite.lu
Writing: ./lu/SmallTalk/doesHaveLeastFavorite.lu
Writing: ./lu/SmallTalk/doesHaveOpinionAboutThing.lu
Writing: ./lu/SmallTalk/doesHavePlansForEvent.lu
Writing: ./lu/SmallTalk/doesHaveRelation.lu
Writing: ./lu/SmallTalk/doesKnowPerson.lu
Writing: ./lu/SmallTalk/greeting.lu
Writing: ./lu/SmallTalk/howQuestion.lu
Writing: ./lu/SmallTalk/whatQuestion.lu
Writing: ./lu/SmallTalk/whereQuestion.lu
Writing: ./lu/SmallTalk/whoQuestion.lu
Writing: ./lu/SmallTalk/whyQuestion.lu
Writing: ./lu/SmallTalk/userFeels.lu
Wrote 17 .lu files to: ./lu/SmallTalk/
Done.
$

An Example Workflow

The workflow used to test the examples for this article includes these steps:

1. Download the DNN model with: npm run get-model
2. Split the SmallTalk.lu file into per-intent .lu files with: npm run intents-to-lu-files
3. Generate the snapshot (.blu) files with: npm run generate
4. Start the microservice with: npm run start
5. Run the sanity-check tests with: npm run test:microservice
6. Review the results and make any needed changes to Fitness.lu, Games.lu, Music.lu, SmallTalk.lu or Time.lu
7. Stop the microservice (i.e. ctrl-c)
8. Repeat (Goto Step 2)

Running the Microservice via Docker

The project repo includes a Dockerfile. To build and run the microservice with Docker:

Build

docker build -t orchestrator-microservice .

Or:

npm run docker:build

Run

docker run -it --rm -p 8000:8000 --env-file ./.env orchestrator-microservice

Or:

npm run docker:run

Note: 2gb should be enough to run the pretrained.20200924.microsoft.dte.00.06.en.onnx model.

However, if more memory is needed you may need to try something like:

docker run -it -m=4g --rm -p 8000:8000 --env-file ./.env orchestrator-microservice

Multilingual Recognition

When using a multilingual DNN model like, pretrained.20210205.microsoft.dte.00.06.unicoder_multilingual.onnx, Orchestrator can classify utterances from a a variety of languages. For example, Orchestrator will assign both“What time is it?” and “Que hora es?” the same label — int this case, “Time”

Limitations

Orchestrator’s available DNN models are powerful but not perfect. It is easy to find misclassifications. In this example project, “Do you know david?” is correctly labeled as SmallTalk:doesKnowPerson, closest to “are you familiar with david”

{
  "resolver": "Dispatch:SmallTalk",
  "label": "SmallTalk:doesKnowPerson",
  "closest_text": "are you familiar with david",
  "score": 0.6574666605549864,
  "processingTime": "21 ms"
}

However, “Do you know joe?” is incorrectly labeled as: Time, closest to “i want to know the date”

{
  "resolver": "Dispatch",
  "label": "Time",
  "closest_text": "i want to know the date",
  "score": 0.3328182735289674,
  "processingTime": "12 ms"
}

Interesting. Also “what do you want to do today” is correctly classified as: SmallTalk:doesHavePlansForEvent, closest to “what plans do you have for today”.

{
  "resolver": "Dispatch:SmallTalk",
  "label": "SmallTalk:doesHavePlansForEvent",
  "closest_text": "what plans do you have for today",
  "score": 0.7238375153677875,
  "processingTime": "21 ms"
}

But, “what do you want to do today?” — with a question mark — is incorrectly labeled as: Time, closest to “do you have the current day”

{
  "resolver": "Dispatch",
  "label": "Time",
  "closest_text": "do you have the current day",
  "score": 0.747118860912231,
  "processingTime": "7 ms"
}

Because DNNs are black boxes, it is hard to know what is going on with these examples. And because Orchestrator is so new there is not a lot of documentation or discussion about its inner workings. But that will change, especially if more people begin to experiment with it.

Summary

Microsoft’s Orchestrator is a powerful tool for classifying utterances. It can run in the cloud or on a sufficiently powerful device. It is easy to train/configure. And as demonstrated by this article it can be used independent of Microsoft Bot Framework — within the the terms of the Microsoft License. The examples in this article and accompanying repo will help to jumpstart the process of learning and experimenting, especially with dispatch scenarios.