Working with LUIS.ai

Intro

Need to derive the intent from a client ??  Azure's offering for natural language processing or NLP is LUIS.ai. LUIS (https://docs.microsoft.com/en-us/azure/cognitive-services/luis/what-is-luis) provides a restful api that allows a client to send utterances to LUIS for it to run against a model that has been built and published. That model can be a prebuilt model or custom one generated by you. Models have intents that have supporting utterances which can have an entity. The basics about these can be found in the link above. Below I will give you my high level perspective.

Intents :

Intents reflect to actions as in ViewCart or FindRooms, the utterances that support the intent need to be crafted to support the target audience.

The utterances should support each other.,

• What rooms are available tomorrow
• Can you find a room for Thursday until Sunday
• I want a room for Friday for 2 nights

I these three examples we see that each has a unique way of asking for availability.

To improve the strength of these utterances we could add some supporting utterances 

• What rooms are available tomorrow
• What are the available rooms for Saturday
• Can you find a room for Thursday until Sunday
• Can you find two rooms for Friday
• I want a room for Friday for 2 nights
• I want a room today for me

 if you find that you have a set of utterances that do not support another this could be a place for a separate intent. 

The utterances should be balanced between the intents. An intent that is weighted more than others can skew your results.

If you are having a performance issue check the count of utterances in your intents. Large intents take more processing to come to a results.

Focus your model to be as direct as possible. The more abstract the models purpose is, larger the model will need to be, and the worse the performance of the model is

Entities

Are to parameters that are required to fulfill the utterance that was sent to LUIS, in the utterance "I want a room for Friday for 2 nights" we have two entities one Date = Friday and the other is duration = 2 nights. With a confidence match to this intent Luis would send back the confidence score and the matched entities.

In this scenario we can use a composite entities to tie these two entities together under a parent entity

Validation, Testing and Optimization

When creating a LUIS model it’s a best practice to reserve ~20% of the training data for validation and another ~20% to completing test against the model to validate the models correctness, recall, and subsequent f-measure.

With your test data import it in to LUIS as a batch test. A batch test allows you to rerun the test and it provides insights to what is working and what is not working in your model, LUIS is a bit finicky about the batch size, the system states that is can handle 1000 test utterances, but I have only been able to import 100, you may have to tune the size of your test data by import trial and error. 

Once you run the test you will find what is working and what is not. 

A deeper look will give provide insight into what's failing in the test data, to validate if the failures are valid or not so that we can take appropriate action. 

The results of the test are summed up in to the intents and entities of the model. The utterances can fall in to one of four categories, True positives, False positives, True negatives, False negatives.

If the utterance matches the labeled intent then it will be a True positive. If the utterance does not match the labeled intent and it does not match the intent that you are reviewing then this is a True negative.

These are the states that we are striving for in our model. By looking at the False sections we will improve our model to be more resilient. Once the model has been updated or the test data has been nullified, a second pass should be run with fresh data to confirm you updates are able to handle the variant utterances and not just the training data.

As you can see this process ends up consuming you test and validation data quite quickly.

Another useful tool is https://qbox.ai/ . This tool reviews your model and uses the utterances in the other intents to find the where there is low confidence in an intent, or intent overlap that can cause integrity issues with the perception of your model.

Conclusion

NLP tools like LUIS are very useful when used properly. The proper use is key, as a newer technology the trend is to use it to try to solve problems where it does not fit. This is a common issue, but it can be a bit more dangerous since the skill set to build good models is not the traditional skills that a developer has. When we are building utterances, mastery of the language is required. This is an interesting domain that will provide many new opportunities moving forward