Chatbot Agent Testing using Botium

Building a chatbot agent for a business is relatively easy and creating a great autonomous user experience is a genuine hurdle. The importance of chatbot agent testing cannot be overstated in the rapidly evolving landscape of conversational AI. Chatbot agents use Natural Language Processing (NLP) for talking and typing interactions to perform specific tasks.

As a QA engineer, ensuring the reliability and performance of our chatbot agent is crucial to delivering a seamless user experience. Chatbot agent testing differs from traditional testing, mainly like interactions. Unlike conventional applications with predefined user journeys, chatbots operate without strict pathways. In traditional testing, interactions follow a structured flow, whereas chatbot agent testing involves analyzing responses to diverse and unpredictable user inputs and navigating dynamic and context-driven conversations.

In one instance, we used to manually test a bot developed with Google Dialog flow, which was really time-consuming. Moreover, testing dynamic responses manually posed additional challenges. To address this complexity, we turned to Botium for its capability to handle and automate the testing of our bot effectively.

Botium is an ideal tool for chatbot agents automated testing, because it simplifies the entire process. It allows us to quickly and effectively test the responses of our chatbot agent without the difficulties of manual testing.

Furthermore, it specializes in handling the dynamic nature of chatbot interactions, making the testing process smoother and more reliable. In basic terms, Botium ensures that our chatbot agent works properly without requiring us to manually test everything.

Botium is similar to a handy toolkit for handling all testing-related tasks. By integrating all the necessary tools in one location, it simplifies the testing procedure.

Botium Testing Toolkit

+-----------------+--------------------------------------------------------+
| Botium Core     | The Heart and Brain of Botium                          |
| Botium CLI      | The Swiss Army Knife of Botium                         |
|                 | (the Botium CLI is a command-line interface designed to|
|                 |   make testing chatbots easier and more efficient).    |
| Botium Bindings | The glue to bind Botium to test runners.               |
| Botium Box      | It's a comprehensive platform that, with its           |
|                 | user-friendly interface and advanced testing tools,    |
|                 | simplifies and improves chatbot testing.               |
| Botium Crawler  | Like a website crawler.                                |
+-----------------+--------------------------------------------------------+

As a flexible testing framework, Botium supports a large variety of chatbot systems like,

• Google Dialog Flow (GDF)
• Amazon Lex
• IBM Watson Assistant
• Rasa
• Cisco Webex Teams
• Microsoft Bot Framework
• Facebook Messenger

Now, let’s get a quick understanding of Botium CLI, which is a vastly used command-line tool for chatbot testing.

Botium CLI

To setting up the environment:

1. Node.js — Ensure that you have Node.js installed on your machine.
2. An IDE for writing scripts (VS Code/IntelliJ IDEA/Eclipse and etc.)
3. Install Botium and Dialogflow CX Connector.

npm install -g botium-cli

npm install -g botium-connector-dialogflowcx

4. Initialize the botium project.

botium-cli init

5. Open the file called botium.json in your working directory and add the Google credentials (Project Id, Agent Id and Location) for accessing your Dialogflow agent. You can be found these details from the Dialogflow CX Console. Verify that the user is authorized to perform the action. The resourcemanager.projects.getIamPolicy permission is required to view details.

Get Shashani Dhanujika’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

And, Botium supports various reporting mechanisms. We can use MochaAwesome for test execution reports and Allure for detailed and interactive reports.

Let’s write this simple example using Botium

The following image shows how we test chatbot agents using Google Dialog flow manually. Let’s have a look using a simple example.

In Botium, we call test cases as convo files. We use the Botium script to write test cases. It allows testers to define conversational test cases in a human-readable format.

Dialogflow CX chatBot agent

Botium supports test scripts using several file formats, Such as Text files, CSV files, Excel files, Yaml files. With any of these file formats, we should use the ‘.convo.’ word as per the example given below.

Example: sample_testcase1.convo.txt

sample_testcase1.convo.txt

The very first line mentions the test name, and ‘#me’ denotes user inputs, while ‘#bot’ represents agent responses.

We can write test cases in scripting files, and we can also automatically generate files using the Botium simulator that we shall discuss in a separate article.

Utterances

Dialogflow CX chatBot testing — manual

As it was mentioned earlier, both user inputs and chatbot agent responses are not static. As it is shown in the above screenshot, we’ve chatted with our test chatbot agent using different kinds of greeting messages. They are dynamic.

Moreover, we can easily handle these kinds of scenarios using Botium utterance files. (Utterances file formats — Text files, CSV files, Excel files, Yaml files)

Now, let’s write this test using an utterance file. By using utterance files, we only need to add the test data set for what we want to test, and we can simply call the reference code (REFCODE) into the test script. With this utterance file, we should use the ‘.utterances.’ word as per the example given below.

example: greeting_utt.utternces.xlsx

greeting_utt.utternces.xlsx

Let’s write the convo file for this example.

sample_testcase2.convo.txt

After executing, we can see Botium automatically run tests with all the utterances in the utterance file. As revealed, it’s really easy to maintain huge number of testcases with botium.

Test Execution

There are four botium concepts that can be worked with. Utterances is one of them, and we shall go through other Botium concepts and botium.json and package.json files in Botium using another article.

Conclusion

Botium is a valuable testing toolkit for chatbot agents, offering automation, flexibility, and support for diverse conversational AI platforms. As a QA engineer, the experience of using Botium for chatbot agent testing is highly positive and enthusiastic.