One question we hear a lot is “What’s the best way to start testing a chatbot? ” The answer is surprisingly simple: Build up knowledge by educating yourself, and you will be able to answer this question yourself. This post will walk you through the major milestones that test automation engineers have to take for building this knowledge. We will suggest readings and other material that you might find helpful as well.

Step One: Build a First Sample Chatbot

The best way to get going is to actually build yourself a sample chatbot. The reason is that some things like natural language processing (NLP) might appear like magic if you are new to the topic. They aren’t. I suggest to take one of these tutorials — don’t worry, all of them have free plans available:

Quickstart: Build an agent (Google Dialogflow)
Getting started with Watson Assistant (IBM Watson Assistant)

Go the whole way from starting the project to publishing a chatbot on a preview website — everything you need is included in those cloud platforms.

I do not recommend to start with a free, self-hosted stack like Rasa, as it might distract from the journey to become a chatbot tester by tearing your focus more towards infrastructure topics and actual chatbot development.

Step Two: Understand Machine Learning Concepts and Metrics

When dealing with non-trivial chatbots, there is always a machine learning component for natural language processing (NLP) involved. You do not have to get an expert in machine learning to efficiently test them, but you definitely have to understand the taxonomy used in machine learning projects. The central concepts to learn are:

Intents, Entities, Utterances
Accuracy, Sensitivity, Specificity, Precision, Recall, F1-Score
Confidence and Confidence Threshold
Confusion Matrix

Here are some links to introductory articles:

Step Three: Get Comfortable with BotiumScript

You learned how to test websites with Selenium. You learned how to test smartphone apps with Appium. You might be tempted to think A chatbot is basically a website answering questions, why not using Selenium or Appium for testing a chatbot ? Chatbots are a new kind of apps that require new kind of tools to perform effective and high-coverage testing — a goal that cannot be reached with slow, real-time, flaky end-2-end tests.

Wrote about this conflict in one of my previous blog posts.

So invest some days to educate yourself with BotiumScript, an easy-to-learn scripting language for describing chatbot test cases. You can start at the official Botium Docs or watch

Shama Ugale in one of her awesome talks:

https://www.youtube.com/watch?v=qIetL10YAU0

Step Four: Get Your Toolstack In Place

In kitchen one of the most important phrases is mise en place — it means to put everything into place before getting started with cooking. Now that you have a good understanding of the basics, it is time to get your tool stack in place you can use for your chatbot testing projects.

Subscribe to a free-forever Botium Box Mini instance to optimize your learning curve — Botium Box includes everything to get you started, all you need is a web browser
For command line aficionados, install Botium CLI, the swiss-army-knife for chatbot testing
If you are used to test runners like Mocha, Jest or Jasmine, you will love Botium Bindings

To get started in Botium Box, read through the Botium Box Wiki with tutorials, howtos, technical background and reference docs.

Step Five: Understand the Challenges of Chatbot Testing

There are some important differences when testing a chatbot as compared to websites or smartphone apps. What will strike experienced test automation engineers most is that a user can ask a chatbot literally anything (and the users will do as soon as the chatbot is live, believe me). Dealing with non-determinism is a critical topic in software testing. Due to the nature of human language it is impossible for software tests to cover all possible situations.

You can read more in one of my previous blog posts.

Because of this it is even more important to understand the required test levels to meet at least the minimum quality standards in software projects.

Regression Testing on API level— Identify flaws in the conversation flow before going to production
NLP Testing — Improve your chatbot understanding
E2E Testing — Verifying the end-user experience
Voice Testing — Understand your users on voice channels
Performance Testing — Ensure your chatbot is responsive under high load
Security Testing — Making your chatbot secure
Monitoring — Get notified when problems arise

Read about some important pitfalls here: 4 DO’s and 3 DON’Ts for Chatbot Testing Strategies

The good news is that you can shine and partially re-use your test automation skills gathered in your previous projects in chatbot testing as well.

Up and Beyond

Wherever you are in your journey, from building your first sample chatbot to running a full-blown continuous chatbot testing pipeline, we’re delighted to have you as part of our Botium community. Please feel free to ask questions on our Botium forum. We look forward to helping you in your journey to become a chatbot testing expert!