The 5 Step Journey to Becoming a Chatbot Tester

The best way to get going is to actually build yourself a sample chatbot.


Florian Treml

2 years ago | 3 min read

One question we hear a lot is “What’s the best way to start testing a chatbot? The answer is surprisingly simple: Build up knowledge by educating yourself, and you will be able to answer this question yourself. This post will walk you through the major milestones that test automation engineers have to take for building this knowledge. We will suggest readings and other material that you might find helpful as well.

Step One: Build a First Sample Chatbot

The best way to get going is to actually build yourself a sample chatbot. The reason is that some things like natural language processing (NLP) might appear like magic if you are new to the topic. They aren’t. I suggest to take one of these tutorials — don’t worry, all of them have free plans available:

Go the whole way from starting the project to publishing a chatbot on a preview website — everything you need is included in those cloud platforms.

I do not recommend to start with a free, self-hosted stack like Rasa, as it might distract from the journey to become a chatbot tester by tearing your focus more towards infrastructure topics and actual chatbot development.

Step Two: Understand Machine Learning Concepts and Metrics

When dealing with non-trivial chatbots, there is always a machine learning component for natural language processing (NLP) involved. You do not have to get an expert in machine learning to efficiently test them, but you definitely have to understand the taxonomy used in machine learning projects. The central concepts to learn are:

  • Intents, Entities, Utterances
  • Accuracy, Sensitivity, Specificity, Precision, Recall, F1-Score
  • Confidence and Confidence Threshold
  • Confusion Matrix

Here are some links to introductory articles:

Step Three: Get Comfortable with BotiumScript

You learned how to test websites with Selenium. You learned how to test smartphone apps with Appium. You might be tempted to think A chatbot is basically a website answering questions, why not using Selenium or Appium for testing a chatbot ? Chatbots are a new kind of apps that require new kind of tools to perform effective and high-coverage testing — a goal that cannot be reached with slow, real-time, flaky end-2-end tests.

Wrote about this conflict in one of my previous blog posts.

So invest some days to educate yourself with BotiumScript, an easy-to-learn scripting language for describing chatbot test cases. You can start at the official Botium Docs or watch

Shama Ugale in one of her awesome talks:

Step Four: Get Your Toolstack In Place

In kitchen one of the most important phrases is mise en place — it means to put everything into place before getting started with cooking. Now that you have a good understanding of the basics, it is time to get your tool stack in place you can use for your chatbot testing projects.

To get started in Botium Box, read through the Botium Box Wiki with tutorials, howtos, technical background and reference docs.

Step Five: Understand the Challenges of Chatbot Testing

There are some important differences when testing a chatbot as compared to websites or smartphone apps. What will strike experienced test automation engineers most is that a user can ask a chatbot literally anything (and the users will do as soon as the chatbot is live, believe me). Dealing with non-determinism is a critical topic in software testing. Due to the nature of human language it is impossible for software tests to cover all possible situations.

You can read more in one of my previous blog posts.

Because of this it is even more important to understand the required test levels to meet at least the minimum quality standards in software projects.

Read about some important pitfalls here: 4 DO’s and 3 DON’Ts for Chatbot Testing Strategies

The good news is that you can shine and partially re-use your test automation skills gathered in your previous projects in chatbot testing as well.

Up and Beyond

Wherever you are in your journey, from building your first sample chatbot to running a full-blown continuous chatbot testing pipeline, we’re delighted to have you as part of our Botium community. Please feel free to ask questions on our Botium forum. We look forward to helping you in your journey to become a chatbot testing expert!


Created by

Florian Treml

Worked as software engineer in various domains, including banking, telecommunications, logistics and welfare. First contact with conversational AI in 2015 when developing a chatbot for supporting distant learning. Two years later the Botium journey began with my close friend, Christoph Börner.







Related Articles