What Testing Is NOT: Common Misconceptions in QA and Test Automation

Introduction

Understanding what testing is not is as important as understanding what it is. Many organizations fail to get value from their quality assurance efforts because they operate under misconceptions about what automated testing and exploratory testing can and cannot do. This article examines the most common misunderstandings and explains how to set realistic goals.

Automated Testing Is Not the Same as Testing

One of the most fundamental misconceptions is that automated testing is equivalent to testing. Referring to Michael Bolton's distinction between testing and checking, automated testing is not really testing—it is checking of facts.

When you have an understanding of the system, you can encode that understanding in the form of automated checks. Running those checks confirms what you already know. Testing, by contrast, is an investigation exercise where the goal is to obtain new information about the system under test through exploration.

Key differences:

Automated checks confirm existing understanding of the system.
Testing requires a human to make sound judgments about usability and to spot anomalies that were not anticipated.
Neither approach should be favored exclusively; both are required to gain insight into application quality.

Automated Testing Is Not Better Than Manual Testing

A common belief is that automated testing is superior to manual testing. This is a misconception. Automated testing and manual testing serve different purposes and complement each other.

Automated checks are good at:

Catching regression issues after changes
Running the same checks repeatedly at regular intervals
Increasing coverage across configurations, browsers, and operating systems

Manual testing is better at:

Spotting unexpected anomalies while exploring the system
Making judgments about usability
Providing quick feedback on a newly developed feature

Organizations should not be lenient toward one method over the other. Both are necessary.

100% Automated Testing Is Not Achievable

Just as there is no practical way of achieving 100% test coverage due to endless possible permutations, the same applies to test automation. You can increase coverage by running automated tests with more data, more configurations, and a wider variety of operating systems and browsers, but achieving 100% remains an unrealistic goal.

More importantly, more tests do not necessarily mean better quality or better confidence. It all depends on how well a test is designed. Instead of aiming for full coverage, focus on the most important areas of functionality that are crucial to the business.

Test Automation Does Not Provide Quick ROI

When implementing a test automation solution, there are other interrelated development activities beyond just scripting test cases. Typically, a framework needs to be developed that supports business-meaningful operations such as:

Test case selection
Reporting
Data-driven testing

The development of the framework is a project in its own right, requiring skilled developers and time to build. Even with a fully functional framework in place, scripting automated checks initially takes longer than executing the same test manually.

For quick feedback on a newly developed feature, checking it manually is usually faster than automating the test. The return on investment (ROI) is realized in the long run, when the same tests need to be executed at regular intervals.

Automated Checks Do Not Guarantee a Higher Rate of Defect Detection

Many vendor-supplied and home-brewed test automation solutions are sophisticated and capable of performing complex operations. However, they will never be able to compete with the intelligence of a human tester who can spot unexpected anomalies while exploring or executing scripted tests.

Ironically, people expect automated testing to find lots of bugs because of allegedly increased test coverage, but in reality this is not the case. Automated tests are good at catching regression issues, but they are not a substitute for human insight in defect discovery.

Exploratory Testing Is Not Ad Hoc Testing

Exploratory testing is often misunderstood as ad hoc testing—testing performed without any upfront planning. This is a misconception.

It is true that exploratory testing does not involve detailed upfront planning of the exact steps to be performed during testing. However, testers usually do some sort of high-level planning where they decide the scope and resources for the testing session. During exploratory testing, testers learn continuously and use that learning to drive their testing based on the feedback they receive at the moment they receive it.

Exploratory Testing Is Not Random Testing

Another major misconception is that exploratory testing is about testing randomly. This is completely untrue. Testers do not test without a method or without making conscientious decisions.

In exploratory testing, every single decision is driven by:

Knowledge of the product
Its context
The feedback received while exploring or trying it out

Testers also start with a goal, some identified risks, and a test charter in mind. Exploratory testing is therefore driven by knowledge, not randomness. To uncover important information about the target system, testers need to use testing skills and techniques wisely, considering the product, its users, and the business context.

Exploratory Testing Is Not Simply a Testing Technique

Framing exploratory testing as a testing technique depends on what is meant by "testing technique."

If a testing technique is understood as a way of performing testing that requires skills, then exploratory testing qualifies.
If a testing technique is understood as a procedure to complete a specific task, then it becomes trickier, because there is no specific task to achieve in exploratory testing—unless the task is seen as finding relevant information about the quality of the target system.

Known testing techniques such as boundary-value analysis (BVA) and pairwise testing exist to address very specific goals, like generating test cases with combinations for all pairs of variables. Exploratory testing is broader: several testing techniques can be used during exploratory testing.

Summary of What Testing Is NOT

Misconception	Reality
Automated testing is testing	Automated testing is checking of facts; testing is investigation
Automated testing is better than manual testing	Both are required and serve different purposes
100% automated testing is achievable	Endless permutations make full coverage unrealistic
Test automation provides quick ROI	Framework development takes time; ROI comes in the long run
Automated checks find more defects	Human testers are better at spotting unexpected anomalies
Exploratory testing is ad hoc testing	It involves high-level planning of scope and resources
Exploratory testing is random testing	Every decision is driven by knowledge, goals, risks, and a charter
Exploratory testing is just a technique	It is a broader approach in which multiple techniques can be used

Conclusion

Setting realistic expectations is possibly the most difficult and challenging aspect of any testing endeavor. Understanding the limitations of automated testing and exploratory testing helps organizations avoid disappointment and invest their QA efforts where they will have the most impact. Automated checks confirm what you already know; human testers discover what you do not.