Exploratory Testing is a Heuristic Approximation of Cleanroom Statistical Testing
March 13th, 2010I figured this out talking to my friend Daniel earlier this week.
In cleanroom software development, a usage model is created of the expected use of the system. This includes states of usage, system stimulus and probability. It’s basically a markov chain of usage from the user perspective.
This model is used to generate test cases, which consist of lists of stimulus inputs. To be complete, stimulus must include everything that can happen to the software, timing differences, OS interleaving, hardware failures, etc. Often times specific classes of test cases are generated to test features of the product that are unlikely to be encountered in practice but that must provide high levels of robustness.
These test cases are a very small subset of the infinite possible number of test cases from any usage model. Because the model is formalized, specific measurements of reliability can be generated from testing.
Exploratory testing, specifically as professed by the context driven school of testing, acknowledges the same limitations. There are an infinite number of possible test cases to execute in a finite amount of time.
Instead of front-loading the model design process to generate test cases, an exploratory tester actively develops a model of usage as she tests, taking into account risk of potential failure, etc.
This reminds me of the contrast between extreme programming and the design part of cleanroom. Both use an incremental approach to software release, peer reviews and strong specifications (in the form of unit tests in XP, box models in cleanroom). Extreme programming does this actively as the software is made and running, while cleanroom creates it “on paper” before code is generated. In the same way, exploratory testing generates the model actively against the running system, while cleanroom creates the model “on paper” before hand.
Exploratory testing is a heuristic approach of covering a sample the most important test cases to report quality related information while cleanroom is a formalized approach to do the same thing.



