Cerrisian Blog

I figured this out talking to my friend Daniel earlier this week.

In cleanroom software development, a usage model is created of the expected use of the system. This includes states of usage, system stimulus and probability. It’s basically a markov chain of usage from the user perspective.

This model is used to generate test cases, which consist of lists of stimulus inputs. To be complete, stimulus must include everything that can happen to the software, timing differences, OS interleaving, hardware failures, etc. Often times specific classes of test cases are generated to test features of the product that are unlikely to be encountered in practice but that must provide high levels of robustness.

These test cases are a very small subset of the infinite possible number of test cases from any usage model. Because the model is formalized, specific measurements of reliability can be generated from testing.

Exploratory testing, specifically as professed by the context driven school of testing, acknowledges the same limitations. There are an infinite number of possible test cases to execute in a finite amount of time.

Instead of front-loading the model design process to generate test cases, an exploratory tester actively develops a model of usage as she tests, taking into account risk of potential failure, etc.

This reminds me of the contrast between extreme programming and the design part of cleanroom. Both use an incremental approach to software release, peer reviews and strong specifications (in the form of unit tests in XP, box models in cleanroom). Extreme programming does this actively as the software is made and running, while cleanroom creates it “on paper” before code is generated. In the same way, exploratory testing generates the model actively against the running system, while cleanroom creates the model “on paper” before hand.

Exploratory testing is a heuristic approach of covering a sample the most important test cases to report quality related information while cleanroom is a formalized approach to do the same thing.

So, I’ve been having some crazy ideas on how to teach a programming class. First, I’d used a spaced repetition selection for homework and potentially quiz questions.

Each homework would be cumulative, with the majority of the questions coming from the current chapter and the previous chapter, but with problems selected from even earlier chapters either weighed so that they increase in difficulty over time or based on what students had trouble with. To aid in this, certain classes of problems would be randomly generated from a homework question bank, which would be a database that ran on my desktop.

Tests would work similarly, potentially with questions randomly selected from the bank for each test, so that each student got an individualized problem. Problems would be identified using a hashcode and solutions to the problems would also be automatically generated.

Tests would include a live coding part done in a lab. Homework assignments would be turned in to subversion.

So, the basic idea for a contest with Engineering Challenge is to have the student teams working at computers with software to create robot AIs. These computers act as clients to a central server which displays results and matches between the AIs. This is the core of the competition.

This networked setup can also be used during the training/warmup session. Each client has a number of simple directions and scenarios built in that it auto grades. As students complete the automatic training, a status message is sent to a different server that allows the facilitators to see which students are behind and need help.

These scenarios are built in a constructivist manner to guide the students through all the features necessary for the competition.

Before the students begin, they input their names, team name, school and possibly seating position so that this information shows up in the necessary displays.

This is more complicated, but it should make running the contest a snap.