By James Whittaker

The sixth ingredient is variation. Tests often get stale (i.e., they stop finding bugs) as they are run over and over on build after build as a product is being constructed. On the one hand, it is important to continue running tests to ensure the product still operates as specified. Indeed, I hesitate to throw any test away. However, becoming reliant on stale tests is too risky. Adding variation to existing tests can range from straightforward reordering of the sequence in which tests are run to more involved solutions of either modifying tests or adding new ones. Hopefully new tests will increase overall coverage and add to our confidence that we’ve tested the software in all the ways it needs to be tested.