Article covering some of Google's Test culture. In the article, there's some focus on the ideas of "incremental testing" and how practices are changing in the software as a service world. Even with all of our drive to find better approaches, one thing to note is that we still believe in the fundamentals of testing. Here's a direct link (they use a bunch of indirection to get to the PDF. So if you can't get it directly, here's the top level link).
IEEE Article
Monday, February 25, 2008 9:24 AM
TotT: Too Many Tests
Thursday, February 21, 2008 9:58 AM
In the movie Amadeus, the Austrian Emperor criticizes Mozart's music as having “too many notes.” How many tests are “too many” to test one function?
Consider the method decide:
public void decide(int a, int b, int c, int d,
int e, int f) {
if (a > b || c > d || e > f) {
DoOneThing();
} else {
DoAnother();
} // One-letter variable names are used here only
because of limited space.
} // You should use better names. Do as I say, not
as I do. :-)
How many tests could we write? Exercising the full range of int values for each of the variables would require 2192 tests. We'd have googols of tests if we did this all the time! Too many tests.
What is the fewest number of tests we could write, and still get every line executed? This would achieve 100% line coverage, which is the criterion most code-coverage tools measure. Two tests. One where (a > b || c > d || e > f) is true; one where it is false. Not enough tests to detect most bugs or unintentional changes in the code.
How many tests to test the logical expression and its sub-expressions? If you write a test of decide where a == b, you might find that the sub-expression a > b was incorrect and the code should have been a >= b. And it might make sense to also run tests where a < b and a > b. So that's three tests for a compared to b. For all of the parameters, that would 3 * 3 * 3 = 27 tests. That's probably too many.
How many tests to test the logical expression and its sub-expressions independently? Consider another version of decide, where the logical sub-expressions have been extracted out:
public void decide(int a, int b, int c, int d,
int e, int f) {
if (tallerThan(a, b)
|| harderThan(c, d)
|| heavierThan(e, f)) {
DoOneThing();
} else {
DoAnother();
}
}
boolean tallerThan(int a, int b) { return a > b; }
// Note “package scope”
boolean harderThan(int c, int d) { return c > d; }
// rather than public; JUnit
boolean heavierThan(int e, int f) { return e > f; }
// tests can access these.
We can write four tests for decide. One where tallerThan is true. One where harderThan is true. One where heavierThan is true. And one where they are all false. We could test each of the extracted functions with two tests, so the total would be 4 + 2 * 3 = 10 tests. This would be just enough tests so that most unintentional changes will trigger a test failure. Exposing the internals this way trades decreased encapsulation for increased testability. Limit the exposure by controlling scope appropriately, as we did in the Java code above.
How many tests is too many? The answer is “It depends.” It depends on how much confidence the tests can provide in the face of changes made by others. Tests can detect whether a programmer changed some code in error, and can serve as examples and documentation. Don't write redundant tests, and don't write too few tests.
Remember to download this episode of Testing on the Toilet and post it in your office. Permalink | Links to this post |
TotT: The Stroop Effect
Thursday, February 07, 2008 12:00 AM
How quickly can you...
- ...read all 25 words out loud: RED, GREEN, BLUE, ... (Try it now!)
- ...say all 25 colors out loud: GREEN, YELLOW, WHITE... (Try it now!)
Did the second task require more time and effort? If so, you're experiencing the Stroop Effect, which roughly says that when a label (in this case, the word) is in the same domain as its content (the color) with a conflicting meaning, the label interferes with your ability to comprehend the content.
What does this have to do with testing? Consider the following code:
public void testProtanopiaColorMatcherIsDistinguishable() {
ColorMatcher colorMatcher = new ColorMatcher(PROTANOPIA);
assertFalse(“BLUE and VIOLET are indistinguishable”,
colorMatcher.isDistinguishable(Color.BLUE, Color.VIOLET));
}
When this test fails, it produces a message like this:
Failure: testProtanopiaColorMatcherIsDistinguishable:
Message: BLUE and VIOLET are indistinguishable
Quick: what caused this error? Were BLUE and VIOLET indistinguishable, or not? If you're hesitating, that's the Stroop Effect at work! The label (the message) expresses a truth condition, but the content (in assertFalse) expresses a false condition. Is the ColorMatcher doing the wrong thing, or is the test condition bogus? This message is wasting your valuable time! Now consider this slight alteration to the test name and test message:
Failure: testProtanopiaColorMatcherCannotDistinguishBetweenCertainPairsOfColors
Message: BLUE and VIOLET should be indistinguishable
Do you find this clearer? Protanopia (reduced sensitivity to the red spectrum) causes certain pairs of colors to be indistinguishable. BLUE and VIOLET should have been indistinguishable, but weren't.
Here are some things to keep in mind when writing your tests:
- When someone breaks your test – will your test name and message be useful to them?
- Opinionated test names like testMethodDoesSomething can be more helpful than testMethod.
- Great test messages not only identify the actual behavior,but also the expected behavior.
- Should is a handy word to use in messages – it clarifies what expected behavior didn't actually happen.
Remember to download this episode of Testing on the Toilet and post it in your office. Permalink | Links to this post |
Announcing the Google Testing Blog - German Version!
Saturday, February 02, 2008 7:19 AM
Posted by Christopher Semturs, Software Engineer in Test, Zurich
For those German-speaking folks among our readers of this English Google Testing Blog we have exciting news: We have just launched the German Testing Blog!
This is a tribute to the fact that the German-speaking software test community is one of the biggest non-english audiences of this blog. The german blog will contain a mix of German versions of postings from this blog as well as unique postings about regional-specific issues from Europe. Some of our biggest engineering offices outside of US are in Europe, e.g. in Switzerland and the UK.
So German speakers around the world check out this new resource for interesting discussions about software test, performance test, load test, and how to create rock-solid software.
Permalink | Links to this post |
