An Ingredients List for Testing - Part Seven (of Seven)
Friday, November 12, 2010
By James Whittaker
When to stop testing? It’s the age old testing question that many researchers have tried to quantify. In fact, the best answer requires no science whatsoever: never. Since testing is infinite, we can never really stop. A more practical answer also surfaces in the real world: when the software ships, you’re done. Of course this is only true for the duration of the ship party, after that testing continues on the next version.
At Google we are experimenting with test completeness measures that describe how well actual testing covers the risk landscape. In other words, we are measuring the extent to which our testing covers the things that require the most testing. Tests that cover the high risk areas well count for more than tests that cover lower risk features. Testing is, after all, a business of risk mitigation.
The set of tools necessary to accomplish this were described in my GTAC 2010 talk which should appear on YouTube soon and are collectively being called Google Test Analytics. More about these tools in future posts.
When to stop testing? It’s the age old testing question that many researchers have tried to quantify. In fact, the best answer requires no science whatsoever: never. Since testing is infinite, we can never really stop. A more practical answer also surfaces in the real world: when the software ships, you’re done. Of course this is only true for the duration of the ship party, after that testing continues on the next version.
At Google we are experimenting with test completeness measures that describe how well actual testing covers the risk landscape. In other words, we are measuring the extent to which our testing covers the things that require the most testing. Tests that cover the high risk areas well count for more than tests that cover lower risk features. Testing is, after all, a business of risk mitigation.
The set of tools necessary to accomplish this were described in my GTAC 2010 talk which should appear on YouTube soon and are collectively being called Google Test Analytics. More about these tools in future posts.
I saw GTA at starwest (ACC specifically). I WANTY!!!!
ReplyDeleteWhen do you publish the presentations of GTAC 2010?
ReplyDeleteLooking forward.... I check almost daily!
ReplyDeleteReally Nice and precise post, looking forward for the presentation.
ReplyDeleteIMO testing never does stop. You just delegate it to your customers.
ReplyDelete@James Whittaker
ReplyDeleteI love reading your thoughts on testing and its fundamentals. I completely believe that testing is handled by the product team and not just individual testers.
I will be at google HQ tomorrow March 30th, 2011. If you are located there I would like to chat with you more about product testing and fundamentals of automation testing. devashish.awasthi at gmail dot com