Saturday, July 17, 2010

There, but for the grace of testing, go I

By James A. Whittaker

I've had more than a few emails about "antenna-gate" asking me to comment and suggesting clever, stabbing rebukes to a fallen competitor. I might aim a few of those at my own team in the future, some were genuinely funny, but none of them will appear here. Instead I offer first a word of caution and second a reflection that my Mom used to intone whenever disaster occurred around her. It's called "counting your blessings."

First, a caution that those of us who live in glass houses really should keep stones at arms length. The only way anyone can rebuke Apple, without risk of waking up one morning sucking on their own foot, is if they write no software or have no users. Apple does a lot of the former and they enjoy many of the latter. Bugs like this make me sick when they are mine and nervous when they aren't. If any tester in the industry isn't taking stock right now then they either aren't producing any software or aren't in possession of any users, at least ones they wish to keep.

Second, taking stock has made me realize that I enjoy some important blessings that make the infinite task of testing so much more manageable. Indeed, the three blessings I count here are really the reason that testing doesn't fail more often than it does.

The Blessing of Unit Testing

I am thankful for early cycle testing thinning out the bug herd. In late cycle testing major bugs are often masked by minor bugs and too many of the latter can hamper the search for the former. Every bug that requires a bug report means lost time. There is the time spent to find the bug; time spent to reproduce and report it; time to investigate its cause and ensure it is not a duplicate; time to fix it, or to argue about whether it should be fixed; time to build the new version and push it to the test lab; time to verify the fix; time to test that the fix introduced no additional bugs. Clearly the smaller the population to begin with, the easier the task becomes. Solid unit testing is a tester's best friend.

The Blessing of Rarity

I am thankful that the vast majority of bugs that affect entire user populations are generally nuisance-class issues. These are typically bugs concerning awkward UI elements or the occasional misfiring of some feature or another where workarounds and alternatives will suffice until a minor update can be made. Serious bugs tend to have a more localized effect. True recall class bugs, serious failures that affect large populations of users, are far less common. Testers can take advantage of the fact that not all bugs are equally damaging and prioritize their effort to find bugs in the order of their seriousness. The futility of finding every bug can be replaced by an investigation based on risk.

Risk analysis is so important that we've built an internal tool to help guide testers in performing it. Code-named "Testify" this tool streamlines the process of risk analysis, at least the way we do it at Google. We're working on open-sourcing an early prototype in time for GTAC 2010 (I can hear my team cringing now ... "you promised it when?").

The Blessing of Repetition

I am thankful that user behavior is highly repetitive. There are features that enjoy heavy usage across user populations and features that are far less popular. Mobile phones are a good example of this. The phone is constantly establishing connections to networks. Certain features like making and receiving calls, texting and so forth are used more often than taking pictures or searching maps. The popularity of user applications is a matter of hard data, not guesswork. Knowing what users do most often, less often and least often means testing resources can be applied with a commensurate amount of force and that testing itself can be patterned after actual usage profiles.

Testers can gain a great deal from taking the user’s point of view and weaving usage concerns into the software testing process. Focusing on the user ensures that high impact bugs are found early and software revisions that break key user scenarios are identified quickly and not allowed to persist.

Apple may be the company in the news today, who knows who it will be tomorrow. Every company that produces software people care about has either been there or will be there. The job is simply too big for perfection to be an option. But there are key advantages we have that make the job manageable.

Put down the stones and make sure that what few blessing we testers possess are being exploited for everything they are worth. Hopefully, your company will be spared and the next time a company suffers such a bug you won't be the one making excuses. Perhaps you'll be lucky enough to be the one saying, "there but for the grace of testing go I."

11 comments:

  1. Timely. Our new class in Software Testing (seminar course July '10) at MUM http://www.cs.mum.edu/courses/cs490/ just had a 7 group presentations the whole day today on several extended topics on software testing.

    "antenna-gate" had also been one of the content of the exchanges yesterday.

    Unit testing is indeed cost effective. Prioritizing based on usage profile is also nice.

    Looking forward to the Risk Analysis tool, "Testify".

    Here's our group presentation material today. We just did it this week in addition to our whole day class lectures on Software Testing.
    http://bit.ly/cwep8S

    We have one more week for this seminar course and it has been fun to develop a software testing mentality.

    Simplicio Gamboa III
    http://twitter.com/pinoystartup

    ReplyDelete
  2. Having tested Radio Frequency devices and having the dreaded recall hanging over your head. The integration testing of the software and hardware becomes an interesting problem.

    We would change the physical environment where the system was running. For example put the system in an oven and turn the heat up or generate high levels of static electricity. Then run our test suite under those conditions to ensure the system is working.

    One source of error we found was that as the tolerances of each component part was pushed to it's limit, you get some interesting cumulative effects that can cause some components to exceed their tolerances causing invalid signals in the system.

    ReplyDelete
  3. The news accounts about Apple's iPhone 4 antenna bug seem to ignore this key issue: Did Apple's QA and product testers know about the bug before, or after, GA product release?

    If Apple didn't know about the bug before product release, then Apple arguably made an innocent mistake because as you mention product testing is very difficult and nobody is perfect. However, if Apple did know about the bug before release, and as a business decision went ahead with it anyway, then perhaps Apple committed fraud.

    http://googcomments.blogspot.com

    ReplyDelete
  4. Another example why I hate people calling this profession "SW Testing",
    When you put the focus and expectations on the SW alone, and neglect to see that every SW must run on HW,
    Thus HW structure should be taken into account when planning the tests.

    And yes, I fully agree that every company has its glitches, if we look at Windows Mobile which still crash almost daily after over 10 years of improvements, and even Palm, which produces rather stable devices, had a glitch in almost every new product (luckily for them, they managed to fix these in following releases).
    Though they seem small - Smart phones are with very high complexity.

    Kobi Halperin

    ------------------

    ReplyDelete
  5. That was indeed a very good read, James.

    Tester-(QA Manager),
    Deven B.

    ReplyDelete
  6. "Bugs like this make me sick when they are mine and nervous when they aren't." Truer words were never spoken.
    When a bug count is low after a test cycle, I know something is amiss.

    ReplyDelete
  7. Looking forward to the risk-analysis tool.

    ReplyDelete
  8. James, You still excite me. Wherever you are, with your material.
    - Hari.

    ReplyDelete
  9. It never ceases to amaze me how many project leaders start with "HOW TO DO IT" before they understand "WHAT NEEDS TO BE DONE." I have over 20 years in the Quality Assurance/Testing area and believe that their is a fundamental problem with the lack of delivering testable requirements. If you understand what the client needs, you have a better chance for success.


    Or I am just kidding myself

    ReplyDelete
  10. As a web developer,you should not consider any bug very simple either it is in database part or in your coding part.
    You have to go through out whole. debugging to make your doubt clear.
    -quiz software

    ReplyDelete

The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.