The Plague of Homelessness
There are two communities who regularly find bugs, the testers who are paid to find them and the users who stumble upon them quite by accident. Clearly the users aren’t doing so on purpose, but through the normal course of using the software to get work (or entertainment, socializing and so forth) done failures do occur. Often it is the magic combination of an application interacting with real user data on a real user’s computing environment that causes software to fail. Isn’t it obvious then that testers should endeavor to create such data and environmental conditions in the test lab in order to find these bugs before the software ships?
Actually, the test community has been diligently making an attempt to do just that for decades. I call this bringing the user into the test lab, either in body or in spirit. My own PhD dissertation was on the topic of statistical usage testing and I was nowhere near the first person to think of the idea as my multi-page bibliography will attest. But there is a natural limit to the success of such efforts. Testers simply cannot be users or simulate their actions in a realistic enough way to find all the important bugs. Unless you actually live in the software you will miss important issues.
It’s like homeownership. It doesn’t matter how well the house is built. It doesn’t matter how diligent the builder and the subcontractors are during the construction process. The house can be thoroughly inspected during every phase of construction by the contractor, the homeowner and the state building inspector. There are just some problems that will only be found once the house is occupied for some period of time. It needs to be used, dined in, slept in, showered in, cooked in, partied in, relaxed in and all the other things homeowners do in their houses. It’s not until the teenager takes an hour long shower while the sprinklers are running that the septic system is found deficient. It’s not until a car is parked in the garage overnight that we find out the rebar was left out of the concrete slab. The builder won't and often can't do these things.
And time matters as well. It takes a few months of blowing light bulbs at the rate of one every other week to discover the glitch in the wiring and a year has to pass before the nail heads begin protruding from the drywall. These are issues for the homeowner, not the builder. These are the software equivalents of memory leaks and data corruption, time is a necessary element in finding such problems.
These are some number of bugs that simply cannot be found until the house is lived in and software is no different. It needs to be in the hands of real users doing real work with real data in real environments. Those bugs are as inaccessible to testers as nail pops and missing rebar are to home builders.
Testers are homeless. We can do what we can do and nothing more. It’s good to understand our limitations and plan for the inevitable “punch lists” from our users. Pretending that once an application is released the project is over is simply wrong headed. There is a warranty period that we are overlooking and that period is still part of the testing phase.
"These are some number of bugs that simply cannot be found until the house is lived in and software is no different. It needs to be in the hands of real users doing real work with real data in real environments."
ReplyDeleteAnd hence - multi-year Betas?
James,
ReplyDeleteIs your dissertation available? That sounds like an interesting read.
-Mark
When I was managing a QA Group, there were two things I would do to get my Analysts “on the floor” with the Users;
ReplyDelete1) I would have them physically sit with the Business (in this case- Sales, Cust Service, Financial Reporting, Print Room, etc.) for a few hours or a whole day and observe how the Users actually used the system. Inevitably, they would also get an earful from the Users about how things could be improved and why was the network so slow and why can’t they get to Gmail, anyway?! This did a few things: it gave the QA people a point of contact in the Business, it showed them actual usage, it gave them some pain points the Users were facing which they could later keep tucked in mind while testing, and it helped foster good will with the Users for IT showing that we were actually listening to them and not just sitting on the other side of the wall, throwing software over whenever we felt like screwing up their day.
2) A little while after a project was released, I liked to have my folks go out to the Business and interview the project sponsor and/or whoever conducted the UAT. If it was a 1-day to 1-week effort, check back in a week; while projects bigger than that would have drop-ins or email follow-ups at one week and one month, just to see if things were meeting their expectations. Certainly, this isn't the job of the QA/STE group, but because we worked so closely with the end-users, I/we found it really useful to have that direct interaction with them to get our finger on the pulse, at the source and not through some sterile after-action report, post-mortem, or project retrospective (though those are all useful tools, too). This relationship also gave us first-line information on any concerns that might not be defects, per se, but were pieces of functionality that the end user just didn’t think were quite right. Again, this first-hand feedback, cobbled together over the many, many projects, made us as a group that much better able to emulate the actions, flows, and true usage patterns of the end user.
In the sense of Homelessness, these things made us frequent visitors to these home-owners, so much so that we could drop in and borrow a cup of sugar and a bagel, or ask to run a print job to see how jobs were queuing up. If you don’t have a home (and if you are a cost center), you better become good friends with those folks with the fancy place on the hill, (the ones who actually pay the bills).
Hi,
ReplyDeleteI've been reading your posts and found them to be very interesting. I'm from Brazil and there are a number of coleagues of mine that aren't confortable reading english. May I unofficially "translate" your posts and post them in portuguese? Of course I would reference the original post.
Thanks.
As already mentioned UAT and blind user tests will fine some of these issues.
ReplyDeleteHowever there are many that will only come around over time.
Long Beta's and where this can't be done then offering discounts to select early "adopters" in return for reports on living with the software over time are also methods where you can keep the user community with you on our journey while not keeping User faith and making sure the company image isn't damaged.
Julio: yes you may. I am honored to be asked. Just tell your soccer team to stop beating ours!
ReplyDeleteWell Joe, better late than too soon I suppose?!? But well aimed my friend. I think that the word beta is an honest label for the break-in period. How long should the break in be? I am still understanding the ins and outs of this labeling scheme at Google. Before I joined it seemed too conservative. I'll keep you posted about how my feelings toward this change.
ReplyDeleteMark, my dissertation was published in 1992 from the University of Tennessee. You can search their site for it as I no longer have an electronic copy. It may also reside in various places that such documents hang out. There is also a paper that summarizes the key result in IEEE Trans. on Software Engineering in 1994 available through the IEEE Digital Library. I hope you do find it interesting. It was a large part of my existence for many years, so much so that I'm glad it's in my rear-view mirror!
ReplyDelete