How Google Tests Software - A Brief Interlude
Tuesday, February 22, 2011
By James Whittaker
These posts have garnered a number of interesting comments. I want to address two of the negative ones in this post. Both are of the same general opinion that I am abandoning testers and that Google is not a nice place to ply this trade. I am puzzled by these comments because nothing could be further from the truth. One such negative comment I can take as a one-off but two smart people (hey they are reading this blog, right?) having this impression requires a rebuttal. Here are the comments:
"A sad day for testers around the world. Our own spokesman has turned his back on us. What happened to 'devs can't test'?" by Gengodo
"I am a test engineer and Google has been one of my dream companies. Reading your blog I feel that Testers are so unimportant at Google and can be easily laid off. It's sad." by Maggi
First of all, I don't know of any tester or developer for that matter being laid off from Google. We're hiring at a rapid pace right now. However, we do change projects a lot so perhaps you read 'taken off a project' to mean something far worse than the reality of just moving to another project. A tester here may move every couple of years or so and it is a badge of honor to get to the point where you've worked yourself out of a job by building robust test frameworks for others to contribute tests to or to pass off what you've done to a junior tester and move on to a bigger challenge. Maggi, please keep the dream alive. If Google was a hostile place for testers, I would be working somewhere else.
Second, I am going to dodge the negative undertones of the developer vs tester debate. Whether developers can test or testers can code seems downright combative. Both types of engineers share the common goal of shipping a product that will be successful. There is enough negativity in this world and testers hating developers seems so 2001.
In fact, I feel a confession coming on. I have had sharp words with developers in the past. I have publicly decried the lack of testing rigor in commercial products. If you've seen me present you've probably witnessed me showing colorful bugs, pointing to the screen and shouting "you missed a spot!" I will admit, that was fun.
Here are some other quotes I have directed at developers:
"You must be smarter than me because I couldn't write this bug if I was trying to."
"What happened, did the compiler get your eye?"
"What do you say to a developer with two black eyes? Nothing, he's already been told twice."
"Did you hear about the developer who locked himself in his car?"
Ah, those were the good old days! But it's 2011 now and I am objective enough to give developers credit when they step up to the plate and do their job. At Google, many have and they are helping to shame the rest into following suit. And this is making bugs harder to find. I waste so little time on low hanging fruit that I get to dig deeper to find the really subtle, really critical bugs. The signal to noise ratio is just a whole lot stronger now. Yes there are fewer developer jokes but this is progress. I have to make myself feel good knowing how many bugs have been prevented instead of how many laughs I can get on stage demonstrating their miserable failures.
This is progress.
And, incidentally developers can test. In some cases far better than testers. Modern testing is about optimizing the places where developers test and where testers test. Getting that mix right means a great product. Getting it wrong puts us back in 2001 where my presentations were a heck of a lot funnier.
In what cases are developers better testers that we are? In what cases are they not only poor testers but we're better off not having them touch the product at all? Well, that's the subject of my next couple of posts. In the meantime...
...Peace.
These posts have garnered a number of interesting comments. I want to address two of the negative ones in this post. Both are of the same general opinion that I am abandoning testers and that Google is not a nice place to ply this trade. I am puzzled by these comments because nothing could be further from the truth. One such negative comment I can take as a one-off but two smart people (hey they are reading this blog, right?) having this impression requires a rebuttal. Here are the comments:
"A sad day for testers around the world. Our own spokesman has turned his back on us. What happened to 'devs can't test'?" by Gengodo
"I am a test engineer and Google has been one of my dream companies. Reading your blog I feel that Testers are so unimportant at Google and can be easily laid off. It's sad." by Maggi
First of all, I don't know of any tester or developer for that matter being laid off from Google. We're hiring at a rapid pace right now. However, we do change projects a lot so perhaps you read 'taken off a project' to mean something far worse than the reality of just moving to another project. A tester here may move every couple of years or so and it is a badge of honor to get to the point where you've worked yourself out of a job by building robust test frameworks for others to contribute tests to or to pass off what you've done to a junior tester and move on to a bigger challenge. Maggi, please keep the dream alive. If Google was a hostile place for testers, I would be working somewhere else.
Second, I am going to dodge the negative undertones of the developer vs tester debate. Whether developers can test or testers can code seems downright combative. Both types of engineers share the common goal of shipping a product that will be successful. There is enough negativity in this world and testers hating developers seems so 2001.
In fact, I feel a confession coming on. I have had sharp words with developers in the past. I have publicly decried the lack of testing rigor in commercial products. If you've seen me present you've probably witnessed me showing colorful bugs, pointing to the screen and shouting "you missed a spot!" I will admit, that was fun.
Here are some other quotes I have directed at developers:
"You must be smarter than me because I couldn't write this bug if I was trying to."
"What happened, did the compiler get your eye?"
"What do you say to a developer with two black eyes? Nothing, he's already been told twice."
"Did you hear about the developer who locked himself in his car?"
Ah, those were the good old days! But it's 2011 now and I am objective enough to give developers credit when they step up to the plate and do their job. At Google, many have and they are helping to shame the rest into following suit. And this is making bugs harder to find. I waste so little time on low hanging fruit that I get to dig deeper to find the really subtle, really critical bugs. The signal to noise ratio is just a whole lot stronger now. Yes there are fewer developer jokes but this is progress. I have to make myself feel good knowing how many bugs have been prevented instead of how many laughs I can get on stage demonstrating their miserable failures.
This is progress.
And, incidentally developers can test. In some cases far better than testers. Modern testing is about optimizing the places where developers test and where testers test. Getting that mix right means a great product. Getting it wrong puts us back in 2001 where my presentations were a heck of a lot funnier.
In what cases are developers better testers that we are? In what cases are they not only poor testers but we're better off not having them touch the product at all? Well, that's the subject of my next couple of posts. In the meantime...
...Peace.
Sure James. Thanks.
ReplyDeleteBy any chance will you be covering the topic - What Google really looks for, when hiring testers?
Since I had the honor of having my last, deliberately dramatic, comment included in this post I feel I should make some sort of reply.
ReplyDeleteFirst of all I am glad that you took the opportunity to clarify the newfound direction of the testing strategy over at Google, it obviously works well for you. I am also happy to see you confess that this direction leads to the conclusion that devs actually can test, and succeed in doing so. You have to understand my confusion since it kind of has been your M.O. to argue the exact opposite.
Having said that, I have given this some thought and I think can actually agree on that what we see here is progress. Although in my, admittedly not very long, career as a Test Engineer I’ve met many devs who I would never want to see as testers, they simply care too little. So I guess one of the challenges would be to structure an organization so that devs actually do care. Which is precisely what you have been describing in your posts. But, and here comes a little confession from me as well, testers must also be proficient at coding stuff. And to be honest, that is what scares me. I didn’t sign up for that. But I’m learning more and more and I think it can be quite fun some times (definitely not all the time.) Heck, I don’t know, perhaps the future for testers in this new way of thinking is bright after all, as long as one keeps it in mind from the start and managers hire testers with documented coding knowledge. People on every level of an organization have to understand and embrace these changes if they are going to be implemented successfully.
Lastly, I would like to add that I have never “hated” developers in the true sense of the word. (They give me work, right?) The importance of having a team where all members strive towards the same goal, which is to make a good product, is undoubted. I’ve never deviated from that conviction. It’s just that in some cases it takes superior intrapersonal skills to also convince the devs ;)
I’m looking forward to the continuation of these posts, which I will follow with interest, fear, confusion and contemplativeness.
Peace
Hi,
ReplyDeleteReading your blogposts gives me an impression that it's not long when we hear of a book titled "How we test software at Google" coming from you.
Should you decide to write one or are in the process I may want to book a copy in advance. Thanks :-)
Raj Ahuja
Great set of posts, looking forward to more.
ReplyDeleteI would be interested to know how you incorporate requirements into testing. We all know that poor requirements leads to poor products, but we've found here, that scripting the tests leads us to improving the requirements dramatically (filling in the gaps, adding clarification).
Also, I see in your post that it seems that TE's can pull the cord (to stop the train, so to speak?) I like that concept but do your TEs have that independence? Here, testing reports into Development so there is a real sense of lack of independence. In your grid reporting structure, do your TEs and SETs have that independence?
What level of experience are the SETs and do you cycle SWE and SETs?
You should probably hire Maggi - you can't train passion.
I think a good tester is who can be converted into developer at anytime and vice-versa.
ReplyDeleteAfter all I think Software Engineer and Software Engineer In Test are just one good programmer wearing different hats.
Glad to see Google are blending testers and developers!
I'm really surprised with the comments about your posts. When I read the last posts I thought I couldn't agree more about you.
ReplyDeleteTo me a good day is when I found a bug after some hard working hours. This means Developers worked well. Sometimes our Developers ask us for help on their own tests.
Quality doesn't belong to Testers. Quality belongs to all involved in a product development. And when everyone starts thinking this way we got more Quality.
I'm sure that Google is a wonderful place to work as a Tester.
My two cents: when dev can "throw code over the wall" to test - they have no incentive for quality. Their deliverable is functionality delivered to QA. QA's deliverable becomes Quality, but this is impossible to "test quality in" to the product. So you get a broken system and hostile environment.
ReplyDeleteWhen development is responsible for their own testing, suddenly they have an incentive for quality. If QA is embedded with the developers, the developers can ask QA to help them improve their testing (and in turn, their quality).
Hi james.
ReplyDeleteYour interesting articles let me make a decision that translate your article into korean and post on my blog to spread your article. Also, I know that it needs your permission. I promiss it will be strictly informational, not a commercial purposes.
As like meggi said, Google is a nice place to testers in korea too, so many people in korea would have interest with your articles. Please, Let them have a chance to access your article more easily.
I ‘m looking forward your positive reply.
Thanks.
PS> If you mind leaving your comment here, please e-mail me to: bbjoony@gmail.com
Thanks.
Great post. I am a developer but I want to learn more and more about testing. I do have one question that I would like to see what everyone's opinion on is. I seem to read alot about how I can write "better" code my making it more testable. While I agree to some degree I don't think that by just having a more testable code base for example the quality of the design of that code is necessarily increased. What do you think?
ReplyDeleteI am one of the few who transitioned through the ranks of software development. I started in support, moved to QA, and then to Development. I will say that I was a better tester before I moved to development. Now when testing others items I find myself thinking along the testing lines of how I would have coded it . This can be limiting. However - I think it's a bonus when I am coding as I think about the what if's and what the user can do versus what the user will do.
ReplyDelete