Wednesday, February 16, 2011

How Google Tests Software - Part Three

By James Whittaker

Lots of questions in the comments to the last two posts. I am not ignoring them. Hopefully many of them will be answered here and in following posts. I am just getting started on this topic.

At Google, quality is not equal to test. Yes I am sure that is true elsewhere too. “Quality cannot be tested in” is so cliché it has to be true. From automobiles to software if it isn’t built right in the first place then it is never going to be right. Ask any car company that has ever had to do a mass recall how expensive it is to bolt on quality after-the-fact.

However, this is neither as simple nor as accurate as it sounds. While it is true that quality cannot be tested in, it is equally evident that without testing it is impossible to develop anything of quality. How does one decide if what you built is high quality without testing it?

The simple solution to this conundrum is to stop treating development and test as separate disciplines. Testing and development go hand in hand. Code a little and test what you built. Then code some more and test some more. Better yet, plan the tests while you code or even before. Test isn’t a separate practice, it’s part and parcel of the development process itself. Quality is not equal to test; it is achieved by putting development and testing into a blender and mixing them until one is indistinguishable from the other.

At Google this is exactly our goal: to merge development and testing so that you cannot do one without the other. Build a little and then test it. Build some more and test some more. The key here is who is doing the testing. Since the number of actual dedicated testers at Google is so disproportionately low, the only possible answer has to be the developer. Who better to do all that testing than the people doing the actual coding? Who better to find the bug than the person who wrote it? Who is more incentivized to avoid writing the bug in the first place? The reason Google can get by with so few dedicated testers is because developers own quality. In fact, teams that insist on having a large testing presence are generally assumed to be doing something wrong. Having too large a test team is a very strong sign that the code/test mix is out of balance. Adding more testers is not going to solve anything.

This means that quality is more an act of prevention than it is detection. Quality is a development issue, not a testing issue. To the extent that we are able to embed testing practice inside development, we have created a process that is hyper incremental where mistakes can be rolled back if any one increment turns out to be too buggy. We’ve not only prevented a lot of customer issues, we have greatly reduced the number of testers necessary to ensure the absence of recall-class bugs. At Google, testing is aimed at determining how well this prevention method is working. TEs are constantly on the lookout for evidence that the SWE-SET combination of bug writers/preventers are screwed toward the latter and TEs raise alarms when that process seems out of whack.

Manifestations of this blending of development and testing are all over the place from code review notes asking ‘where are your tests?’ to posters in the bathrooms reminding developers about best testing practices, our infamous Testing On The Toilet guides. Testing must be an unavoidable aspect of development and the marriage of development and testing is where quality is achieved. SWEs are testers, SETs are testers and TEs are testers.

If your organization is also doing this blending, please share your successes and challenges with the rest of us. If not, then here is a change you can help your organization make: get developers fully vested in the quality equation. You know the old saying that chickens are happy to contribute to a bacon and egg breakfast but the pig is fully committed? Well, it's true...go oink at one of your developer and see if they oink back. If they start clucking, you have a problem.


  1. Thanks for sharing your experiences. I like the way you do things at google :-)

  2. Hey James,

    I haven't been in a testing position for very long, but I've been actively scouring the blogs and reading up on different methodologies in my free time to supplement my hands-on experience. The lack of experience may prove my following point to not really be the case in some companies, but I'll mention it anyways:

    The one major issue I've noticed has been that developers who test their own code tend to overlook glaring mistakes. Even going back to when I was going to school to be a developer (and have since decided that development is not for me), we all constantly ended up peer-testing rather than testing our own code because of the inattentional blindness effect. We are so focused on what our code is supposed to do that we overlook side effects that could vastly affect a customer.

    How do you get around this? My current company tries to keep a 1 to 1 ratio (although it's not quite there, we definitely have more on the development side), and it's been working amazingly well for us. Here's what I tend to see on a daily basis:

    Dev: Hey Shaun, I checked in some code last night, it should be in this morning's build. I tested it myself, so you can probably just give it a passing glance and close the issue as completed.

    Me: Yeah okay. (At which point even spending 10 minutes with the feature I find some issues with it, most are minor but occassionally something major)

    Me: Hey I thought you said you tested this...

    Dev: Yeah well one of the other devs must have done something to break my stuff/I don't see that issue on my machine/Are you sure you installed the latest build/That's not a bug it's a feature/etc.

    Some devs are better than others about the thoroughness of their testing, but they still miss things. Without dedicated testers, I can't see a product shipping without some major bugs in it.

    Sorry this was kind of a long comment, I just got to work and saw the new blog post and had to mention this. This is actually the first blog I've ever commented on, so hopefully I provided some sort of usefulness to the discussion.

    Thanks for reading,
    Shaun Hershey

  3. This makes a lot of sense. We have similar experience at Nokia where I have worked in different (server side Java) projects and as you may be aware, Nokia has had to grow up in a hurry in the past three years regarding server side development.

    A few things I have learned the hard way is that dedicated test engineers are nice until they leave your team and the build breaks and nobody really knows how to fix things. I find it is absolutely essential that all engineers can step in the role of writing any kind of test. It is vital that the team understands the test infrastructure and knows how to fix, utilize and add to it. It is also essential that engineers take the pain when the quality degrades. You have to make it their problem. If you break it, you fix it.

    That being said, it is nice to have somebody around to to dig in deeper where needed. Finding bugs is a skill and requires an attitude and attention to details that not all engineers have. In my view such people add most value if the engineers do their best to cover as much as possible with tests themselves. This frees up the test engineer for focusing on those areas that actually require their skills.

    So, unit tests, integration tests, acceptance tests, performance tests, etc. are not something I would put a dedicated engineer on.

    Regarding quality requirements. They seem abstract at first sight but they don't have to be. You can almost always translate quality requirements into functional requirements. The best way for this is to define them in terms of scenarios and then come up with acceptance tests for those.

    Finally, quality requirements tend to come on the table once you have a quality problem. As you mentioned, preventing a quality problem is better than fixing that problem. One way of preventing quality issues is to have enough good, experienced, senior engineers on the team. This sounds simplistic but I've seen projects where test engineers were brought in to fix a project that had essentially only junior engineers and a lot of quality problems. It doesn't work. It's better to have seniors on the team to do the job properly from day one and it costs much less in the end.

    Finally, monitoring is a great alternative to testing. If you have good monitoring in place, it will be fairly easy to see when you have a performance issue. I'd prefer to have a system that can tell me how its performing over an extensive test suite for that system that tells me how the system performs.

  4. Shaun,

    I'm not from Google, but I can probably shed some light on what you are seeing. In the teams that I have lead, I try to bring in a few key testing dimensions. Having a liberal sprinkling of all these test dimensions strengthens not only the quality of the component that is getting the change, but also reduces the risk of integration level issues.

    * Component Oriented Testing - testing the actual component itself.
    * Interdependency Testing - testing other components that consume or are consumed by the component being developed.
    * System Level Testing - testing broader system functionality that touches the component in question.

    What this breeds into the developers is an awareness of not only their change but also the impact that their change has on the system and on other components. Most good engineers when exposed to this information begin to "get it" and are more careful within that context.

    My understanding is that google is _very_ strong on ensuring the inter-dependencies between components are absolutely clear and absolutely visible to the developers.

    This paired with a focus on automated testing will lower the effort for the engineers, while keeping them aware. That solves the first part of your conversation.

    The second part is disassociating a breakage from a person. Complex systems have complex interactions and need sometimes things break. Investing in tools to detect, identify and remove issues is critical. Make it impartial through centralized testing, automated identification of regressing changelists go a long way to making it an engineering behaviour rather than a personal issue.

    My understanding is that google has virtually institutionalized a lot of what I've said above so engineers can take risks, find it hard to blame others and take ownership not only for the changes they make, but also the impact on the system.

    (I could go on for pages... But I won't :)

  5. Appreciate your sharing. My current BU definitely splits dev and test, and keeps dev:test ratio as 3:1. The dev almost don't do any unit testing. It makes tester life very difficult. Your points give us very good inspiration.

  6. Great Post James ,
    I completely agree - DevTest is an essential for good quality ,
    I’ve been working on several project with a lot of legacy code ( not testable code ) and
    Building quality into those products was a huge challenge ( which required serious refactoring )
    From my experience creating a single testing platform for SWE , STE & SE was one of the major success factors ,
    Each one have different needs but still have a common ground , finding it hooking everyone to the same platform / infrastructure
    Was important to us ,
    Also having the TE working with the product owners and understanding the customer was another key factor .
    Still this change requires a lot of resources and isn’t trivial .
    What about performance / security , are those treated the same , are they part of the SDLC or just a service ?

  7. Hi James (long time no see),

    i totally agree with the comment you make about the ownership of quality.

    The model we strive for is that of defect prevention rather than detection.

    Stuart Taylor

  8. In am a test engineer and Google has been one of my dream companies. Reading your blog I feel that Testers are so un-important at Google and can be easily laid off. Its sad.

  9. A sad day for testers around the world. Our own spokesman has turned his back on us. What happened to "devs can't test"? I always knew there was something fishy going on. At least the Bach brothers are still kicking it old school with a compelling, fun and efficient buccaneer twist.

  10. Wow. The comments are becoming larger than my posts! You guys are making my job really easy. Answers coming, I promise! Please stay tuned.

  11. it is very interesting the way you guys are working to make dev and testing blend together... it is a real challenge and your approach seems to be very good.... I would like to see more details on the work of the TE...

  12. I agree that 'quality' can not be 'tested in'. But the approach you describe appears to go-ahead and attempt to do something just, if not more, difficult. You suggest that a programmer will produce quality work by just coding 'better'. While a skilled and experienced programmer is capable of producing high quality software, who will tell them when they don't or can't? We are all potentially victims of the Dunning–Kruger effect, and as such we need co-workers to help.

    There are a host of biases that stop a programmer, product owner or project manager from questioning their work. The confirmation and congruence bias to name just two. These are magnified by group-think, and without the input of a more independent, experienced and skilled critical thinker, soon allow mistakes to occur.

    Think of it this way, how do you know your products are good enough? how do you know they are not plagued by flaws? Flaws like: a message that tells me my payment method is invalid - before I've entered one or the absence of a scale on the iPhone maps app.


  13. This will work if there is no much integration involved. If a team in geo position A writes code which gets integrated to Team B in Geo position B, manager will face issue of "its not my issue". To add more complexity, who will be doing System testing?
    Another issue with mindset of developers. They think, yes, this the way it is supposed to work and no need to test that. e.g. if the web app works on IE7, no need to test on IE8. Surely it is third party issue, but we need to remember it is our application which is going out to user and he/she wont care if the issue is in third party or our app. For user, it is issue.

    We have application developed for live streaming. Developer knows that yes, it is going to take 7 secs if bandwidth is say 1mbps. Hence ignores this 7 secs. But user will no accept this.

    Blending of Tester with Developer is good idea when tester can force developer to write "testable" code, provide hooks in code for testers to automate the testing.

  14. Shaun Hershey is right.

    The condition for what you describe can only happen when the developers have no deadline to meet and has no personal interest in the code.

    And with application interacting with other application get more in depth then before.

    if a developer has time to work on testing his app, he will be more productive working on paying back his "loan" on all the temp workaround he has implemented in the past due to time constraint.

  15. yeah... I now I see the quality of Google and MS which try to follow devs own the quality and testing. User experiance is terrible and it is going down. Crashes and obvious UI bags are not acceptable.


The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.