Tuesday, January 05, 2010

Interviewing Insights and Test Frameworks

By James A. Whittaker


  1. Hi James,

    Eric Jacobson goes in the same way trying to hire some tester:

    I believe a good tester believe in not what you say to test, he can't make assumptions and test everything from primary functionality of tested object, to the unincredible things that a user can do.

    Good post!

  2. Great post.

    I like those categories and think they probably make sense. But when trying to identify myself with one or the other, I couldn't. My strategy would represent a combination of all of them.

    I'm focused on achieving as much coverage as possible in as few tests as possible and have had good success with pairwise and more thorough combinatorial testing approaches.

    How I'd start testing Google Maps "Get Directions" functionality would be a case in point. Here, I suggest the 37 tests I'd recommend executing first - from 6.5 billion possible tests. My approach is described in detail in this video in 2.5 mins: https://hexawise.com/intro_moviehttps://hexawise.com/intro_movie (from 2:25 - 4:50)

    Let's look at which of your frameworks my tests fall into:

    Input Domain Framework? Check. e.g., Search for location by normal addresses, city names, GPS locations, & even favorite foods.

    Divide and Conquer Framework? Yup, at least partially. These tests, to use your words, are "concerned with reducing the size of the problem to something manageable." 6.5 trillion possible tests to the first 37 tests I'd perform. Also, we've reduced the size of the SUT by dividing and conquering scope by cutting out: non-US locations, iPhone-based "on the go" usage, etc.

    Fishbowl Framework? I'm not as clear as what you mean by this. Sounds like it may cover both savvy testers who want to quickly test multiple states and inept bunglers who said "I dunno, I'd just kinda, like... uh... poke at it 'n stuff and like, you know,... see if anything busts." For the goal of "creating big visible changes", would the "zoom all the way in" & "pan far right" across multiple tests ranging from very short trips to much longer trips count (particularly since ea type of view would also be captured with these)?

    Storybook Framework? Check. And, to be fair to the valid criticism you level against this framework, these 37 tests are guilty of not going as far beyond expected / happy path scenarios as they should... yet.

    Pessimists Framework? Yep. ...

    [One of two - apologies for the length of my reply]

  3. [Continued... two of two]

    Pessimists Framework? Yep. This is an area where this strategy truly shines. These types of tests lend themselves to capturing slightly diabolical ones that include particularly easy to miss combinations. What I'm thinking here is not "sugar in the gas tank" scenarios, I'm thinking "16 year old aggressive driver who hasn't quite learned how to use a clutch who is driving in the rain on an oil-slicked surface while speeding then slamming on the breaks" kinds of tests. In software testing terms, that could be testing for the "change credit card? = Y" (relatively unusual for users to choose) while also using Google Chrome as a browser. This combination recently seemed to be responsible for erasing my Expedia reservation. The 37 tests I've described in the video segment above would test for 100% of such "odd-ball" combinations. Every pair of values, no matter how unlikely for a user to use/choose/experience them together, would be tested for in at least 1 of the 37 tests.

    Incidentally, to test Google Maps, of course we'd want to conduct more than 37 tests. The next tests I'd try (after talking to SME’s to get additional inputs that aren’t apparent to me), would be (a) negative tests, then (b) "3-way interactions" generated on the "Create Tests" screen. There would be 216 3-way tests that would cover even more odd-ball scenarios, including at least one test for every single possible combos of 3 values, e.g., (i) "Drag on the blue line = yes" and (ii) "perform order of the actions listed in the test case from right to left" and (iii) "See traffic at a future time = rush hour."

    I'd be very interested in your thoughts on: (a) the viability and likely effectiveness of the approach I've outlined, and (b) any risks you might see in using combinatorial testing test design in combination with other approaches, like Tours.

    Thanks again for this post. I'll send a few strong applicants your way if I can.

    - Justin

  4. Interesting.

    When testing an application you want as much coverage of the system as possible. By implication, this means that no single test framework/style is "the one true way".

    You need the Input Driven approach for your protocols and APIs to make sure that these behave as expected. For protocols and APIs this should lead to automated black-box integration/regression tests.

    You need Divide and Conquer to get an even spread of tests. You need Fishbowl to provide ad-hoc testing to see what happens with your application in generally untested scenarios. These help to see the bigger picture.

    Your Storybook testers are good for assessing the usability of the product. Can the user accomplish the tasks outlined in the requirements? Are there any areas that would be complex to the users?

    Your Pessimists are good at picking up the edge cases. They will be good for the security testing/protocol fuzzing. If done at the API/protocol level, these should lead to automated tests.

  5. If we are not interested in an interview, can we still be involved in the study? I'd like to be involved in studies whenever I can!

  6. James,
    Great observation. I think the approach is contextual and I see value with most of these approaches depending on the context. although the last item appears to give good coverage towards the application stability and various other constraints + assumption, but with the combination other categories to an extent.

    I am interested to understand the other patterns are you running into in your observation.


  7. There is no better or worse for each testing framework. Actually, for a platform or a enterprise application project, we have to use different framework in different age. For example, in the early age for designing, I usually use input and out-put set; When coding finished, I used storybook; In system testing phase, I need test Pessimists...etc. So the most important thing is not tool, not method, not framework we need use. The most important is do we understand this product exactly? Do we understand deeply with the features we will test? Then we can think about how can we test it.

  8. This is a very interesting analysis. I like how you present your point of view in this blog. It give me good thinking about application testing. The sure way about testing is more a method than a technical solution.
    I think that the best testing is to cross test method and technics.

  9. Hi there, I've been searching for these job offers on Google Jobs. Should I sent my resume via any other channel? :D

  10. I would like to be part of the effort. I would add the following types I have seen:
    1. The "clueless" tester - someone who thinks testing is easy, anyone cand do it, and does understand the true complexity (we don't hire these).
    2 The "one way" tester - someone has been testing for a while, but has one "hammer" (framework?) they use over and over and over, miss errors, and don't understand there are many ways (we try to educate them).
    3. The "automation" tester - someone who thinks the tools will save them, all you need to do is create automation, they are very good at automation, and the tools "framework" is sufficient (we want these people but need to expand their view).

  11. http://at4qa.blogspot.com/2010/01/interesting-post-from-googles-qa-about.html

  12. I've noticed a similar pattern in the three test groups I've managed.

    What I've ended up doing is approaching a piece of new functionality (or a pernicious bug) and assigning testers or test approaches - what you're calling frameworks in this post - to cover the different aspects.

    While it's good to have a team with individuals strong in each of the frameworks you note I think it's better for a manager to match up testers and tools to hit the bug or new functionality with as many different frameworks as possible.

    In this sense I typically operate from the "Divide and Conquer Framework" and then apply the other frameworks to what's left after the divide and conquer. But that implies a 'default' framework I tend to be most comfortable with.

    When interviewing for positions I try to craft questions that expose what approach the tester seems most comfortable in based upon their resume. This doesn't always work, but by creating test scenarios based upon their resume I have a better shot at uncovering their assumptions and seeing if they'll introduce new assumptions and hence new approaches to my team.

    I'd love to work on this - if you're looking for folks who recognize this and try to use it to solve testing problems I'm happy to participate. I find these frameworks often cluster not only around how people test but testing tools as well.

  13. I think all this staff should be taken all to different development stages:
    Input/Output looks like Unit tests;
    Divide & Conquer - should be automation tests;
    Storybook - automated/manual acceptance test;
    FishBowl and Pessemistic - is more like 'the last try' to break the app so it's place - closer to release.

    So once engineer want to use just one/few frameworks - he can be part of the team for focussed tasks like unit testing only. Of course the best is candidate who can use all of this frameworks as needed since I don't think Google apps need part and not all the parts of this puzzle.

  14. James, I would like to participate in this study, is it still possible?

  15. I'm interested in this as well. I think it might point to an approach that would help eliminate test approach biases.

  16. Hi James

    My question is out of line with this interview but I have no other platform or media in which I can communicate with you in that case.

    I am from South Africa and would like to know how can I participate in this google ongoing interviews, I always look for testers job on google site but no luck.

    Please Help.

    Sbonelo Ndlovu

  17. This strikes me as a very broad analysis of different testers approaches to examining the software they have been given. I really like your framework classifications but I am surprised that all the lines between the frameworks are so clearly drawn.

    I would have thought before now that most manual testers would use a methodology which would be a hybrid of all these rather than a clearcut one of many.

    Personally my approach would be mish-mash of most of them:

    Depending on the software type, my hybrid framework is like this:
    * Proving the software works as it should (applying the claims made by the technical/functional specs/storyboards/developers in the pub)
    * Indulge in a short period of unscripted testing where I play with the product (the aim being 50% to break and 50% to find areas to suggest enhancements for
    * Carry out a short period of entering unexpected input into all the data entry fields in the product
    * Finish with stress testing.

    Guess I am not Google material then. :)

  18. PS if you are interested in reading I have just started writing a testing blog for my company here:

  19. Jim, I would love to discuss this topic with you. I was responsible for establishing a straregy to create testing center of excellence at GMAC. And now am trying to establish similar competencies at GM.

    I would be interested in comparing the industries and as well geography. Is Google finding it easier to access real capable testers.

    Hope you are interested in discussing.


    You can find my background on linkedin. Or at WWW.Georgenajm.com

  20. I would also like to be part of the effort.
    My strategy would depend on the phase in which testing is required and the premier goal of the application. Then some happy day scenarios and then the most time will probably be spent on the rainy day scenarios, where my real strenght is and creativity is required.

    Regards from the netherlands

  21. I like the ideas but jargon can mean different things to different people if they have not heard of your framework. Testing the product: if you cannot access the code/components it must be tested against the specifications (black-box), therefore the techniques are: boundary value analysis, equivalence partition analysis, state transition testing, decision table testing and use case testing. the skill is to determine which combination to use. Q: What are edge cases? A: Just another test case but probably one the developer never thought of! Please send me more info - I am interested.

  22. @Justin

    My co-worker Yvette Nameth calls this the kitchen sink framework.

  23. James,

    Thanks for responding. In that case, definitely put me down as a "kitchen sinker."

    I like the term in that it strongly implies that the pairwise testing / combinatorial testing strategy I described results in "throwing everything but the kitchen sink" at the system under test. That is a fair approximation of what I'm trying to do.

    However, in that a "kitchen sink" approach could be attempted with or without placing a priority on efficiency, I'd like to state for the record that I seek to be a "hyper-efficient kitchen sinker". :) The 37 tests from selected from the 6.5 billion I described earlier were explicitly designed with maximum defect-triggering efficiency in mind.

    I enthusiastically agree with what you wrote on page 23 of your insightful, recently-published book, Exploratory Software Testing, namely, "We must identify all the things that can be varied during a test and make sure we make smart decisions when we choose specific variations and exclude (out of necessity) other variations." That quote, in a nutshell, summarizes what the specialized applied statistics-based field of Design of Experiments is all about. There is a science to making smart decisions about what should be varied when.

    Another way to look at Design of Experiments-based software testing design methods (which would include pairwise, n-wise, hexawise and orthogonal array-based testing techniques) is that they are structured methods of "hyper-efficient kitchen sinking."

    I would be very interested in hearing about your experiences with using pairwise and other similar approaches (and/or feedback on our Hexawise combinatorial test case design tool). I'm personally helping teams to combine pairwise and more thorough 3-wise testing techniques with the "scenario operators" idea you articulated in Exploratory Testing on p. 68. Early results have been extremely promising; including the scenario operators as inputs into the test case generating tool has helped to find defects the teams otherwise would have missed.

    - Justin Hunter

  24. My thoughts are similar to what Stas G.O. Satin had to say - each of these testing 'frameworks' are better known in the 'subject' of testing under the names of: performance testing & stress testing - fishbowl framework; unit testing - input domain framework & divide and conquer; system testing - story book testing & divide and conquer; stress testing & invalid scenario testing - Pessimists framework.

    I repeat 'divide and conquer' in system testing in addition to unit testing as that is a strategy used in both cases.

    Actually the 'testing frameworks' given here are the "philosophies" applied to a corresponding testing approach.

  25. The TDD tester. The one who just writes tests before he needs a new feature, and after discovering a bug before fixing. Not that hard ;-)

    If you have too many sad paths compared to happy paths then take those tests, and group them separately.

  26. Hello James,

    It's interesting reading your posts. Every time I learn new stuff that really makes to experiment on it. I have closely followed your insights about the various testers approaches. I am also kinda Justin, neither follows one nor all. It depends on what platform I am working on. Some times I create a typical hybrid framework which covers most of the application in all aspects. Since am not a performance tester, I test the performance by observation, of course not limited to it. But I concentrate on functionality, GUI and Usability. I am a Firefox and Google fan And I want to learn as much as I can about these two, again, not limited to just these two. Count me IN for the study you are conducting.

  27. This is a fantastic article and in my view it provides a perfect basis for writing guidelines on how to design test cases (eg for a Test Process). That being said, I think a good Tester should adapt to the need.....I think "Input Domain" when designing for a junior tester, I think "divide and conquer" when the project is massive and the features are many, I think "fishbowl" when there is a heavy integration component, I think "Storybook" when heading towards regression/sign-off or when I have extra resources to deploy who are product specialists but are not testers, and for every other case and for my default case I think "Pessimists".

    Im not saying my way is the right way either. Im just another user with another view :)

  28. I think the testers who are commenting here are trying to flatter you by glossing over a mistake that you've made. These 'frameworks' are not truly distinct, but rather superficially different ways of describing the same or similar processes coming from people with less or more experience and skill.

    The 'Input Domain' framework is a naive attempt at the 'Divide and Conquer' framework. Avoid testers who think exclusively in terms of 'Input Domain', they'll not give you any risk mitigation greater than that which good unit tests provide.

    'Story Book' is a way to prioritize 'Divide and Conquer' common in environments where requirements are either minimal or missing because it couples requirements gathering with test planning; you see this a lot in Agile. While it has some mileage it will often miss the corner cases that will only be caught by the 'Pessimists framework', which is simply a different prioritization based on the theory that corner cases have a statistically higher rate of return when trying to identify defects. I've found to be true in practice.

    So, all of this boils down to what you describe as 'Divide and Conquer', with the exception of the 'Fish Bowl framework', which I think is what is commonly referred to ad-hoc testing, but I can't be sure from the level of detail you've provided in your description.

    'Divide and Conquer' is better described as 'Formal Test Planning', and you should be looking for testers who excel at creating their own test plans without the assistance of any particular tool, or expectation of any particular type of development life cycle. The prioritization shouldn't matter, and experienced testers will choose a prioritization to match the project.

  29. Nice post Sir!
    During initial stage of my testing career i think i have used input domain framework but now, i am incline more towards pessimist framework

  30. This comment has been removed by the author.

  31. So, those sound a like like some of these (page 2 of 5):


    ...a set of "general test techniques" described by James Bach in 2006. In both cases, there appears to be some overlap or the sense that we really get down to splitting hairs by sorting characteristics of out into multiple distinct approaches. Also with both articles, there are occurrences of things that don't really seem to fit into the category of an approach to actual test case design/identification. In this article (Mr. Whittaker's), that would seem to the described Fishbowl approach, while in Mr. Bach's piece, the Automatic Testing technique seems out of place; unlike with the described Stress Testing or Scenario testing approaches, Automatic Testing requires that one performs some level of test case design/identification beforehand. Indeed, "automated testing" is largely the process of automating the execution of previously-identified test scripts--even if the particulars (input data, etc...also pre-defined to an extent, even if largely varied) are dynamically generated in the automated process.

    Anyway, I only first encountered Mr. Bach's paper today, well after having first read this article, and I thought the similarities were interesting. Both are very useful sources of information that have given me food for thought, so thanks very much to both authors and all contributing commentators here.

  32. Hi James

    I think what ever the frameworks or Methodologies are all should be based on the application or product to be tested, but as for as i am concerned i am strong believer of Exploratory testing framework and i achieved some great results in some of my projects as a lone tester at the same time documenting the tests are are equally important.
    May not work always in every projects so i practice that after implementing any testing methodologies are frameworks, make the team to freely play with the application with random testing and was very successful.


  33. Would like to participate in this study.
    Whats next?


The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.