So when some of us at Google learned that we weren't able to use our favorite mocking libraries when writing for our favorite mobile platform (Android, naturally), we decided to do something about it.
With a bit of setup, this can be done in Android, too. We've put together a tutorial describing our approach. It's the first installment in what we hope will be a series of articles on android app development and testing, from the perspective of Googlers who are not actually on the Android team. We'd love to hear whether you find it useful.
Old habits die hard
Wednesday, July 08, 2009 1:53 PM
Why are we embarrassed to admit that we don't know how to write tests?
9:49 AM
Take your average developer and ask "do you know language/technology X?" None of us will feel any shame in admitting that we do not know X. After all there are so many languages, frameworks and technologies, how could you know them all? But what if X is writing testable code? Somehow we have trouble answering the question "do you know how to write tests?" Everyone says yes, whether or not we actually know it. It is as if there is some shame in admitting that you don't know how to write tests. Now I am not suggesting that people knowingly lie here, it is just that they think there is nothing to it. We think: I know how to write code, I think my code is pretty good, therefore my code is testable! I personally think that we would do a lot better if we would recognize testability as a skill in its own right. And as such skills are not innate and take years of practice to develop. We could than treat it as any other skill and freely admit that we don't know it. We could than do something about it. We could offer classes, or other materials to grow our developers, but instead we treat it like breathing. We think that any developer can write testable code. It took me two years of writing tests first, where I had as much tests as production code, before I started to understand what is the difference between testable and hard to test code. Ask yourself, how long have you been writing tests? What percentage of the code you write is tests? Here is a question which you can ask to prove my point: "How do you write hard to test code?" I like to ask this question in interviews and most of the time I get silence. Sometimes I get people to say, make things private. Well if visibility is your only problem, I have a RegExp for you which will solve all of your problems. The truth is a lot more complicated, the code is hard to test doe to its structure, not doe to its naming conventions or visibility. Do you know the answer? We all start at the same place. When I first heard about testing I immediately thought about writing a framework which will pretend to be a user so that I can put the app through its paces. It is only natural to thing this way. This kind of tests are called end-to-end-tests (or scenario or large tests), and they should be the last kind of tests which you write not the first thing you think of. End-to-end-tests are great for locating wiring bugs but are pretty bad at locating logical bugs. And most of your mistakes are in logical bugs, those are the hard ones to find. I find it a bit amusing that to fight buggy code we write even more complex framework which will pretends to be the user, so now we have even more code to test. Everyone is in search of some magic test framework, technology, the know-how, which will solve the testing woes. Well I have news for you: there is no such thing. The secret in tests is in writing testable code, not in knowing some magic on testing side. And it certainly is not in some company which will sell you some test automation framework. Let me make this super clear: The secret in testing is in writing testable-code! You need to go after your developers not your test-organization. Now lets think about this. Most organizations have developers which write code and than a test organization to test it. So let me make sure I understand. There is a group of people which write untestable code and a group which desperately tries to put tests around the untestable code. (Oh and test-group is not allowed to change the production code.) The developers are where the mistakes are made, and testers are the ones who feel the pain. Do you think that the developers have any incentive to change their behavior if they don't feel the pain of their mistakes? Can the test-organization be effective if they can't change the production code? It is so easy to hide behind a "framework" which needs to be built/bought and things will be better. But the root cause is the untestable code, and until we learn to admit that we don't know how to write testable code, nothing is going to change...
Separation anxiety?
Monday, July 06, 2009 3:00 PM
We all have separation anxiety. Its a human tendency. We love inertia, and we don’t like change. But why should your code have separation anxiety ? Its not human (even though it might try and grow a brain of its own at times)!
I bring this up because I got the chance to work with someone who had some questions on how to test UI code. Fairly innocuous question, and in most cases, my response would have been, keep the UI code simple and free of any logic, and don’t worry too much about it. Or you could write some nice end to end / integration tests / browser based tests. So with that response set in mind, I set off into the unknown. Little was I to know was that was the least of my concerns. As I sat down to look at the code, I saw that there were already tests for the code. I was optimistic now, I mean, how bad could things be if there are already tests for it ?
Well, I should remember next time to actually look at the tests first. But anyways, there were tests, so I was curious what kinds of tests they wanted to write and what kind of help I could provide, if any. So it turns out, the class was some sort of GUI Component, which basically had some fields, and depending on whether they were set of not, displayed them as widgets inside of it. So there were, say, 5 fields, and differing combinations of what was set would produce different output. The nice thing was that the rendered data was returned as a nice Java object, so you could easily assert on what was set, and get a handle on the widgets inside of it, etc. So the method was something along the following lines :
public RenderedWidgetGroup render() {
RenderedWidgetGroup group =
createWidgetGroup(this.componentId,
this.parent);
if (this.name = null) {
return group;
}
group.addWidget(new TextWidget(this.name));
group.addWidget(
new DateWidget(
this.updatedTimestamp == null ?
this.createdTimestamp : this.updatedTimestamp));
return group;
}
So far, so good, right ? Nice, clean, should be easy to test if we can set up this component with the right fields. After that, it should just be a few tests based on the different code paths defined by the conditionals. Yeah, thats what I thought too. So, me, being the naive guy that I was, said, yeah, that looks nice, should be easy to test. I don’t see the problem.
Well, then I was taken to the tests. The first thing I see is a huge test. Or atleast thats what I think it is. Then I read it more closely, and see its a private method. It seems to take in a bunch of fields (Fields with names that I distinctly remembered from just a while ago) and churn out a POJO (Plain Old Java Object). Except this POJO was not the GUI Component object I expected. So obviously, I was curious (and my testing sensors were starting to tingle). So I followed through to where it was called (which wasn’t very hard, it was a private method) and lo and behold, my worst fears confirmed.
The POJO object generated by the private method was passed in to the constructor of the GUI component, which (on following it further down the rabbit hole) in its constructor did something like the following :
public MyGUIComponent(ComponentId id,
Component parent,
MyDataContainingPOJO pojo) {
super(id, parent);
readData(pojo);
} And of course, readData, as you would expect, is a :
- Private method
- Looks through the POJO
- If it finds a field which is not null then it sets it in the Gui Component
And of course, without writing the exact opposite of this method in the unit test, it just wasn’t possible to write unit tests. And sudddenly, it became clear why they were having trouble unit testing their Gui Components. Because if they wanted to test render (Which was really their aim), they would have to set up this POJO with the correct fields (which of course, to make things more interesting / miserable, had sub POJOs with sub POJOs of their own. Yay!) to be exercised in their test.
The problem with this approach is two fold :
- I need tests to ensure that the parsing and reading from the POJO logic is sound, and that it correctly sets up the GUI Component.
- Every time I want to test the render logic, I end up testing (unintentionally, and definitely unwantedly) the parsing logic.
This also adds, as I saw, obviously complicated pre test setup logic which should not be required to test something completely different. This is a HUGE code smell. Unit testing a class should not be an arduous, painful task. It should be easy as setting up a POJO and testing a method. The minute you have to perform complicated setup to Unit test a class (Note, the keyword is unit test. You can have integration tests which need some non trivial setup.), stop! There is something wrong.
The problem here is that there is a mixing of concerns in the MyGuiComponent class. As it turns out, MyGuiComponent breaks a few fundamental rules of testability. One, it does work in the constructor. Two, it violates the law of demeter, which states that the class should ask for what it needs, not what it doesn’t need to get what it needs (Does that even make sense ?). Thirdly, it is mixing concerns. That is, it does too much. It knows both how to create itself as well as do the rendering logic. Let me break this down further :
Work in the constructor
If you scroll back up and look at the constructor for MyGuiComponent, you will see it calling readData(pojo). Now, the basic concept to test any class is that you have to create an instance of the class under test (unless it has static methods. Don’t get me started on that…). So now, every time you create an instance of MyGuiComponent, guess what ? readData(pojo) is going to get called. Every. Single. Time ! And it cannot be mocked out. Its a private method. And god forbid if you really didn’t care about the pojo and passed in a null. Guess what ? It most probably will blow up with a NullPointerException. So suddenly, that innocuous method in the constructor comes back to haunt you when you write yours tests (You are, aren’t you ?).
Law of Demeter
Furthermore, if you look at what readData(pojo) does, you would be even more concerned. At its base, MyGuiComponent only cares about 6 fields which it needs to render. Depending on the state of each of these fields, it adds widget. So why does the constructor ask for something totally unrelated ? This is a fundamental violation of the Law of Demeter. The Law of Demeter can be summed up as “Ask for what you need. If you need to go through one object to get what you need, you are breaking it.”. A much more fancier definition can be found on the web if you care, but the minute you see something like A.B().C() or something along those lines, there is a potential violation.
In this case, MyGuiComponent doesn’t really care about the POJO. It only cares about some fields in the POJO, which it then assigns to its own fields. But the constructor still asks for the POJO instead of directly asking for the fields. What this means is that instead of just creating an instance of MyGuiComponent with the required fields in my test, I now have to create the POJO with the required fields and pass that in instead of just setting it directly. Convoluted, anyone ?
Mixing Concerns
Finally, what could be considered an extension of the previous one, but deserves a rant of its own. What the fundamental problem with the above class is that it is mixing concerns. It knows both how to create itself as well as how to render itself once it has been created. And the way it has been coded enforces that the creation codepath is executed every single time. To provide an analogy for how ridiculous this is, it is like a a Car asking for an Engine Number and then using that number to create its own engine. Why the heck should a car have to know how to create its engine ? Or even a car itself ? Similarly, why should MyGuiComponent know how to create itself ? Which is exactly what is happening here.
Solution
Now that we had arrived at the root of the problem, we immediately went about trying to fix it. My immediate instinct was to pull out MyGuiComponent into the two classes that I was seeing. So we pulled out a MyGuiComponentFactory, which took up the responsibility of taking in the POJO and creating a GuiComponent out of it. Now this was independently testable. We also added a builder pattern to the MyGuiComponent, which the factory leveraged.
class MyGuiComponentFactory {
MyGuiComponent createFromPojo(ComponentId id,
Component parent,
MyDataContainingPOJO pojo) {
// Actual logic of converting from pojo to
// MyGuiComponent using the builder pattern
}
}
class MyGuiComponent {
public MyGuiComponent(ComponentId id,
Component parent) {
super(id, parent);
}
public MyGuiComponent setName(String name) {
this.name = name;
return this;
}
}
And now, suddenly (and expectedly, I would like to add), the constructor for MyGuiComponent becomse simple. There is no work in the constructor. The fields are set up through setters and the builder pattern. So now, we started writing the unit tests for the render methods. It took about a single line of setup to instantiate MyGuiComponent, and we could test the render method in isolation now. Hallelujah!!
Disclaimer :
No code was harmed / abused in the course of the above blog post. There were no separation issues whatsoever in the end, it was a clean mutual break!
By James A. Whittaker
The plague of repetitiveness
If aimlessness is the result of ‘just doing it’ then repetitiveness is the result of ‘just doing it some more.’ Pass after pass, build after build, sprint after sprint, version after version we test our product. Developers perform reviews, write unit tests and run static analyzers. But we have little insight into this effort and can't take credit for it. Developers test but then we retest. We can’t assume anything about what they did so we retest everything. As our product grows in features and bug fixes get applied, we continue our testing. It isn’t long until new tests become old tests and all of them eventually become stale.
This is Boris Beizer’s pesticide paradox. Pesticide will kill bugs, but spray the same field enough times with the same poison and the remaining bugs will grow immune. Rinse and repeat is a process for cleaning one’s hair, not testing software. The last thing we need is a build full of super-bugs that resist our ‘testicide.’ Even worse, all that so-called successful testing will give us a false sense of thoroughness and make our completeness metrics a pack of very dangerous lies. When you aren't finding bugs it's not because there are no bugs, it's the repetition that's creating the pesticide paradox.
Farmers know to adjust their pesticide formula from time to time and also to adjust the formula for the specific type of pest they expect in their fields. They do this because they understand the history of pesticide they used and know they can't get by with brute force repetition of they same old poison. Testers must pay attention to their test results too and watch for automation that isn’t adding value. A healthy injection of variation into automation is called for. Change the order of tests, change their data source, find new environments, modify input values do something the bugs aren’t prepared for.
The 7 Plagues of Software Testing
Monday, June 22, 2009 1:18 PM
By James A. Whittaker
The Plague of Aimlessness
Lore. It’s more than just a cool word. It conjures up a sorcerous image in one’s mind of ancient spell books and learned wizards with arcane and perilously attained knowledge.
And it’s exactly what we lack in software testing. Testing lore? Are you kidding me? Where is it? Who’s bogarting it? Can I have a hit?
The software testing industry is infected with the plague of aimlessness. We lack lore; we lack a body of knowledge that is passed from wizard to apprentice and written down in spell books for the industrious to study. Our apprentices are without their masters. We must all reinvent the wheel in the privacy of our offices only to have other testers the world over reinvent it in theirs.
I suggest we stop this nonsense. Testing is far too aimless. We test because we must or our manager tells us to do so. We automate because we can or because we know how to, not because it is part of some specific and proven strategy and certainly not because our lore dictates it. Is there a plan or some documented wisdom that guides our testing or are we just banging on the keyboard hoping something will fail? Where are the testing spell books? Surely the perilously attained knowledge of our tester forebears is something that we can access in this age of readily available information?
When a hunter makes a kill, they remember the terrain and circumstances. They pass this knowledge on to their successors. Over time they understand the habits of their prey and the collective knowledge of many hunters makes the job of future hunters far easier. When you see this terrain, you can expect game to behave in this manner. Can we say the same of testing? How well do we learn from each other? Do our ‘eureka moments’ get codified so that future testers will not have to suffer the aimless thrashing that we suffered? Can we say when you see functionality like that, the best way to test it is like this?
The plague of aimlessness is widespread. The need for testing lore is acute. Nike tells us to ‘just do it’ but what applies to exercise is not good advice for software testing. The next time you find yourself ‘just doing’ testing, pause for a moment and ask yourself ‘what is my goal?’ and ‘is there a purpose to this test?’ If the answer doesn’t immediately come to mind, you’re aimless, just doing it, and relying on luck and the sheer force of effort to find your quarry.
Luck has no place in sorcery or hunting and it has no place in testing. Luck is a nice happenstance, but it cannot be our plan A. Watch for the plague of aimlessness. Document your successes, scrutinize your failures and make sure you pass on what you learn from this introspection to your colleagues.
Be their wizard. Build a testing spell book and share it with others on your team. Over time you’ll banish the plague of aimlessness.
GTAC: Call for Proposals
Wednesday, June 17, 2009 8:51 AM
Posted by Juergen Allgayer - GTAC Conference Chair
GTAC 2009: Testing for the web
The 4th Google Test Automation Conference brings together a selected set of industry practitioners around the topic of software testing and automation. This annual conference provides a forum for presentations and connects professionals with each other. To increase outreach, presentations are published online for everybody to see.
This years theme is Testing for the Web, topics may include:
- Testing the UI of modern web applications (HTML5, Ajax)
- Testing applications on mobile devices
- Testing in the cloud
- Web testing tools (Selenium, Webdriver and co)
- Testing distributed asynchronous applications
- Testing for web browser compatibility
- Testing large storage systems
- Load and performance testing
- Finding and reproducing failures that matter
- It seemed like a good idea (things you expected to work, but that didn't)
Presentations are targeted at experienced engineers actively working on problems of quality, test automation and techniques, but also include students and academics. We encourage innovative ideas, controversial experiences, problems, and solutions that further the discussion of software engineering and testing. Presentations are 45 min in length and speakers should be prepared for an active question and answer session following their presentation. While ideas are good, ideas refined by experience are even more interesting to participants at GTAC.
The conference is a two day event comprised of a single track of talks. Our philosophy is to engage a small set of active participants who all experience the same topics carrying the discussions into lightning talks, speaker Q&A, and topical discussion groups. Each year we have worked to identify a location that has a unique profile of technology professionals. This year the conference will be held at the Google office in Zurich, Switzerland on October 21 and 22, 2009.
Submission of Proposals
Please email a detailed and extended abstract (one page at most) to gtac-2009-cfp@google.com. Your submission must include the name of topic, author(s), affiliation, and an outline of the proposed talk. We strongly recommend you to also submit one or two highlight slides of the talk. Submit your proposal before August 1, 2009. We will acknowledge reception within one business day. Where employer or disclosure authorization is needed, authors need to obtain it prior to submitting. The program committee will evaluate proposals based on quality and relevance. All submissions will be held confidentially prior to contacting the selected presenters.
Notification of Acceptance
Notification of acceptance will be sent out on or before August 8, 2009. Authors of accepted proposals will present at the conference and their talk will be made available to the public on YouTube.
Copyright
GTAC requires authors to present at the conference and permit their presentation to be made available on YouTube.
Attendees
To ensure active participation and provide a variety of technical perspectives, we select applying attendees. Further information will be published via a call for participation at a later time.
Important Dates
August 1 - Deadline for presentation proposals
August 8 - Notification of acceptance
October 21+22 - GTAC conference (Zurich, Switzerland)
Questions
If you have questions regarding the submission process or potential topics please email us at: gtac-2009@google.com
We will add more information to the Google Testing Blog as we get closer to the dates.
Google Test Automation Conference 2009 - Zurich, Switzerland
Sunday, June 14, 2009 9:31 PM
We have already received several inquiries about this year's GTAC - thanks for your enthusiasm, here's the news you've been waiting for - we will host the GTAC 2009 October 21 and 22 at the Google offices in Zurich, Switzerland.
As with previous years, the focus of the conference will be on solving software engineering challenges using tools and automation. This year will have a special focus on "Testing for the Web". We are looking forward to getting together to sharing lessons learned and practical experience testing web apps, services, and systems. We are also encouraging a discussion on effectively testing apps and services for mobile devices.
We will have a call for proposals coming out very soon - watch this space!
One of the strengths of the conference is that it's driven by a peer group and vocal participation. As in previous years, GTAC is an invitation only conference to share great ideas and to have your thoughts challenged and refined. When you apply, we want you to tell us what ideas and questions you'll bring to the conference, and how you can further the discussion. We will open the application process in late July 2009.
Please send suggestions, questions and recommendations to: gtac-2009@google.com or post your comments here to this blog.
