The Advantages of Unit Testing Early

by Shyam Seshadri

Nowadays, when I talk with (read: rant at) anyone about why they should do test driven development or write unit tests, my spiel has gotten extremely similar and redundant to the point that I don't have to think about it anymore. But even when I do pairing with skeptics, even as I cajole and coax testable code or some specific refactorings out of them, I wonder, why is it that I have to convince you of the worth of testing ? Shouldn't it be obvious ?

And sadly, it isn't. Not to many people. To many people, I come advocating the rise of the devil itself. To others, it is this redundant, totally useless thing that is covered by the manual testers anyway. The general opinion seems to be, "I'm a software engineer. It is my job to write software. Nowhere in the job description does it say that I have to write these unit tests." Well, to be fair, I haven't heard that too many times, but they might as well be thinking it, given their investment in writing unit tests. And last time I checked, an engineer's role is to deliver a working software. How do you even prove that your software works without having some unit tests to back you up ? Do you pull it up and go through it step by step, and start cursing when it breaks ? Because without unit tests, the odds are that it will.

But writing unit tests as you develop isn't just to prove that your code works (though that is a great portion of it). There are so many more benefits to writing unit tests. Lets talk in depth about a few of these below.

Instantaneous Gratification

The biggest and most obvious reason for writing unit tests (either as you go along, or before you even write code) is instantaneous gratification. When I write code (write, not spike. That is a whole different ball game that I won't get into now), I love to know that it works and does what it should do. If you are writing a smaller component of a bigger app (especially one that isn't complete yet), how are you even supposed to know if what you just painstakingly wrote even works or not ? Even the best engineers make mistakes.

Whereas with unit tests, I can write my code. Then just hit my shortcut keys to run my tests, and voila, within a second or two, I have the results, telling me that everything passed (in the ideal case) or what failed and at which line, so I know exactly what I need to work on. It just gives you a safety net to fall back on, so you don't have to remember all the ways it is supposed to work in. Something tells you if it is or not.

Also, doing Test Driven Development when developing is one of the best ways to keep track of what you are working on. I have times when I am churning out code and tests, one after the other, before I need to take a break. The concept of TDD is that I write a failing test, and then I write just enough code to pass that test. So when I take a break, I make it a point to leave at a failing test, so that when I come back, I can jump right back into writing the code to get it to pass. I don't have to spend 15 - 20 minutes reading through the code to figure out where I left off. My asserts usually tell me exactly what I need to do.

Imposing Modularity / Reusability

The very first rule of reusable code is that you have to be able to instantiate an instance of the class before you can use it. And guess what ? With unit tests, you almost always have to instantiate an instance of the class under test. Therefore, writing a unit test is always a first great step in making code reusable. And the minute you start writing unit tests, most likely, you will start running into the common pain points of not having injectable dependencies (Unless of course, you are one of the converts, in which case, good for you!).

Which brings me to the next point. Once you start having to jump through fiery hoops to set up your class just right to test it, you will start to realize when a class is getting bloated, or when a certain component belongs in its own class. For instance, why test the House when what you really want to test is the Kitchen it contains. So if the Kitchen class was initially part of the House, when you start writing unit tests, it becomes obvious enough that it belongs separately. Before long, you have modular classes which are small and self contained and can be tested independently without effort. And it definitely helps keep the code base cleaner and more comprehensible.

Refactoring Safety Net

Any project, no matter what you do, usually ends up at a juncture where the requirements change on you. And you are left with the option of refactoring your codebase to add / change it, or rewrite from scratch. One, never rewrite from scratch, always refactor. Its always faster when you refactor, no matter what you may think. Two, what do you do when you have to refactor and you don't have unit tests ? How do you know you haven't horribly broken something in that refactor ? Granted, IDE's such as Eclipse and IntelliJ have made refactoring much more convenient, but adding new functionality or editing existing features is never simple.

More often than not, we end up changing some undocumented way the existing code behaved, and blow up 10 different things (it takes skill to blow up more, believe me, I have tried). And its often something as simple as changing the way a variable is set or unset. In those cases, having unittests (remember those things you were supposed to have written?) to confirm that your refactoring broke nothing is godsend. I can't tell you the amount of times I have had to refactor a legacy code base without this safety net. The only way to ensure I did it correct was to write these large integration tests (because again, no unit tests usually tends to increase the coupling and reduce modularity, even in the most well designed code bases) which verified things at a higher level and pray fervently that I broke nothing. Then I would spend a few minutes bringing up the app everytime, and clicking on random things to make sure nothing blew up. A complete waste of my time when I could have known the same thing by just running my unit tests.

Documentation

Finally, one of my favorite advantages to doing TDD or writing unit tests as I code. I have a short memory for code I have written. I could look back at the code I wrote two days ago, and have no clue what I was thinking. In those cases, all I have to do is go look at the test for a particular method, and that almost always will tell me what that method takes in as parameters, and what all it should be doing. A well constructed set of tests tell you about valid and invalid inputs, state that it should modify and output that it may return.

Now this is useful for people like me with short memory spans. But it is also useful, say, when you have a new person joining the team. We had this cushion the last time someone joined our team for a short period of time, and when we asked him to add a particular check to a method, we just pointed him to the tests for that method, which basically told him what the method does. He was able to understand the requirements, and go ahead and add the check with minimal handholding. And the tests give a safety net so he doesn't break anything else while he was at it.

Also useful is the fact that later, when someone comes marching through your door, demanding you fix this bug, you can always make sure whether it was a a bug (in which case, you are obviously missing a test case) or if it was a feature that they have now changed the requirements on (in which case you already have a test which proves it was your intent to do it, and thus not a bug).

Permalink | Links to this post | 5 comments  

Software Testing Categorization

by Miško Hevery


You hear people talking about small/medium/large/unit/integration/functional/scenario tests but do most of us really know what is meant by that? Here is how I think about tests.

Unit/Small

Lets start with unit test. The best definition I can find is that it is a test which runs super-fast (under 1 ms) and when it fails you don't need debugger to figure out what is wrong. Now this has some implications. Under 1 ms means that your test cannot do any I/O. The reason this is important is that you want to run ALL (thousands) of your unit-tests every time you modify anything, preferably on each save. My patience is two seconds max. In two seconds I want to make sure that all of my unit tests ran and nothing broke. This is a great world to be in, since if tests go red you just hit Ctrl-Z few times to undo what you have done and try again. The immediate feedback is addictive. Not needing a debugger implies that the test is localized (hence the word unit, as in single class).

The purpose of the unit-test is to check the conditional logic in your code, your 'ifs' and 'loops'. This is where the majority of your bugs come from (see theory of bugs). Which is why if you do no other testing, unit tests are the best bang for your buck! Unit tests, also make sure that you have testable code. If you have unit-testable code than all other testing levels will be testable as well.

A KeyedMultiStackTest.java is what I would consider great unit test example from Testability Explorer. Notice how each test tells a story. It is not testMethodA, testMethodB, etc, rather each test is a scenario. Notice how at the beginning the test are normal operations you would expect but as you get to the bottom of the file the test become little stranger. It is because those are weird corner cases which I have discovered later. Now the funny thing about KeyedMultiStack.java is that I had to rewrite this class three times. Since I could not get it to work under all of the test cases. One of the test was always failing, until I realized that my algorithm was fundamentally flawed. By this time I had most of the project working and this is a key class for byte-code analysis process. How would you feel about ripping out something so fundamental out of your system and rewriting it from scratch? It took me two days to rewrite it until all of my test passed again. After the rewrite the overall application still worked. This is where you have an AHa! moment, when you realize just how amazing unit-tests are.

Does each class need a unit test? A qualified no. Many classes get tested indirectly when testing something else. Usually simple value objects do not have tests of their own. But don't confuse not having tests and not having test coverage. All classes/methods should have test coverage. If you TDD, than this is automatic.

Medium/Functional

So you proved that each class works individually, but how do you know that they work together? For this we need to wire related classes together just as they would be in production and exercise some basic execution paths through it. The question here we are trying to answer is not if the 'ifs' and 'loops' work, (we have already answered that,) but whether the interfaces between classes abide by their contracts. Great example of functional test is MetricComputerTest.java. Notice how the input of each test is an inner class in the test file and the output is ClassCost.java. To get the output several classes have to collaborate together to: parse byte-codes, analyze code paths, and compute costs until the final cost numbers are asserted.

Many of the classes are tested twice. Once directly throughout unit-test as described above, and once indirectly through the functional-tests. If you would remove the unit tests I would still have high confidence that the functional tests would catch most changes which would break things, but I would have no idea where to go to look for a fix, since the mistake can be in any class involved in the execution path. The no debugger needed rule is broken here. When a functional test fails, (and there are no unit tests failing) I am forced to take out my debugger. When I find the problem, I add a unit test retroactively to my unit test to 1) prove to myself that I understand the bug and 2) prevent this bug from happening again. The retroactive unit test is the reason why the unit tests at the end of KeyedMultiStackTest.java file are "strange" for a lack of a better world. They are things which I did not think of when i wrote the unit-test, but discovered when I wrote functional tests, and through lot of hours behind debugger track down to KeyedMultiStack.java class as the culprit.

Now computing metrics is just a small part of what testability explorer does, (it also does reports, and suggestions) but those are not tested in this functional test (there are other functional tests for that). You can think of functional-tests as a set of related classes which form a cohesive functional unit for the overall application. Here are some of the functional areas in testability explorer: java byte-code parsing, java source parsing, c++ parsing, cost analysis, 3 different kinds of reports, and suggestion engine. All of these have unique set of related classes which work together and need to be tested together, but for the most part are independent.

Large/End-to-End/Scenario

We have proved that: 'ifs' and 'loops' work; and that the contracts are compatible, what else can we test? There is still one class of mistake we can make. You can wire the whole thing wrong. For example, passing in null instead of report, not configuring the location of the jar file for parsing, and so on. These are not logical bugs, but wiring bugs. Luckily, wiring bugs have this nice property that they fail consistently and usually spectacularly with an exception. Here is an example of end-to-end test: TestabilityRunnerTest.java. Notice how these tests exercises the whole application, and do not assert much. What is there to assert? We have already proven that everything works, we just want to make sure that it is wired properly.

Permalink | Links to this post | 8 comments  

The Plague of Amnesia

By James A. Whittaker

Memory is supposed to be the first thing to go as one ages but in the grand scheme of engineering, software development can hardly be called elderly. Indeed, we’re downright young compared to civil, mechanical, electrical and other engineering disciplines. We cannot use age as an excuse for our amnesia.

There are two types of amnesia that plague software testers. We have team amnesia that causes us to forget our prior projects, our prior bugs, tests, failures and so forth. It’s time we developed a collective memory that will help us to stop repeating our mistakes. Every project is not a fresh start, it's only a new target for what is now a more experienced team. The star ship Enterprise keeps a ship's log. A diary that documents its crews’ adventures and that can be consulted for details that might help them out of some current jam. I'm not advocating a diary for test teams, but I do want a mechanism for knowledge retention. The point is that as a team we build on our collective knowledge and success. The longer the Enterprise crews' memory, the better their execution.

Quick, tell me about the last major failure of your teams’ product? Does your team have a collective memory of common bugs? Do you share good tests? If one person writes a test that verifies some functionality does everyone else know this so they can spend their time testing elsewhere? Do problems that break automation get documented so the painstaking analysis to fix them doesn’t have to be repeated? Does the team know what each other is doing so that their testing overlaps as little as possible? Is this accomplished organically with dashboards and ongoing communication or are the only sync points work-stopping, time-wasting meetings? Answer honestly. The first step to recovery is to admit you have a problem.

The second type of memory problem is industry amnesia. When I mentioned Boris Beizer and the pesticide paradox in my last post, how many of you had to look it up? And those who did know about it, how's your AJAX skills? Be honest...there are folks with both a historical perspective and modern skills but they are far too rare. Our knowledge, it seems, isn’t collective, it’s situational. Those who remember Beizer's insights worked in a world where AJAX didn’t exist, those who webspeak without trouble missed a lot of foundational thinking and wisdom. Remembering only what is in front of us is not really memory.

Industry amnesia is a real problem. Think about it this way: That testing problem in front of you right now (insert a problem that you are working on here) has been solved before. Are you testing an operating system? Someone has, many of them. Web app? Yep, that’s been done. AJAX? Client-server? Cloud service? Yes, yes and yes. Chances are that what you are currently doing has been done before. There are some new testing problems but chances are the one in front of you now isn’t one of them. Too bad the collective memory of the industry is so bad or it would be easy to reach out for help.

Let me end this column by pointing my finger inward: How will we (Google) test the newly announced Chrome operating system? How much collective knowledge have we developed from Chrome and Android? How much of what we learned testing Android will help? How much of it will be reusable? How easily will the Chrome and Andriod test teams adapt to this new challenge? Certainly many test problems are ones we've faced before.

Will we remember?

Permalink | Links to this post | 4 comments  

Update! Google Test Automation Conference: Website available

by Juergen Allgayer, Conference Chair

The organization of the 4th Google Conference on Test Automation (GTAC 2009), held in Zurich October 21-22, is well underway and we are looking forward to an exciting event. We are pleased to announce the GTAC website and that presenters will be able to apply for sponsorship.

Sponsorship available
One presenter per accepted proposal will be able to apply for sponsorship from Google. For sponsored presenters Google will arrange and pay travel and lodging (within reason). Further details will be announced to speakers of accepted proposals.

Website on-line (http://www.gtac.biz)
The conference website is now on-line with lots of information about the event, on how to submit proposals, about the venue, and much more.

Permalink | Links to this post | 4 comments  

Old habits die hard

by Stephen Ng and Noel Yap

Old habits die hard. Particularly when it comes to testing--once you get used to working in a project blanketed with lots of fast-running unit tests, it's hard to go back.

So when some of us at Google learned that we weren't able to use our favorite mocking libraries when writing for our favorite mobile platform (Android, naturally), we decided to do something about it.

As many of you already know (since you read this blog), mocking combined with dependency injection is a valuable technique for helping you write small, focused unit tests. And those tests in turn can help you structure your code so that it's loosely coupled, making it more readable and maintainable.

With a bit of setup, this can be done in Android, too. We've put together a tutorial describing our approach. It's the first installment in what we hope will be a series of articles on android app development and testing, from the perspective of Googlers who are not actually on the Android team. We'd love to hear whether you find it useful.

Permalink | Links to this post | 5 comments  

Why are we embarrassed to admit that we don't know how to write tests?

Take your average developer and ask "do you know language/technology X?" None of us will feel any shame in admitting that we do not know X. After all there are so many languages, frameworks and technologies, how could you know them all? But what if X is writing testable code? Somehow we have trouble answering the question "do you know how to write tests?" Everyone says yes, whether or not we actually know it. It is as if there is some shame in admitting that you don't know how to write tests.

Now I am not suggesting that people knowingly lie here, it is just that they think there is nothing to it. We think: I know how to write code, I think my code is pretty good, therefore my code is testable!

I personally think that we would do a lot better if we would recognize testability as a skill in its own right. And as such skills are not innate and take years of practice to develop. We could than treat it as any other skill and freely admit that we don't know it. We could than do something about it. We could offer classes, or other materials to grow our developers, but instead we treat it like breathing. We think that any developer can write testable code.

It took me two years of writing tests first, where I had as much tests as production code, before I started to understand what is the difference between testable and hard to test code. Ask yourself, how long have you been writing tests? What percentage of the code you write is tests?

Here is a question which you can ask to prove my point: "How do you write hard to test code?" I like to ask this question in interviews and most of the time I get silence. Sometimes I get people to say, make things private. Well if visibility is your only problem, I have a RegExp for you which will solve all of your problems. The truth is a lot more complicated, the code is hard to test doe to its structure, not doe to its naming conventions or visibility. Do you know the answer?

We all start at the same place. When I first heard about testing I immediately thought about writing a framework which will pretend to be a user so that I can put the app through its paces. It is only natural to thing this way. This kind of tests are called end-to-end-tests (or scenario or large tests), and they should be the last kind of tests which you write not the first thing you think of. End-to-end-tests are great for locating wiring bugs but are pretty bad at locating logical bugs. And most of your mistakes are in logical bugs, those are the hard ones to find. I find it a bit amusing that to fight buggy code we write even more complex framework which will pretends to be the user, so now we have even more code to test.

Everyone is in search of some magic test framework, technology, the know-how, which will solve the testing woes. Well I have news for you: there is no such thing. The secret in tests is in writing testable code, not in knowing some magic on testing side. And it certainly is not in some company which will sell you some test automation framework. Let me make this super clear: The secret in testing is in writing testable-code! You need to go after your developers not your test-organization.

Now lets think about this. Most organizations have developers which write code and than a test organization to test it. So let me make sure I understand. There is a group of people which write untestable code and a group which desperately tries to put tests around the untestable code. (Oh and test-group is not allowed to change the production code.) The developers are where the mistakes are made, and testers are the ones who feel the pain. Do you think that the developers have any incentive to change their behavior if they don't feel the pain of their mistakes? Can the test-organization be effective if they can't change the production code?

It is so easy to hide behind a "framework" which needs to be built/bought and things will be better. But the root cause is the untestable code, and until we learn to admit that we don't know how to write testable code, nothing is going to change...

Permalink | Links to this post | 11 comments  

Separation anxiety?

by Shyam Seshadri

We all have separation anxiety. Its a human tendency. We love inertia, and we don’t like change. But why should your code have separation anxiety ? Its not human (even though it might try and grow a brain of its own at times)!

I bring this up because I got the chance to work with someone who had some questions on how to test UI code. Fairly innocuous question, and in most cases, my response would have been, keep the UI code simple and free of any logic, and don’t worry too much about it. Or you could write some nice end to end / integration tests / browser based tests. So with that response set in mind, I set off into the unknown. Little was I to know was that was the least of my concerns. As I sat down to look at the code, I saw that there were already tests for the code. I was optimistic now, I mean, how bad could things be if there are already tests for it ?

Well, I should remember next time to actually look at the tests first. But anyways, there were tests, so I was curious what kinds of tests they wanted to write and what kind of help I could provide, if any. So it turns out, the class was some sort of GUI Component, which basically had some fields, and depending on whether they were set of not, displayed them as widgets inside of it. So there were, say, 5 fields, and differing combinations of what was set would produce different output. The nice thing was that the rendered data was returned as a nice Java object, so you could easily assert on what was set, and get a handle on the widgets inside of it, etc. So the method was something along the following lines :

public RenderedWidgetGroup render() {
    RenderedWidgetGroup group =
       createWidgetGroup(this.componentId,
                         this.parent);
    if (this.name = null) {
        return group;
    } 
    group.addWidget(new TextWidget(this.name));
    group.addWidget(
       new DateWidget(
           this.updatedTimestamp == null ?
                this.createdTimestamp : this.updatedTimestamp));
    return group;
}

So far, so good, right ? Nice, clean, should be easy to test if we can set up this component with the right fields. After that, it should just be a few tests based on the different code paths defined by the conditionals. Yeah, thats what I thought too. So, me, being the naive guy that I was, said, yeah, that looks nice, should be easy to test. I don’t see the problem.

Well, then I was taken to the tests. The first thing I see is a huge test. Or atleast thats what I think it is. Then I read it more closely, and see its a private method. It seems to take in a bunch of fields (Fields with names that I distinctly remembered from just a while ago) and churn out a POJO (Plain Old Java Object). Except this POJO was not the GUI Component object I expected. So obviously, I was curious (and my testing sensors were starting to tingle). So I followed through to where it was called (which wasn’t very hard, it was a private method) and lo and behold, my worst fears confirmed.

The POJO object generated by the private method was passed in to the constructor of the GUI component, which (on following it further down the rabbit hole) in its constructor did something like the following :

public MyGUIComponent(ComponentId id,
                     Component parent,
                     MyDataContainingPOJO pojo) {
   super(id, parent);
   readData(pojo);
} 

And of course, readData, as you would expect, is a :

• Private method


• Looks through the POJO


• If it finds a field which is not null then it sets it in the Gui Component

And of course, without writing the exact opposite of this method in the unit test, it just wasn’t possible to write unit tests. And sudddenly, it became clear why they were having trouble unit testing their Gui Components. Because if they wanted to test render (Which was really their aim), they would have to set up this POJO with the correct fields (which of course, to make things more interesting / miserable, had sub POJOs with sub POJOs of their own. Yay!) to be exercised in their test.

The problem with this approach is two fold :

1. I need tests to ensure that the parsing and reading from the POJO logic is sound, and that it correctly sets up the GUI Component.


2. Every time I want to test the render logic, I end up testing (unintentionally, and definitely unwantedly) the parsing logic.

This also adds, as I saw, obviously complicated pre test setup logic which should not be required to test something completely different. This is a HUGE code smell. Unit testing a class should not be an arduous, painful task. It should be easy as setting up a POJO and testing a method. The minute you have to perform complicated setup to Unit test a class (Note, the keyword is unit test. You can have integration tests which need some non trivial setup.), stop! There is something wrong.

The problem here is that there is a mixing of concerns in the MyGuiComponent class. As it turns out, MyGuiComponent breaks a few fundamental rules of testability. One, it does work in the constructor. Two, it violates the law of demeter, which states that the class should ask for what it needs, not what it doesn’t need to get what it needs (Does that even make sense ?). Thirdly, it is mixing concerns. That is, it does too much. It knows both how to create itself as well as do the rendering logic. Let me break this down further :

Work in the constructor

If you scroll back up and look at the constructor for MyGuiComponent, you will see it calling readData(pojo). Now, the basic concept to test any class is that you have to create an instance of the class under test (unless it has static methods. Don’t get me started on that…). So now, every time you create an instance of MyGuiComponent, guess what ? readData(pojo) is going to get called. Every. Single. Time ! And it cannot be mocked out. Its a private method. And god forbid if you really didn’t care about the pojo and passed in a null. Guess what ? It most probably will blow up with a NullPointerException. So suddenly, that innocuous method in the constructor comes back to haunt you when you write yours tests (You are, aren’t you ?).

Law of Demeter

Furthermore, if you look at what readData(pojo) does, you would be even more concerned. At its base, MyGuiComponent only cares about 6 fields which it needs to render. Depending on the state of each of these fields, it adds widget. So why does the constructor ask for something totally unrelated ? This is a fundamental violation of the Law of Demeter. The Law of Demeter can be summed up as “Ask for what you need. If you need to go through one object to get what you need, you are breaking it.”. A much more fancier definition can be found on the web if you care, but the minute you see something like A.B().C() or something along those lines, there is a potential violation.

In this case, MyGuiComponent doesn’t really care about the POJO. It only cares about some fields in the POJO, which it then assigns to its own fields. But the constructor still asks for the POJO instead of directly asking for the fields. What this means is that instead of just creating an instance of MyGuiComponent with the required fields in my test, I now have to create the POJO with the required fields and pass that in instead of just setting it directly. Convoluted, anyone ?

Mixing Concerns

Finally, what could be considered an extension of the previous one, but deserves a rant of its own. What the fundamental problem with the above class is that it is mixing concerns. It knows both how to create itself as well as how to render itself once it has been created. And the way it has been coded enforces that the creation codepath is executed every single time. To provide an analogy for how ridiculous this is, it is like a a Car asking for an Engine Number and then using that number to create its own engine. Why the heck should a car have to know how to create its engine ? Or even a car itself ? Similarly, why should MyGuiComponent know how to create itself ? Which is exactly what is happening here.

Solution

Now that we had arrived at the root of the problem, we immediately went about trying to fix it. My immediate instinct was to pull out MyGuiComponent into the two classes that I was seeing. So we pulled out a MyGuiComponentFactory, which took up the responsibility of taking in the POJO and creating a GuiComponent out of it. Now this was independently testable. We also added a builder pattern to the MyGuiComponent, which the factory leveraged.

class MyGuiComponentFactory {
    MyGuiComponent createFromPojo(ComponentId id,
                                 Component parent,
                                 MyDataContainingPOJO pojo) {
     // Actual logic of converting from pojo to
     // MyGuiComponent using the builder pattern
    }
}
class MyGuiComponent {
    public MyGuiComponent(ComponentId id,
                         Component parent) {
       super(id, parent);
   }
    public MyGuiComponent setName(String name) {
       this.name = name;
       return this;
   }
} 

And now, suddenly (and expectedly, I would like to add), the constructor for MyGuiComponent becomse simple. There is no work in the constructor. The fields are set up through setters and the builder pattern. So now, we started writing the unit tests for the render methods. It took about a single line of setup to instantiate MyGuiComponent, and we could test the render method in isolation now. Hallelujah!!

Disclaimer :

No code was harmed / abused in the course of the above blog post. There were no separation issues whatsoever in the end, it was a clean mutual break!

Permalink | Links to this post | 3 comments