Thursday, March 20, 2008

TotT: TestNG on the Toilet

Recently, somewhere in the Caribbean Sea, you implemented the PirateShip class. You want to test the cannons thoroughly in preparation for a clash with the East India Company. This requires that you run the crucial testFireCannonDepletesAmmunition() method many times with many different inputs.

TestNG is a test framework for Java unit tests that offers additional power and ease of use over JUnit. Some of TestNG's features will help you to write your PirateShip tests in such a way that you'll be well prepared to take on the Admiral. First is the @DataProvider annotation, which allows you to add parameters to a test method and provide argument values to it from a data provider.

public class PirateShipTest {
  @Test(dataProvider = "cannons")
  public void testFireCannonDepletesAmmunition(int ballsToLoad,
         int ballsToFire,
         int expectedRemaining) {
    PirateShip ship = new PirateShip("The Black Pearl");
    for (int i = 0; i < ballsToFire; i++) {
    assertEquals(ship.getBallsRemaining(), expectedRemaining);
  @DataProvider(name = "cannons")
  public Object[][] getShipSidesAndAmmunition() {
    // Each 1-D array represents a single execution of a @Test that
    // refers to this provider. The elements in the array represent
    // parameters to the test call.
    return new Object[] {
      {5, 1, 4}, {5, 5, 0}, {5, 0, 5}

Now let's focus on making the entire test suite run faster. An old, experienced pirate draws your attention to TestNG's capacity for running tests in parallel. You can do this in the definition of your test suite (described in an XML file) with the parallel and thread-count attributes.

<suite name="PirateShip suite" parallel="methods" thread-count="2">

A great pirate will realize that this parallelization can also help to expose race conditions in the methods under test.
Now you have confidence that your cannons fired in parallel will work correctly. But you didn't get to be a Captain by slacking off! You know that it's also important for your code to fail as expected. For this, TestNG offers the ability to specify those exceptions (and only those exceptions) that you expect your code to throw.

@Test(expectedExceptions = { NoAmmunitionException.class })
public void testFireCannonEmptyThrowsNoAmmunitionException() {
  PirateShip ship = new PirateShip("The Black Pearl");

Remember to download this episode of Testing on the Toilet and post it in your office.


  1. Nice short TotT article.

    First impressions though is that the TestNG features you mentioned (not a library I am vary familiar with) are offered by JUnit as well.

    The dataprovider annotation seems analogous to JUnit's @RunWith(Parameterized.class) annotation. And the expectedExceptions similarly analogous to JUnit's @Test(expected=Exception.class) annotation.

    All said, would be interested to hear if there are subtle differences here, or perhaps a future article on some more powerful features of TestNG.

  2. PHPUnit also supports a @dataProvider annotation (and quite some other annotations, for instance to get more meaningful code coverage information).

  3. I am quite happy to see you finally mentioning TestNG!

    @ben: in fact there is a very big difference between JUnit @RunWith(Parameterized) and TestNG data providers. JUnit parameterized approach is just able to pass parameter to your test instance through constructor parameters. So, basically you are restricted to use a single data source per test class. TestNG data providers are feeding parameters directly to @Test methods, so you can have as many data sources per test class.
    There are more subtle differences like the test class instantiation model which is different (and sometimes considering that some data sources are expensive may lead to expensive tests). Also TestNG support eager and lazy data providers (I am not sure if the later is available in JUnit.

    Regarding the expectedException: TestNG was created before JUnit4, so I was glad to see this feature (as a couple of others) getting picked by the JUnit team.


    .w( the_mindstorm )p.
    TestNG co-founder
    EclipseTestNG Creator

  4. Actually, this looks much more like with @Theory introduced on JUnit 4.4. It was an addition made from Popper library.

  5. Hello I just entered before I have to leave to the airport, it's been very nice to meet you, if you want here is the site I told you about where I type some stuff and make good money (I work from home): here it is


The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.