Unfortunately, a myriad of factors can make a test flaky. Today, we tackle a simple example: file access from a unit test. Take this function and its test:
data_file = open('/leases/gap', 'w+')
What if /leases/gap already exists and contains some data? testCreateGapLease will fail. This is a general problem where preconditions are assumed to be correct. This could just as easily happen by assuming a database contains the proper information (or no information). What if another test that uses that file was running concurrently?
If you really want to test your code using live resources, always check your assumptions. In this case, clearing the file at the start of the test can reduce its brittleness:
Unfortunately, this doesn't completely eliminate the flakiness of our test. If /leases/gap is an NFS path or can be written to by a different test, our test can still fail unexpectedly. It's better for the test to use a unique resource. This can be accomplished with a small refactoring of CreateGapLease:
def CreateGapLease(self, lease_path=None):
if lease_path is None:
lease_path = '/leases/gap'
The callers of CreateGapLease can continue invoking it as usual, but the unit test can pass in a different path:
lease_file = os.path.join(FLAGS.test_tmpdir, 'gap')
Of course, to make your test as fast as possible, it would be better to forgo disk access altogether by using a mock file system.
Remember to download this episode of Testing on the Toilet and post it in your office.
Due to illness availability of the PDF will be slightly delayed
The PDF is now available at the above link