Friday, June 13, 2008

Taming the Beast (a.k.a. how to test AJAX applications) : Part 1

Posted by Markus Clermont, John Thomas

This is the first in a two part blog series titled 'Taming the Beast: How to test AJAX applications". In this part we discuss some philosophies around web application testing and how it should be done the 'right' way. In part 2, we walk through a real example of designing a test strategy for an AJAX application by going 'beyond the GUI.'

Typically we address the problem of testing an AJAX application through a plethora of big end-to-end tests and (hopefully) high unit-test coverage. Here we outline the main problems with this approach and demonstrate an effective testing strategy for a sample GWT-based application which goes beyond "testing just through the GUI."

Problems with GUI-only testing
In general testing through the GUI:
  • is expensive (takes long to write the tests and execution is resource-intensive)
  • gives limited insight into the system
  • often take only the 'happy paths' into account
  • combines multiple aspects in a single test
  • is slow and brittle
  • needs a lot of maintenance
  • is hard to debug

And while unit tests don't suffer from many of these problems, they alone are not sufficient mainly because they:
  • give little insight how the components interact with each other
  • don't give confidence that the business logic and functionality of the system meets the requirements

Although there is no 'one size fits all' solution, there are some basic principles we can use to solve the testing problem for web applications:
  • Invest in integration tests (identify the smallest sub-system)
  • Separation of Concerns (don't do the set-up through the interface you are testing)
  • Test each interface separately (mock out everything that you are not testing)
  • Consider dependencies in production (figure out how dependencies can fail, and test that)
  • Use a mix of strategies and tools. There is no silver bullet.
  • And no... you cannot scrap all of your end-to-end tests

A recipe for testing goodness
Using the principles above we can build up a recipe for testing web applications. In the second part of our blog we will walk through each of these steps for a real web application.
  1. Explore the system's functionality
  2. Identify the system's architecture
  3. Identify the interfaces between components
  4. Identify dependencies and fault conditions
  5. For each function:
  • Identify the participating components
  • Identify potential problems
  • Test in isolation for problems
  • Create a 'happy path' test

End note: value of a test
A question commonly asked by developers when writing tests is, "is this really worth my time?" The short answer is "always!". Since fixing a bug is way more expensive than preventing it in the first place, writing good tests is always worth the time.

While there are many different classifications of tests, the most common way of classifying them is based on their size and the areas of a product they test. Each test answers specific questions about the product:
  • Unit test: is the method fulfilling its contract?
  • Small integration test: Can two classes interact with each other?
  • Medium integration test: Is a class interacting properly with all its dependencies? Does it anticipate and handle errors correctly? Are the needed functions exposed on an API/GUI?
  • Sub-system test: Can two sub-systems interact with each other? Does one of them anticipate all errors of the other and does it deal with them appropriately?
  • System test: Does the entire system behave as expected?
Keeping this questions in mind, testing at various levels allows us to write more focused and meaningful tests. Remember that effective tests are those that provide quick and useful feedback, i.e. quickly identify issues and pin point the exact location of the issue.

In the next episode we will walk through the recipe proposed above using a real web application.


  1. Somehow, personally I am disppointed with Part 1 - that aims at explaining philosophies around web application testing in "right" way and ended up in making very generic statements/suggestions like

    (Looked more like a FAQ about Testing (not even web app testing))

    " Explore the system's functionalityIdentify the system's architectureIdentify the interfaces between componentsIdentify dependencies and fault conditionsFor each function:Identify the participating componentsIdentify potential problemsTest in isolation for problemsCreate a 'happy path' test

    Invest in integration tests (identify the smallest sub-system)Separation of Concerns (don't do the set-up through the interface you are testing)Test each interface separately (mock out everything that you are not testing)Consider dependencies in production (figure out how dependencies can fail, and test that)

    May be articulation was not appropriate ...

    Hope to see something really "thought" provoking in Part 2 (in line with what techies normally expect with Google)


  2. Interesting. I look forward to seeing more regarding your example application and how to test it.

  3. Could you eventually provide a downloadable pdf document as you did with previous posts :) ?

  4. Interesting post, and yes we certainly agree that AJAX web applications can be quite difficult to test.

    But the picture you seem to paint is rather bleak, and it need not be.

    The situation for testing an AJAX application is actually easier than you may think, provide that you have the right tools and by using their underlying technology base can "do the right things" at the right time.

    * To overcome the expense of creating functional tests it makes sense to have a good test recorder, which takes away most of the pain of creating tests.

    * Tests can be brittle, so you should have some kind of functional test playback feature than automatically adapts the bahavior of the playback engine to overcome inconsequential changes in the pages tested.

    * To split out tests of individual features you need some kind of CallScript capability that lets you organize your tests easily into groups of related functionality.

    But you are dealing with AJAX -- and it seems that you left out the most important aspect of all: test synchronization. Tests of AJAX type applications scream out for playback synchronization -- but not by adding Waits or Delays or Sleeps that never work in practice. There has to be a way built into your playback engine to interact with the DOM of the page and sense when it is OK to continue playback.

    We believe if you have the right facilities in the test engine the functional testing job for an AJAX application can produce very good results.

  5. When is Part 2 coming ...?

    Waiting ...


  6. It might also be good to mention other test categories as well that relate to this such as: security, performance, etc... I look forward to Part II.


The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.