Signing Off

By James Whittaker

This will be my last post on this blog. Tomorrow is my last day at Google. It was a great ride and a great pleasure to work alongside such brilliant engineers. I will hand over this blog to another test director and then find a new place for whatever blogging I do in the future.

Follow me on Twitter (@docjamesw) if you are interested in where I land and to find my next blog outlet.

Peace.

Permalink | Links to this post | 0 comments  

How Google Tests Software

By James Whittaker

The publishing industry may be slow, but eventually they deliver. Amazon is now listing How Google Tests Software that I wrote with Jason Arbon and Jeff Carollo for presale. Find it at: http://goo.gl/Dg3qG

Permalink | Links to this post | 7 comments  

RPF: Google's Record Playback Framework

By Jason Arbon

At GTAC, folks asked how well the Record/Playback (RPF) works in the Browser Integrated Test Environment (BITE). We were originally skeptical ourselves, but figured somebody should try. Here is some anecdotal data and some background on how we started measuring the quality of RPF.
The idea is to just let users use the application in the browser, record their actions, and save them as a javascript to play back as a regression test or repro later. Like most test tools, especially code generating ones, it works most of the time but its not perfect. Po Hu had an early version working, and decided to test this out on a real world product. Po, the developer of RPF, worked with the chrome web store team to see how an early version would work for them. Why chrome web store? It is a website with lots of data-driven UX, authentication, file upload, and it was changing all the time and breaking existing Selenium scripts: a pretty hard web testing problem, only targeted the chrome browser, and most importantly they were sitting 20 feet from us.

Before sharing with the chrome web store test developer Wensi Liu, we invested a bit of time in doing something we thought was clever: fuzzy matching and inline updating of the test scripts. Selenium rocks, but after an initial regression suite is created, many teams end up spending a lot of time simply maintaining their Selenium tests as the products constantly change. Rather than simply fail like the existing Selenium automation would do when a certain element isn’t found, and require some manual DOM inspection, updating the Java code and re-deploying, re-running, re-reviewing the test code what if the test script just kept running and updates to the code could be as simple as point and click? We would keep track of all the attributes in the element recorded, and when executing we would calculate the percent match between the recorded attributes and values and those found while running. If the match isn’t exact, but within tolerances (say only its parent node or class attribute had changed), we would log a warning and keep executing the test case. If the next test steps appeared to be working as well, the tests would keep executing during test passes only log warnings, or if in debug mode, they would pause and allow for a quick update of the matching rule with point and click via the BITE UI. We figured this might reduce the number of false-positive test failures and make updating them much quicker.

We were wrong, but in a good way!

We talked to the tester after a few days of leaving him alone with RPF. He’d already re-created most of his Selenium suite of tests in RPF, and the tests were already breaking because of product changes (its a tough life for a tester at google to keep up with the developers rate of change). He seemed happy, so we asked him how this new fuzzy matching fanciness was working, or not. Wensi was like “oh yeah, that? Don’t know. Didn’t really use it...”. We started to think how our update UX could have been confusing or not discoverable, or broken. Instead, Wensi said that when a test broke, it was just far easier to re-record the script. He had to re-test the product anyway, so why not turn recording on when he manually verified things were still working, remove the old test and save this newly recorded script for replay later?

During that first week of trying out RPF, Wensi found:

• 77% of the features in Webstore were testable by RPF
• Generating regression test scripts via this early version of RPF was about 8X faster than building them via Selenium/WebDriver
• The RPF scripts caught 6 functional regressions and many more intermittent server failures.
• Common setup routines like login should be saved as modules for reuse (a crude version of this was working soon after)
• RPF worked on Chrome OS, where Selenium by definition could never run as it required client-side binaries. RPF worked because it was a pure cloud solution, running entirely within the browser, communicating with a backend on the web.
• Bugs filed via bite, provided a simple link, which would install BITE on the developers machine and re-execute the repros on their side. No need for manually crafted repro steps. This was cool.
• Wensi wished RPF was cross browser. It only worked in Chrome, but people did occasionally visit the site with a non-Chrome browser.

So, we knew we were onto something interesting and continued development. In the near term though, chrome web store testing went back to using Selenium because that final 23% of features required some local Java code to handle file upload and secure checkout scenarios. In hindsight, a little testability work on the server could have solved this with some AJAX calls from the client.

We performed a check of how RPF faired on some of the top sites of the web. This is shared on the BITE project wiki. This is now a little bit out of date, with lots more fixes, but it gives you a feel for what doesn’t work. Consider it Alpha quality at this point. It works for most scenarios, but there are still some serious corner cases.

Joe Muharsky drove a lot of the UX (user experience) design for BITE to turn our original and clunky developer and functional-centric UX into something intuitive. Joe’s key focus was to keep the UX out of the way until it is needed, and make things as self-discoverable and findable as possible. We’ve haven't done formal usability studies yet, but have done several experiments with external crowd testers using these tools, with minimal instructions, as well as internal dogfooders filing bugs against Google Maps with little confusion. Some of the fancier parts of RPF have some hidden easter eggs of awkwardness, but the basic record and playback scenarios seem to be obvious to folks.

RPF has graduated from the experimental centralized test team to be a formal part of the Chrome team, and used regularly for regression test passes. The team also has an eye on enabling non-coding crowd sourced testers generate regression scripts via BITE/RPF.

Please join us in maintaining BITE/RPF, and be nice to Po Hu and Joel Hynoski who are driving this work forward within Google.

Permalink | Links to this post | 3 comments  

GTAC Videos Now Available

By James Whittaker

All the GTAC 2011 talks are now available at http://www.gtac.biz/talks and also up on You Tube. A hearty thanks to all the speakers who helped make this the best GTAC ever. 


Enjoy!

Permalink | Links to this post | 0 comments  

ScriptCover makes Javascript coverage analysis easy

By Ekaterina Kamenskaya, Software Engineer in Test, YouTube

Today we introduce the Javascript coverage analysis tool, ScriptCover. It is a Chrome extension that provides line-by-line Javascript code coverage statistics for web pages in real time without any user modifications required. The results are collected both when the page loads and as users interact with it. The tool reports details about total web page coverage and for each external/internal script, as well as annotated code sources with individually highlighted executed lines.

Short report in Chrome extension’s popup, detailing both overall scores and per-script coverage.

Main features:

• Report current and previous total Javascript coverage percentages and total number of instrumented code instructions.
• Report Javascript coverage per individual instruction for each internal and external script.
• Display detailed reports with annotated Javascript source code.
• Recalculate coverage statistics while loading the page and on user actions.

Sample of annotated source code from detailed report. First two columns are line number and number of times each instruction has been executed.

Here are the benefits of ScriptCover over other existing tools:

• Per instructions coverage for external and internal scripts: The tool formats original external and internal Javascript code from ‘<script>’ tags to ideally place one instruction per line and then calculates and displays Javascript coverage statistics. It is useful even when the code is compressed to one line.

• Dynamic: Users can get updated Javascript coverage statistics while the web page is loading and while interacting with the page.

• Easy to use: Users with different levels of expertise can install and use the tool to analyse coverage. Additionally, there is no need to write tests, modify the web application’s code, save the inspected web page locally, manually change proxy settings, etc. When the extension is activated in a Chrome browser, users just navigate through web pages and get coverage statistics on the fly.

• It’s free and open source!

         
Want to try it out? Install ScriptCover and let us know what you think.

We envision many potential features and improvements for ScriptCover. If you are passionate about code coverage, read our documentation and participate in discussion group. Your contributions to the project’s design, code base and feature requests are welcome!

Permalink | Links to this post | 2 comments  

Google Test Analytics - Now in Open Source

By Jim Reardon

The test plan is dead!

Well, hopefully.  At a STAR West session this past week, James Whittaker asked a group of test professionals about test plans.  His first question: “How many people here write test plans?”  About 80 hands shot up instantly, a vast majority of the room.  “How many of you get value or refer to them again after a week?”  Exactly three people raised their hands.

That’s a lot of time being spent writing documents that are often long-winded, full of paragraphs of details on a project everyone already knows to get abandoned so quickly.

A group of us at Google set about creating a methodology that can replace a test plan -- it needed to be comprehensive, quick, actionable, and have sustained value to a project.  In the past few weeks, James has posted a few blogs about this methodology, which we’ve called ACC.  It's a tool to break down a software product into its constituent parts, and the method by which we created "10 Minute Test Plans" (that only take 30 minutes!)

Comprehensive
The ACC methodology creates a matrix that describes your project completely; several projects that have used it internally at Google have found coverage areas that were missing in their conventional test plans.

Quick
The ACC methodology is fast; we’ve created ACC breakdowns for complex projects in under half an hour.  Far faster than writing a conventional test plan.

Actionable
As part of your ACC breakdown, risk is assessed to the capabilities of your appliciation.  Using these values, you get a heat map of your project, showing the areas with the highest risk -- great places to spend some quality time testing.

Sustained Value
We’ve built in some experimental features that bring your ACC test plan to life by importing data signals like bugs and test coverage that quantify the risk across your project.

Today, I'm happy to announce we're open sourcing Test Analytics, a tool built at Google to make generating an ACC simple -- and which brings some experimental ideas we had around the field of risk-based testing that work hand-in-hand with the ACC breakdown.

Defining a project’s ACC model.

Test Analytics has two main parts: first and foremost, it's a step-by-step tool to create an ACC matrix that's faster and much simpler than the Google Spreadsheets we used before the tool existed.  It also provides visualizations of the matrix and risks associated with your ACC Capabilities that were difficult or impossible to do in a simple spreadsheet.

A project’s Capabilities grid.

The second part is taking the ACC plan and making it a living, automatic-updating risk matrix.  Test Analytics does this by importing quality signals from your project: Bugs, Test Cases, Test Results, and Code Changes.  By importing these data, Test Analytics lets you visualize risk that isn't just estimated or guessed, but based on quantitative values.  If a Component or Capability in your project has had a lot of code change or many bugs are still open or not verified as working, the risk in that area is higher.  Test Results can provide a mitigation to those risks -- if you run tests and import passing results, the risk in an area gets lower as you test.

A project’s risk, calculated as a factor of inherent risk as well as imported quality signals.

This part's still experimental; we're playing around with how we calculate risk based on these signals to best determine risk.  However, we wanted to release this functionality early so we can get feedback from the testing community on how well it works for teams so we can iterate and make the tool even more useful.  It'd also be great to import even more quality signals: code complexity, static code analysis, code coverage, external user feedback and more are all ideas we've had that could add an even higher level of dynamic data to your test plan.

An overview of test results, bugs, and code changes attributed to a project’s capability.  The Capability’s total risk is affected by these factors.

You can check out a live hosted version, browse or check out the code along with documentation, and of course if you have any feedback let us know - there's a Google Group set up for discussion, where we'll be active in responding to questions and sharing our experiences with Test Analytics so far.

Long live the test plan!

Permalink | Links to this post | 9 comments  

Google JS Test, now in Open Source

By Aaron Jacobs

Google JS Test is a JavaScript unit testing framework that runs on the V8 JavaScript Engine, the same open source project that is responsible for Google Chrome’s super-fast JS execution speed. Google JS Test is used internally by several Google projects, and we’re pleased to announce that it has been released as an open source project.

Features of Google JS Test include:

• Extremely fast startup and execution time, without needing to run a browser.

• Clean, readable output in the case of both passing and failing tests.

• An optional browser-based test runner that can simply be refreshed whenever JS is changed.

• Style and semantics that resemble Google Test for C++.

• A built-in mocking framework that requires minimal boilerplate code (e.g. no $tearDown or$verifyAll calls), with style and semantics based on the Google C++ Mocking Framework.

• A system of matchers allowing for expressive tests and easy to read failure output, with many built-in matchers and the ability for the user to add their own.
  
  

See the Google JS Test project home page for a quick introduction, and the getting started page for a tutorial that will teach you the basics in just a few minutes.

Permalink | Links to this post | 3 comments