GTAC: Call for Proposals

Posted by Lydia Ash - GTAC Conference Chair

Overview
The Test Automation Conference has been hosted by Google over the past several years bringing together a select set of industry practitioners as speakers and participants around the topic of software testing and automation. The annual conference provides a forum for presentations on important topics in our field connecting professions with others in their field through collaboration and targeted breakout sessions, but also has served the testing community by bringing the presentations public allowing anyone access to the information.

GTAC conference participants are typically software engineers who are actively working on problems of software quality, automated testing, and testing techniques. GTAC conferences cover a number of areas, but many presentations focus on themes such as advanced techniques in quality evaluation, advanced approaches to automating software testing, and experiences and findings from quality efforts on software projects.

The Conference
The conference is a two day event comprised of a single track of presentations. The philosophy is to engage a small set of active participants who all experience the same topics carrying the discussions into lightning talks, speaker Q&A, and topical discussion groups. The emphasis for the 2008 GTAC conference is solving the hard engineering problems in the quality of our software. Each year we have worked to identify a location that has a unique profile of technology professionals. This year the conference will be held in Seattle, WA, USA on October 23 and 24.

Presentations are targeted at experienced engineers actively working on software quality. We encourage innovative ideas, controversial experiences, problems, and solutions that further the discussion of engineering and software quality. Presentations are generally 45 min in length and speakers should be prepared for an active question and answer session following their presentation.

Process of Selection
All presentation submissions will be handled electronically. Presentation proposals should be a relatively detailed extended abstract including the topic, outline, and details of what will be presented. Presentation proposals should be emailed to gtac@google.com
All presentation proposals must be received by June 6, 2008. Where employer or disclosure authorization is needed, the author(s) will need to obtain this prior to submitting their proposal.

The program committee will evaluate proposals based on the quality and relevance. All submissions will be held confidentially prior to contacting the selected presenters and the publication in the proceedings.

Notification
Accepted proposal authors will be contacted on or before July 10 to confirm their availability and travel needs. Authors of proposals not selected will be notified on or before July 10. Accepted proposals will be presented at the conference and made available to the public on YouTube.

Copyright
TAC requires presenters to present at the conference and permit their presentation to be made available on YouTube.

Important Dates for Presentations
June 6 - Final deadline for proposal submissions for presentations
July 10 - Deadline for selected presenters to be contacted by the selection committee and notified of their acceptance
October 23 and 24 - GTAC conference in Seattle

Attendees
TAC conference has worked to invite a select audience, each member applies to the conference for committee selection. This is to ensure active participation from each attendee and provide a variety of technical perspectives to interact, discuss, and network.

Important Dates for Attendees
July 7 - Call for attendee profile submissions
July 25 - Deadline for attendee profiles
August 11 - Selected attendees to be notified by the conference committee, registration opens
August 29 - Registration deadline, wait-list opens
September 19 - Wait list notifications and attendee closure
October 23 and 24 - GTAC conference in Seattle

Questions
If you have questions about the submission process or potential topics please send a mail to us at: gtac@google.com
Please see the Google Testing blog for more information: http://googletesting.blogspot.com/search/label/GTAC

Permalink | Links to this post | 1 comments  

GTAC 2008 in Seattle

Before the end of our last Google Test Automation Conference in August 2007, we were already getting questions from participants and blog readers wondering about the next conference. Now we can tell you when and where that will happen (drum roll please)... the 2008 Test Automation Conference will be held October 23 and 24 in Seattle. More details will be coming shortly...

As with previous years, the focus of the conference will be solving software engineering challenges using tools and automation, with special focus on SaaS. Engineers in the testing world are frequently so busy shipping software that they do not take the time to share the technical details of the work they are doing, the approaches that are working, and the lessons they have learned. There will be a call for proposals in late April.

As has been the precedent in previous years, our conference is for active and vocal participation, not quiet attendance. GTAC is a place to share great ideas and to get challenged. As we have done previously, attendants will apply by proposing what they will bring to the conference and how they can further the discussions. Applications for attendance will be opened in late June.

We are hard at work developing the conference. Please send suggestions, questions and recommendations to: gtac@google.com or post your comments here to the blog.

Permalink | Links to this post | 4 comments  

Testing systems with large and complex test spaces

Posted by Adam Porter, Professor Department of Computer Science, University of Maryland, and Associate Director, University of Maryland Institute for Advanced Computing Studies

[From time to time, we invite software testing experts to write about their ideas and research. - Ed.]

Motivation:

Software systems are getting larger, more complex and more configurable all the time. While beneficial in many ways, these changes dramatically increase testing obligations.

For example, today's systems are increasingly built to run on multiple OS, compiler and library platforms. They are composed from multiple components, each with multiple versions. They are configured by manipulating numerous compile- and run-time options. Additionally, distributed applications add an allocation dimension in which runtime topologies can vary widely.

This situation is further complicated by agile and flexible development practices in which systems are evolved incrementally over short, but varying update cycles and in which development teams may be geographically distributed.

Basically, each new configuration dimension increases the number of potential runtime configurations combinatorially. Since each of these configurations might behave differently or host different bugs, each of these configurations, at least in theory, must be tested. Furthermore, this increased amount of testing must be done in shorter and shorter time frames because the systems themselves are changing faster than ever.

The Skoll project:

Our research aims to create better, faster and more powerful techniques for testing these kinds of systems. Our vision is to redesign traditional testing processes so that they can be executed around-the-world, around-the-clock. These processes are logically divided into multiple tasks that are distributed intelligently to client machines around the world and then executed by them. The results from these distributed tasks are returned to central collection sites where they are merged and analyzed to complete the overall QA process.

At the second Google Test Automation Conference, I and my colleague Atif Memon presented an infrastructure and approach called Skoll that was created to support this vision. The video is embedded below.

Skoll in action:

To give you a better sense of how this works we're going to walk you through how we set up a Skoll server to run a simplified version of the continuous build, test and integration process we've developed to run the MySQL Build Farm Initiative. This process runs on computing resources that are temporarily volunteered by members of the MySQL developer and user communities. After reading the rest of this post, we hope you too will be ready and willing to volunteer as well!

MySQL, as you probably know, is a very popular open source database comprising over 2 million lines of code. It is developed by a world wide community and runs on many different platforms, has over 150 configuration options, and allows substantial static and runtime end-user customization. For instance, you can select different database front- and back-ends, and you can run the system with different runtime topologies to support database replication. In short, it's exactly the kind of system we had in mind when we created Skoll.

We first started speaking with MySQL developers after the first Google Test Automation Conference. They were interested in running a continuous build, integration and test (CBIT) process. They had several goals, such as: testing a broader variety of configurations, not just a handful of popular ones; giving developers greater visibility into the quality and stability of the system; improving bug fix turnaround time; and managing test data to enable long term statistical analysis and improved decision making.

Since they couldn't seem to find anything off-the-shelf that was sufficiently targeted to their needs, we worked with them to develop a Skoll-based CBIT process. This process has several parts: defining a configuration model, implementing a sampling strategy, executing the tests, and analyzing and visualizing the results.

We will discuss each of these below. Readers who just want to run a client can jump straight to the Section marked Test execution.

Some more details:

Configuration Model: We are starting out by looking at 23 options. There are some inter-option constraints as well. For example, a configuration can compile in support for either the libedit library (--with-libedit) or the readline library (--with-readline), but not both. Here's a ling to the current configuration model . We will expand this model as we gain more experience with this process and more insight into the key issues concerning MySQL.

Sampling strategy: There are over 48 million unique configurations in this test space. Since testing 1 configuration can take up to 2 hours and because new releases come out more or less daily, exhaustive testing of each check-in is clearly impossible. Therefore, we only test specially-chosen subsets of configurations in which all t-way (2 <= t <= 4) combinations of option settings are tested at least once. Our particular algorithm for selecting these configurations, which we invented with Myra Cohen of the University of Nebraska, works incrementally. First it tests 3 sets of ~23 configurations, each set of which covers all combinations of settings between every pair of options (i.e., 2-way combinations). We then move up to testing 3 sets of ~84 configurations that cover all settings of every triple of options and then move up again to cover all quadruples (~272 configurations), also 3 times each. This allows us to test low level interactions quickly, even if new releases come in before all levels of interactions can be tested. As we gain experience with this process, we will evaluate whether and how much we may need to increase t.

Test execution: To participate in this process, users can go to our MySQL 5.1 project page. On that page you can find links to download a client along with instructions on how to install and run it. This client is a simple Perl script that connects to a Skoll server and asks for a test job. The server examines its internal databases, selects an outstanding job and returns it to the client who then executes it. Currently, testing a configurations involves compiling MySQL in a specific configuration and then running about 750 MySQL-supplied tests on that configuration. The test scripts determine which test cases can be run in the current configuration and runs them. After completing the tests, the client uploads the results to the server.

Feedback, analysis and visualization: For this process, we are interested in understanding where configuration-related bugs might be hiding. To figure this out, we periodically analyze test results by building a classification tree for each test that has failed in a minimum number of configurations. These trees model test failures in terms of the configuration option and settings that were set when the tests failed. Users can look at a web page to see the current test results . This page shows summary statistics for each build ID. Clicking on a build ID takes you to a detail page listing each test that failed a minimum number of times. Each test presents classification information, both in a raw form and as an interactive treemap.

Current status:

We have just started running this process on a continuous basis using an a small number of MySQL developer community machines. We hope to bring many more test machines online in the coming weeks. Please check out the results web page to watch our progress. And don't let everyone else have all the fun -- download your client today! Thanks!

Permalink | Links to this post | 0 comments  

GTAC videos now online

Posted by Allen Hutchison, Engineering Manager

We finished up the second annual Google Test Automation Conference on Saturday with a boat cruise around Manhattan. There's still a lot of work for us to do to wrap up all the loose ends, but one that we have gotten to right away is posting the videos on YouTube.



We hope that you enjoy the videos, and we'd like you to join everyone already conversing on our Google Group. Several people have posted their reviews of the conference; you can find them in the comments on the GTAC Community Thread, or through a blog search for GTAC. Finally, our team posted several conference photos on Picasa Web Albums.

Permalink | Links to this post | 8 comments  

GTAC Community Thread

Posted by Allen Hutchison, Engineering Manager

We're moments away from the beginning of the Google Test Automation Conference. If you are attending in person, then I'll see you shortly. If you plan to watch the videos on YouTube, then keep an eye on this space. We'll be posting links to the talk videos throughout the next few days. The conference isn't just about the talks, though: this is a community conference. So use the comment space on this post to point people to your GTAC Live Blog, your pictures from the conference, and your thoughts about the conference.

We also have a Google Group set aside for discussion about the conference and test automation. There are several people there from past conferences as well as this one.

Permalink | Links to this post | 29 comments  

GTAC registration closed

Posted by Harry Robinson, Software Engineer in Test

Thank you to everyone who applied to attend the Google Test Automation Conference in New York on August 23rd-24th. We received lots of applications for a very limited number of seats, so the choices were not easy.

The sign-up website is now closed, and this week we'll be sending out acceptances and wait list notices. If you get an acceptance, congratulations -- we look forward to seeing you in New York. If you don't receive one, you'll be on the wait list, and we'll let you know if spaces become available. Either way, we'll let you know. If you don't hear from us in the next week, drop us a note.

All the presentations from the conference will be posted on YouTube Google Channel within a New York minute after the speaker leaves the podium (well, maybe 72 New York minutes). We look forward to seeing you at the conference.

Permalink | Links to this post | 4 comments  

Sign up to attend our Test Automation Conference

posted by Allen Hutchison, Engineering Manager

A few months ago we announced that we would be holding a Google Test Automation Conference in New York City on August 23-24. Today, I'm happy to tell you that we've finalized the speaker list and have opened the attendee registration process.

The conference is completely free; however, because we have room for only 150 people, everyone who wishes to attend must apply and be accepted. You can apply to attend via this link, and we'll let you know by June 15 if your application has been accepted.

At last year's conference, we found that the informal hallway discussions were tremendously valuable to everyone who came, and we would like to continue that custom. Therefore, the main point in the application process is to tell us how you can contribute to the conversation and what you would like to learn from the conference.

To get you started, here is what our speakers will be talking about:

• Apple Chow and Santiago Etchebehere -- Building a flexible and extensible automation framework around Selenium
• Ali-Akber Saifee -- muvee Framework for Autonomous Testing
• Olivier Warin -- Spirent's WorkSuite Manager
• Douglas Sellers -- CustomInk Domain Specific Language for automating an AJAX based application
• Julian Harty -- Mobile Wireless Test Automation
• Matt Heusser and Sean McMillan -- Examining Interaction-Based testing
• Risto Kumpulainen -- Automated testing for F-Secure's Linux/UNIX Anti-Virus products
• Sergio Pinon -- User Interface Functional Testing with AFTER (Automated Functional Testing Engine in Ruby)
• Vivek Prahlad -- Functional Testing Swing applications with Frankenstein
• Simon Stewart -- Web Driver
• Adam Porter and Atif Memon -- Skoll distributed continuous quality assurance system
• Cedric Beust -- TestNG
• Hadar Ziv -- Specification-based Testing

We look forward to your application and to a great conference.

Permalink | Links to this post | 15 comments  

GTAC Speaker Submission Window Closed

Posted by Allen Hutchison, Engineering Manager

Thank you to everyone who submitted a proposal to speak at the Google Test Automation Conference. We'll be reading each proposal closely and selecting those that we think are best for the conference. If you submitted a proposal, then we'll be getting in touch with you directly to let you know if it was accepted or not.

If you didn't submit a proposal, but are interested in attending, then check back on the Google Testing Blog on May 7 to see the conference schedule and instructions for how to apply to attend.

Permalink | Links to this post | 1 comments  

Google Test Automation Conference submission deadline coming soon

Posted by Allen Hutchison, Engineering Manager

If you recall, we announced the Google Test Automation Conference here about a month ago. To everyone who has submitted a proposal to speak at the conference, thank you. To those of you who haven't yet submitted your proposal, I want to remind you that the deadline for submissions is this Friday, April 6.

We're looking for speakers with exciting ideas and new approaches to test automation. If you have a subject you'd like to talk about, please send an email to gtac-submission@google.com and include a description of your 45 minute session in 500 words or less (no attachments, please). Deadline for submissions is April 6.

Permalink | Links to this post | 3 comments  

2nd Annual Google Test Automation Conference

Posted by Allen Hutchison, Engineering Manager

Some of the most difficult challenges in creating great software are guaranteeing it works every time, for every customer, ensuring that it will scale well, and making it accessible to all users. Over the years, languages have become easier to work with, frameworks have become extensible for the creation of several products, and integrated development environments have made the software developer faster and more productive. But automation techniques, extensible testing frameworks, and easy-to-use test tools have lagged behind. While there are many good solutions for automated testing, there is plenty of room for innovation.

I'm happy to announce that Google will be hosting our 2nd Annual Google Test Automation Conference (GTAC) in our New York office on August 23 and 24, 2007. Our goal is to create a collegial atmosphere where participants can discuss challenges facing people on the cutting edge of test automation, evaluate solutions for meeting those challenges, and have a little fun.

Call for Proposals
We're looking for speakers with exciting ideas and new approaches to test automation. If you have a subject you'd like to talk about, please send an email to gtac-submission@google.com and include a description of your 45 minute session in 500 words or less (no attachments, please). Deadline for submissions is April 6.

We're planning to have 10 people give presentations at the conference, followed by adequate time for discussion. If you'd like to attend as a non-speaker, please check back to this page on May 7, when we'll post our slate of speakers and information about how to attend.

Permalink | Links to this post | 28 comments