Tuesday, April 01, 2014

The *Real* Test Driven Development


Update: APRIL FOOLS!


by Kaue Silveira

Here at Google, we invest heavily in development productivity research. In fact, our TDD research group now occupies nearly an entire building of the Googleplex. The group has been working hard to minimize the development cycle time, and we’d like to share some of the amazing progress they’ve made.

The Concept

In the ways of old, it used to be that people wrote tests for their existing code. This was changed by TDD (Test-driven Development), where one would write the test first and then write the code to satisfy it. The TDD research group didn’t think this was enough and wanted to elevate the humble test to the next level. We are pleased to announce the Real TDD, our latest innovation in the Program Synthesis field, where you write only the tests and have the computer write the code for you!

The following graph shows how the number of tests created by a small feature team grew since they started using this tool towards the end of 2013. Over the last 2 quarters, more than 89% of this team’s production code was written by the tool!

See it in action:

Test written by a Software Engineer:

class LinkGeneratorTest(googletest.TestCase):

  def setUp(self):
    self.generator = link_generator.LinkGenerator()

  def testGetLinkFromIDs(self):
    expected = ('https://frontend.google.com/advancedSearchResults?'
                's.op=ALL&s.r0.field=ID&s.r0.val=1288585+1310696+1346270+')
    actual = self.generator.GetLinkFromIDs(set((1346270, 1310696, 1288585)))
    self.assertEqual(expected, actual)

Code created by our tool:

import urllib

class LinkGenerator(object):

  _URL = (
      'https://frontend.google.com/advancedSearchResults?'
      's.op=ALL&s.r0.field=ID&s.r0.val=')

  def GetLinkFromIDs(self, ids):
    result = []
    for id in sorted(ids):
      result.append('%s ' % id)
    return self._URL + urllib.quote_plus(''.join(result))

Note that the tool is smart enough to not generate the obvious implementation of returning a constant string, but instead it correctly abstracts and generalizes the relation between inputs and outputs. It becomes smarter at every use and it’s behaving more and more like a human programmer every day. We once saw a comment in the generated code that said "I need some coffee".

How does it work?

We’ve trained the Google Brain with billions of lines of open-source software to learn about coding patterns and how product code correlates with test code. Its accuracy is further improved by using Type Inference to infer types from code and the Girard-Reynolds Isomorphism to infer code from types.

The tool runs every time your unit test is saved, and it uses the learned model to guide a backtracking search for a code snippet that satisfies all assertions in the test. It provides sub-second responses for 99.5% of the cases (as shown in the following graph), thanks to millions of pre-computed assertion-snippet pairs stored in Spanner for global low-latency access.



How can I use it?

We will offer a free (rate-limited) service that everyone can use, once we have sorted out the legal issues regarding the possibility of mixing code snippets originating from open-source projects with different licenses (e.g., GPL-licensed tests will simply refuse to pass BSD-licensed code snippets). If you would like to try our alpha release before the public launch, leave us a comment!

33 comments:

  1. This is my favorite april fool's day prank up to now. :)

    As a next step i suggest you find a way to automate the test writing.
    Oh, and please sign me up for your alpha release ;)

    ReplyDelete
  2. For the record, the code inferencer demo program mentioned, Djinn is available at http://hackage.haskell.org/package/djinn and can be installed as the Cabal package "djinn" if you have the Haskell Platform installed. Also, you can do this sort of code inferencing in straight Haskell: see Oleg's demo code here: http://okmij.org/ftp/Haskell/types.html#de-typechecker

    Parametricity is great. In all seriousness, and not just on this first day of the month!

    ReplyDelete
  3. Will I be able to add security loopholes in Google production code by submitting such code + some innocent testcases to some unknown open source project?

    ReplyDelete
  4. Does the team now only write tests for "real TDD"?

    ReplyDelete
  5. I am interested to alpha test this public Program synthesis tool

    ReplyDelete
  6. Very interesting, I definitely want to try the alpha release!

    ReplyDelete
  7. If this is for real, I'd love to try this else it's a good prank, google!

    ReplyDelete
  8. This is very clever prank! Why don't we train Google Brain to write code that requires less testing... at least avoiding mistakes that humans make :-) I'll probably pass on alpha opportunity, but do count me in for beta!

    ReplyDelete
  9. where one would write the test first and then write the code to satisfy it.

    I don't think it would be possible to write the test case and the write the code to satisfy it. If it is possible , it will be the revolution in software era.

    ReplyDelete
  10. Kaue, could you spoil Mew's location, please? Kidding!!!
    BTW, I hope this isn't another prank.. got really excited about it! Great job, anyways!
    Cheers!

    ReplyDelete
  11. Awesome. Is this going to be available for Java as well? The example snippet seems to be in Python.

    ReplyDelete
  12. Fiction imitates real life: I just published a PLDI'14 paper on actually (no joke) generating code for TDD: https://homes.cs.washington.edu/~perelman/publications/pldi14-tds.pdf
    Unfortunately, I have not yet achieved 100x programmer productivity improvements using the real thing.

    ReplyDelete
  13. I dont think this is impossible. The possibility of auto code generation that matches your tests is smart way for going about writing all possible used cases. How ever I am wondering if there is a possibility to even write all test cases given a few cases are written.
    So humans write a few testcases, Code writes more edge test cases be following QA guidelines. These then drive production code. I think this is effective utilization of human knowledge

    ReplyDelete
  14. I'd like to please consider me for the alpha release

    ReplyDelete
  15. This is the only April Fool's prank that got me this year...

    ReplyDelete
  16. I don't bite it! I want to see that!
    So my days are numbered as a developer...Never too late to catch my guitar again and hit the road :-)

    ReplyDelete
  17. Please Consider me for Alpha Release.

    ReplyDelete
  18. I'll start with of course this was an April fool's joke. Wow.

    But for those getting excited at the prospect...

    Of course, writing tests takes about 100th of the time of designing application structure and the writing code, doesn't it? The productivity gains would be nothing like as high as the suggestion.

    My guess would be that best one could hope for in a tool such as this (when it finally does appear) would be around doubling the output.

    Having something write code for you may be fantasy today, but reality soon; the productivity gains are just PURE fantasy.

    ReplyDelete
  19. real intresting stuff, would like to try it

    ReplyDelete
  20. This sounds great! I'd like to try it.

    ReplyDelete
  21. Sounds like an interesting idea, would be happy to try it!

    Thank you

    ReplyDelete
  22. I got to "where you write only the tests and have the computer write the code for you!" before checking the publication date.

    ReplyDelete

The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.