This is a great concept, and one that I've had a lot of questions about in my test role in a CI-focused team.Two specific questions:1) How well does this scale when you have a much more larger and more complicated graph of test (i.e. orders of magnitude)?2) When running system-level, rather than unit-level tests, can one really map tests to limited domains enough to significantly cut down the test fanout for a given set of changes?Thanks for any feedback.
Great concept; one that has been discussed in my team more than once.Two questions: o Does this really work when the number of tests, number of software modules and the corresponding graph complexity gets orders of magnitude bigger? o Any thoughts on if/how this can apply when the tests are more system-oriented and hence most tests have a transitive relationship with most software modules?Thanks for this post...love reading the Google testing blog.
That's amazing. But, how do you maintain the dependencies? Is it each individual test owner's responsibility to set his/her tests dependencies?
Can you give more info about the dependency graph generation?Is it an automated process (ie: based on build system / makefiles / ...) or do the developers need to manually specify the dependencies?
Intresting, i have been thinking about this idea for a long time. Could be useful for upgrades aswell.How does the framework handle callback interfaces?-Kristoffer
It's an interesting idea, applying the dependency management to the test runs and not only the builds.But I have a question: in the first example you can track down that both change #2 and change #3 are breaking gmail_server_tests independently. But how do you go back in the version control system and fix those changes individually? Do you go back to pre-change #2 and then serialize the (fixed) changes? Another question is that now you can know earlier if it was change #2 or #3 that broke the tests, but you will not know (till later) if the combination of change #2 and #3 (also) breaks the tests.Is this a tradeoff you agree upon since maybe individual changes breaking the build are more common than combination of changes?Thanks for the interesting post!
I do not quite understand, but I'll learn it. Thank you for this interesting information.
Does the dependency graph get updated by the compiler at compile time ? This would work great for currently extant/new features tied to the current dependency graph/map but how do the dependencies tie in with third party or external systems (same as the call back question posted by an earlier comment) that could get impacted by a change made within google's code base? Are their dependencies also mapped into this graph? Honestly, do you guys ever miss any tests that should have been run - how often are these a result of machine error vs human error?
@John1. This scales pretty well at the current scale as described in http://google-engtools.blogspot.com/2011/05/welcome-to-google-engineering-tools where all these numbers correspond to same code repository. Yes, it works! :)2. System level tests are the ones that tend to be flaky, so even when they cannot be cut down to an extent as much as the unit tests, running them fewer that at every change definitely helps. @Alex/@PJayTycy : Build rules are used to maintain dependencies. See similar discussion in: http://google-engtools.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html. The build rules are defined as per compiler specifications, and then the dependency graph generation is automatic.@krisskross Can you elaborate more?@PJ: There are occasional instances when we do miss tests because of software bug, but it is rare enough that teams can continue to rely on the system.
doit is an open-source project that was created to be able to handle this kind of "runs only those tests for every change".For running python unittests, there is py.test plugin that integrates with doit to automatically find imports for python files and keep track of dependencies.
It sounds like this is more about helping the RCA process than saving time/resources in running the tests. It's great to have a quick feedback about how your single change passed/failed the regression suite, but like John mentioned, the system-level tests never map to a single component, and so the graph becomes much more complex. In our previous build system we had a similar system which was based on code coverage data. Test selection was automatic based on the code every changeset changed.
nice..its really usefull, i have try n make it like a personal barometer.
The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.