Published 2010-06-30 @ 00:34
I’ve been working on this idea with autotest.
Here is the state machine that autotest currently uses:
Autotest starts by running all tests. When there is a failure, autotest reruns the failures until they’re all passing. Once they’re all passing, autotest reruns all tests to ensure you didn’t break anything.
This last state transition from red to green is where I’d like to change things. Here is what I’m thinking:
I want to make a stack, a tree really, of failure paths. These paths will be defined by the autotest “flavor” that is running (vanilla, rails, rspec, etc.). Illustrated above is a fictional rails setup with failures. The top two boxes are the same scenario and run the same.
On the left is “feature development mode”. (I’m not settled on these names.) In that mode, autotest would run all leaf nodes all the time. After getting test_a to pass, the other test failures would rerun as well as moving up to running all of test_x.rb to verify nothing else in test_x.rb is broken. This strategy is specialized to provide the absolute fastest feedback you can get with a minimal amount of extra tests run on any given pass.
On the right is “refactoring mode”. In that mode, autotest would run all failures at the deepest stratum. After getting test_a to pass, autotest reruns the other failing test methods. Once they’re all passing, it moves up one level and reruns the failing test files. This strategy is specialized to allow you to focus on fixing intentional breakage as you refactor or otherwise change a single point in your code.
The difference between the two is potentially subtle. I’m not entirely sure it is necessary at this point… but I need to run with both strategies in order to really tell.
What do you think? Is refactoring mode necessary? Is this entire thing overkill?