Hunting down random test failures can be very very difficult, sometimes impossible, but minitest-bisect makes it easy.
minitest-bisect helps you isolate and debug random test failures.
If your tests only fail randomly, you can reproduce the error
consistently by using
--seed <num>, but what then? How do you figure
out which combination of tests out of hundreds are responsible for the
failure? You know which test is failing, but what others are causing
it to fail or were helping it succeed in a different order? That’s
what minitest-bisect does best.
$ minitest_bisect --seed 3911 example/test*.rb reproducing... reproduced # of culprit methods: 64 # of culprit methods: 64 # of culprit methods: 32 # of culprit methods: 16 # of culprit methods: 8 # of culprit methods: 8 # of culprit methods: 4 # of culprit methods: 2 # of culprit methods: 2 # of culprit methods: 1 Minimal methods found in 10 steps: ruby -Ilib -e 'require "./example/test_bad1.rb" ; require "./example/test_bad4.rb"' -- --seed 3911 -s 48222 -n '/^(?:TestBad1\#test_bad1_1|TestBad4\#test_bad4_4)$/' Final reproduction: Run options: --seed 3911 -s 48222 -n "/^(?:TestBad1\\#test_bad1_1|TestBad4\\#test_bad4_4)$/" # Running: .F Finished in 0.505776s, 3.9543 runs/s, 1.9772 assertions/s. 1) Failure: TestBad4#test_bad4_4 [/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/example/helper.rb:16]: muahahaha order dependency bug! 2 runs, 1 assertions, 1 failures, 0 errors, 0 skips
gem install minitest-bisectIf you want to hack on minitest-bisect, clone it from GitHub:
git clone git://github.com/seattlerb/minitest-bisect