minitest-bisect

helps you isolate and debug random test failures

Hunting down random test failures can be very very difficult, sometimes impossible, but minitest-bisect makes it easy.

minitest-bisect helps you isolate and debug random test failures.

If your tests only fail randomly, you can reproduce the error consistently by using --seed <num>, but what then? How do you figure out which combination of tests out of hundreds are responsible for the failure? You know which test is failing, but what others are causing it to fail or were helping it succeed in a different order? That’s what minitest-bisect does best.

• minitest_bisect first runs your tests on a per-file basis to minimize the number of tests you need to sift through.
• minitest_bisect next runs the minimized files and figures out your exact failure reproduction.

$ minitest_bisect --seed 3911 example/test*.rb
reproducing...
reproduced
# of culprit methods: 64
# of culprit methods: 64
# of culprit methods: 32
# of culprit methods: 16
# of culprit methods: 8
# of culprit methods: 8
# of culprit methods: 4
# of culprit methods: 2
# of culprit methods: 2
# of culprit methods: 1

Minimal methods found in 10 steps:

ruby -Ilib -e 'require "./example/test_bad1.rb" ; require "./example/test_bad4.rb"' -- --seed 3911 -s 48222 -n '/^(?:TestBad1\#test_bad1_1|TestBad4\#test_bad4_4)$/'

Final reproduction:

Run options: --seed 3911 -s 48222 -n "/^(?:TestBad1\\#test_bad1_1|TestBad4\\#test_bad4_4)$/"

# Running:

.F

Finished in 0.505776s, 3.9543 runs/s, 1.9772 assertions/s.

  1) Failure:
TestBad4#test_bad4_4 [/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/example/helper.rb:16]:
muahahaha order dependency bug!

2 runs, 1 assertions, 1 failures, 0 errors, 0 skips

Get The Code

If you just want to use minitest-bisect, you can install it via RubyGems:

gem install minitest-bisect

If you want to hack on minitest-bisect, clone it from GitHub:

git clone git://github.com/seattlerb/minitest-bisect

Latest Activity