Flaky Specs #1
Interaction between dbcleaner and seeded values in subsequent specs
Last updated
Was this helpful?
Interaction between dbcleaner and seeded values in subsequent specs
Last updated
Was this helpful?
I'm looking at test suite reliability (one of my many hats).
One came up yesterday in the context of a build (and a rebuild cleared it suggesting the issue was isolated to the test ordering and interaction).
was the failing component
Seeing this I tried to reproduce it locally (I just copied the whole chunk of files onto the command line and ran it) - I got the same two failures as the reporter.
At that point, I was able to open an issue capturing this - and the less than stimulating bisection quest began.
RSpec's --bisect option is pretty clever - given some failing spec list, it first runs the specs completely to verify the issue is reproduced (and separates the failures from the successes)
It runs a second test to see if this appears to be order dependent (this failed for me a few times while I was working, and I had to do the first few bisections manually). If that works, it does subsequent halving (bisection search) of the problem until it can offer a minimum failing ordering. It works on the example, not file level, so the output is often a little too granular, and in my case it came in the reverse order, however it did produce the right set:
One thing to factor in is time - since this is not running in parallel from what I can see - you'll want to capture as small a set up front as you can (in my case, I was able to omit all of the files listed after the first failing test, since I knew these ran in order instead of randomized), as you'll run every example multiple times.
it turns out the underlying problem was an interaction between some seed data that's imported during spec_helper or rails_helper being loaded (this happens twice, once in the end to end setup, once in the general unit test setup) from a csv file (outside of the normal "seed data" format), and being cleared by one of two tests using a :truncation db cleaner strategy. A little looking around led me to find work done last June to introduce db cleaner and use truncation (instead of transactional guarantees) due to a system test hanging in a create statement during CI test runs.
I'm in the process of removing that since it might actually be worse than an occasional hang - I'm also running the import feed spec locally 100 times to observe it completes in the same time consistently without hanging, so far so good, but I think this may only manifest in travis when running the spec suite.