Test regressions since 11496eb5761761b (Fix source filtering) about sybil

[cjw296]

Where and how are you running the tests? CircleCI seems happy, which suggests you're doing something different.

[mgorny]

I think what makes the difference is that I'm running them on tmpfs which just happens to order files differently here.

cd /tmp
git clone https://github.com/simplistix/sybil
cd sybil
PYTHONPATH=. pytest

[cjw296]

Okay, and why are you doing that?

My inclination here is just to close this ticket with "sorry, run the tests like the CI does", they also pass for me fine on macOS for local dev, so please try and provide a compelling reason for what you're doing...

[mgorny]

Gentoo's purpose in running tests on production systems is to ensure that packages that are installed on production systems actually work and potential issues that aren't trivially reproducible are caught before they explode in users' face.

In this case, it's not "me being wrong vs CI being right". It's just a case of unpredictable ordering that can be influenced by random factors (like files being extracted/checked out in specific order). FWICS the file list was sorted prior to the mentioned commit and the mentioned commit removed sorting for some reason.

Would it be of any help if I created a PR to fix that? As I said, one of the failures can be trivially resolved by restoring sort. I'm working on the other one still.

[mgorny]

As I suspected, the fix is trivial and does not have any downsides. I've filed #35 with it.