Best practices for automated testing in downstream libraries? about mujoco

[yuvaltassa]

Great question!

One answer is to make golden tests but

1. Run for not too many timesteps (10-100) and assert difference to some epsilon that is large for numerics but small for the physics (1e-6ish).
2. Make it very easy for you to regenerate your golden data.

This way your test will tell you if Something Very Bad happened and maybe you'll find yourself updating the goldens a few times after seeing that everything is actually fine.

Another answer is to pin to a version and only upgrade when you really want a new feature, but then check carefully that everything is okay.

The definition of "everything is okay" really depends on your use case. If you are writing a suite of benchmarks then just pin the version and never upgrade. If you are doing something else then write whatever corresponds to "is everything okay?" for you...

Does this make sense?

[sibocw]

Thanks Yuval! These are very helpful. I will also specify the type of runner when running these tests with GitHub Actions. Consequently, it might become a good idea to avoid running the tests locally on machines with different architectures and rely on CI/CD instead.