PackagesNotFoundError when setting up virtual environment about coding-for-economists

[banksad]

Sounds good, I tested the Dockerfile before and it started to run fine but stalled when trying to resolve all the package conflicts.

Will test this tomorrow

[aeturrell]

Unfortunately this looks like a Windows issue; conda forge has pre-built binaries for MacOS and Linux only, so running the build command on Windows will fail.

More generally, the environment is a mess! This is chiefly because of the large number of packages. To make the environment more consistent and reproducible, I've thought about:

1. cutting down the number of packages
2. building a docker container (tried this a couple of times without success)
3. having different environments for different parts of the book (not clear if this is supported by Jupyter Book though)
4. using poetry instead (tried this but again little success in getting a reproducible environment)

It would be desirable to have an environment that reliably builds across Mac, Linux, and even Windows, and a pre-built Docker container that users could interact with via Binder (this option is available in the book but doesn't currently work). It looks like 1 is probably the best long-term solution—but it would be useful to know the packages that cause the inconsistency so we could work around them.

[banksad]

I've had a look at package conflicts and there are quite a lot of them (conflicts2.txt).

I tried installing in sections and also experimented with pip-compile and there doesn't seem to be a set of specific versions that avoid conflicts. As you say, it's pretty difficult / impossible to achieve this given the number of packages.

Agree with the four options you've suggested (poetry looks interesting).

I guess in the interim it's sufficient to just have contributors install an environment as close to yours as possible - given all of the packages are being used for a small set of isolated examples.

Therefore it might be good to extract the specific versions of packages you wrote the book with using bash conda list --export > package-list.txt, so that others can sequentially install that list, and then at least the book can be tested in a Windows environment where people have sequentially installed that list. Then when new packages are added or versions are upgraded, tests can be rerun from this point. What do you reckon?

[aeturrell]

That is indeed quite a list of inconsistencies! I've created a package-list.txt file now, so you should be able to see exact versions. I'll have a think about how to tackle approach 1.

[aeturrell]

Merging of #21 sees reproducible builds (at least for the most recent commit) arrive via a dockerfile. I've also pushed a pre-built image to the Coding for Economists dockerhub—be warned though, it's 12GB! Using Mamba, I managed to get the build time down to something quite sensible.

Can you check if you can build or alternatively pull down and run the image?

[banksad]

Success! Took about an hour

[aeturrell]

An hour is a long time, but a previous attempt took 6 hours to build so I'm counting this as a win on both reproducibility and build-time. There's a future question of reducing the number of packages to make everything a bit more agile (and perhaps allow dev not in a docker env...), but I'll close this as the immediate concern is addressed.