crypt.h: No such file or directory about colbert

[filpia]

@okhat I had success by running cp /usr/include/crypt.h /opt/conda/include/python3.8/ to copy the crypt.h file into the location where it's expected at runtime. In your case I think you'd modify to cp /usr/include/crypt.h /opt/conda/envs/colbert/include/python3.8/.

Another choice depending on how much you're able to modify your environment is...

conda install --channel=conda-forge libxcrypt

export CPATH=/opt/conda/include/

or possibly even export CPATH=/opt/conda/envs/colbert/include.

Either way, good luck!

[Jimmy9507]

Yeah I also copied crypt.h to python environment and it works. But wondering if there is a easy way to include it in conda env creation step.

[filpia]

Digging into this a bit more, I have a hunch that the issue may be in conda-forge as mentioned here.

I copied /usr/include/crypt.h to the /opt/conda/include/crypt.h and that fixed my issue. I'm looking close into whether there's a modification to the conda build to ensure that crypt.h is installed in the include/ of the conda environment being used.

[filpia]

I just started seeing this issue today too in a sagemaker-deployed job and haven't figured out a fix for it. Are you using a Red Hat Linux base Docker image?

[okhat]

Ah thanks @filpia for the catch.... I don't really have a strong insight on this right away, it's entirely outside colbert per se, it's just the compiler as you noted

[jinghan23]

Same here. Copying from /usr/include/ works and curious why

[filpia]

@jinghan23 because in this case pytorch is looking for crypt.h in a specific directory. Previous to the conda-forge change, the file was being created at the anticipated location but following the changes it seems like it's being created elsewhere. This is just a lazy fix to ensure that the file exists where pytorch is expecting. FWIW I'm seeing similar behavior with theano