desihub / desiconda

Tools to install DESI dependencies from scratch.

desiconda

Introduction

This package contains scripts for installing conda and all compiled dependencies needed by the spectroscopic pipeline.

Quick start installation

To install desiconda, start with a terminal where you have not configured a different desiconda / desi_environment, then:

# set target
prefix=/global/common/software/desi/users/${USER}/desiconda
mkdir -p ${prefix}

local_copy=/global/cfs/cdirs/desi/users/${USER}/desiconda
git clone https://github.com/desihub/desiconda ${local_copy}
cd ${local_copy}

unset PYTHONPATH
export DCONDAVERSION=$(date '+%Y%m%d')-2.0.1.dev
PREFIX=${prefix} ./install.sh |& tee install.log

To load this version of desiconda:

module use ${prefix}/${DCONDAVERSION}/modulefiles
module load desiconda

Then install a suite of desispec, desiutil, etc. modules:

source scripts/bootstrap-desi.sh

After that you can module load desiutil etc.

Example

Imagine you wanted to install a set of dependencies for DESI software on a cluster (rather than manually getting all the dependencies in place). You plan on installing desiconda in your home directory (${HOME}/software/desi) and you want to have the custom string "my-desiconda" associated with your installation.

You git-cloned desiconda using:

git clone https://github.com/desihub/desiconda /path-to-git-clone/desiconda

You also put all the commands for dependencies you want to install and customizations in the "conf/mypkgs-pkgs.sh" and "conf/myenv-env.sh" files you created (based on the existing conf/default-pkgs.sh and conf/nersc-env.sh), respectively.

This install.sh script, in the top-level directory, will create the environment and install the dependencies and module files. When you run this script, it will download many MB of binary and source packages, extract files, and compile things. It will do this in your current working directory. Also the output will be very long, so pipe it to a log file:

$> DCONDAVERSION=my-desiconda PREFIX=$HOME/software/desi CONF=myenv PKGS=mypkgs /path-to-git-clone/desiconda2/install.sh 2>&1 | tee log

If everything worked, then you can see your new desiconda install with:

$> module use $HOME/software/desi/desiconda/$dcondaversion/modulefiles
$> module avail desiconda

And you can load it with:

$> module load desiconda/$dcondaversion

Configuration

If environment and package files (conf/[envtag]-env.sh and conf/[pkgtag]-pkgs.sh) for your use case already exists in the "conf" directory, then just use them. Otherwise, create or edit files in the "conf" subdirectory that are named after the environment and set of packages you wish to create and install. See existing files conf/default-env.sh and conf/default-pkgs.shfor spectroscopic pipeline dependencies on cori.

For ${NERSC_HOST} == "datatran" installs use:

PREFIX=${prefix} PKGS=datatran install.sh

as in the example above.

Contents of installation

The installation directory (assuming the installation script was called with ${DCONDAVERSION} and ${PREFIX}) will contain directories and files:

${PREFIX}/desiconda/${DCONDAVERSION}/conda
${PREFIX}/desiconda/${DCONDAVERSION}/aux
${PREFIX}/desiconda/${DCONDAVERSION}/modulefiles/desiconda/$DCONDAVERSION
${PREFIX}/desiconda/${DCONDAVERSION}/modulefiles/desiconda/.version_$DCONDAVERSION

desiconda's Issues

pyfftw

pyfftw is required for the blinding (and then likely the BAO reconstruction) codes applied to LSS catalogs. If it could become part of the desi environment, that would be wonderful, but this is not urgent. (The pip install is easy and works for me currently, but since this will be part of official LSS catalogs for Y1, it would be good to have a stable desi version associated with that.)

Please make sure hpsspy 0.4.0 is installed

The hpsspy version needs to be bumped to 0.4.0. It's already on PyPI. We need this to run some final tests on the backup system.

Please add sqlalchemy to the list of conda packages

sqlalchemy is now a requirement of desispec.

Add kapteyn package

This is needed by tilepicker.

Please install the latest desiconda on datatran

The datatran software stack shares desimodules with cori and edison, so now it is looking for a version of desiconda that is not installed. As a result, certain automated processes are not running with the correct version of Python.

desimodules/17.6(51):ERROR:105: Unable to locate a modulefile for 'desiconda/20170613-1.1.4-spectro'
desimodules/17.6(51):ERROR:151: Module 'desimodules/17.6' depends on one of the module(s) 'PW'
desimodules/17.6(51):ERROR:102: Tcl command execution failed: if { [info exists env(NERSC_HOST)] } {
    module load desiconda/20170613-1.1.4-spectro
    prereq desiconda
}

Traceback (most recent call last):
  File "/global/homes/b/bweaver/software/code/datatran/decamUtil/trunk/bin/transfer_noao_to_nersc", line 3, in <module>
    from decamUtil.transfer import main
  File "/global/homes/b/bweaver/software/code/datatran/decamUtil/trunk/py/decamUtil/transfer.py", line 18, in <module>
    from shutil import chown
ImportError: cannot import name chown

Extraneous script defined but not used.

What is the purpose of this line? It is not used in install.sh itself, nor is it exported so that it can be used by other programs.

OpenBLAS / Lapacke

Some packages require lapacke.h headers which are not included when we are using OpenBLAS instead of MKL. If installing OpenBLAS from ubuntu packages (i.e. in the Docker files), we should also install the lapacke-dev package. If installing OpenBLAS from source, we should add the installation of the lapacke headers.

None of this matters for MKL-based installs, which is what we use at NERSC and why we did not notice this with the imaging stack on those machines.

Install NOIRLab software

For DR1, especially public DR1, it would be useful to have some NOIRLab software installed in the DR1 desiconda package. Specifically:

• sparclclient: client for retrieving full spectra.
• astro-datalab: client for querying Data Lab databases, etc.

These don't have an especially long list of requirements. In fact, in a recent test, I was able to pip install --no-deps for these packages and they worked fine in the test-24.4 environment. In other words, most likely all the requirements are already included in desiconda.

Remove stale branches?

The basemap and collabsu branches have already been merged, 18.5 is presumably quite out-of-date, and noconda I'm guessing is an experiment from a few years ago.

Decide on how speclite will be installed

Recent desiconda and pipeline software stacks have speclite installed in two different places. For example in release 22.2, conda list reports that speclite 0.14 is installed, while module list shows speclite 0.15. Presumably the desiInstalled module is hiding the conda or pip installed version of speclite.

This strikes me as somewhat dangerous. If nothing else, when someone is creating a custom environment, they may end up with a version of speclite that is not actually used by the current pipeline.

specutils

specutils does not appear to be part of the standard desiconda installation. I believe in previous releases (2020-2021 era) we had installed it by hand in some desiconda distributions. Currently it is not installed in 22.2 or 22.5. This causes problems for some Prospect notebooks among other things. I realize we don't use specutils much outside of Prospect, but is there a specific reason that forbids us from installing it, or is it just historical reasons?

Load cudatoolkit on Perlmutter

GPU-enabled code requires

module load cudatoolkit

on Perlmutter, but this is not yet handled automatically.

Likely the best solution is to add code somewhere in modulefile.gen so that the cudatoolkit module is loaded in the desiconda module file for installations on Perlmutter.

mkdocs

mkdocs is now needed to build desidatadocs. It is installed in desiconda/20220119-2.0.1 but not desiconda/20230111-2.1.0. It should be part of the standard packages installed with desiconda.

Add fpoffline package

I would like to include desihub/fpoffline in the list of DESI packages used by desiconda.

Shall I make a PR with this added to scripts/bootstrap-desi.sh or is there a different procedure?

Update to fitsio 1.0.5

It looks like the DESI conda env (and corresponding jupyter kernel) use fitsio 0.9.5:

> source /project/projectdirs/desi/software/desi_environment.sh master
> python -c 'import fitsio; print(fitsio.__version__)'
0.9.11

GFA data written since 20191208 cannot be read with 0.9.11 (for unknown reasons) but the latest 1.0.5 can read old and new GFA data.

To reproduce the problem reading recent GFA data with 0.9.11:

import fitsio
fitsio.read_header('/project/projectdirs/desi/spectro/data/20191212/00031386/gfa-00031386.fits.fz', ext='FOCUS1')

which results in OSError: FITSIO status = 204: keyword value is undefined.

Please add fitsverify to the desiconda packages

fitsverify only requires cfitsio.

Investigate installing a (desi)conda environment on top of a base environment.

We currently use miniconda to install a base conda environment. miniconda is tied to Python 3.10. If a later version of Python is desired, then that has to be installed as a separate environment on top of the base environment.

There are a few specific issues that would need to be investigated:

• A module file that performs the equivalent of conda activate would be required. This is certainly possible, it would just require some attention to detail and testing.
  • Perhaps one module initiates the base environment then another module performs the conda activate. This could allow multiple installed environment on top of the same base environment.
• An alternate environment would change assumptions about the desiconda directory structure relative to the base environment. We would need to track down where directories are specified and test.
  • Related: would a separate aux installation be needed for each environment?
• How hard would this hit quotas on /global/common/software/desi?

bootstrap script assumes "master" branches

scripts/bootstrap-desi.sh still assumes that the default branch is 'master' not 'main'. In addition, some packages have hard-coded tags that are out-of-date. For example desitree should now be installed as version 0.6.0, not 0.5.0.

new imaging desiconda for DR8; fix ceres issue

@tskisner @dstndstn @mlandriau

Looking ahead to the DR8 production run, it would be useful to have a new version of the imaging desiconda packages with the tractor/ceres dependencies properly compiled. I've been poking around at the DR7 log files and found many errors along what I've pasted below. Thanks.

module use /global/common/software/desi/$NERSC_HOST/desiconda/20180512-1.2.5-img/modulefiles
module load desiconda

from tractor import ceres
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/aux/lib/python3.6/site-packages/tractor/ceres.py in swig_import_helper()
     13         try:
---> 14             return importlib.import_module(mname)
     15         except ImportError:

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/conda/lib/python3.6/importlib/__init__.py in import_module(name, package)
    125             level += 1
--> 126     return _bootstrap._gcd_import(name[level:], package, level)
    127 

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/conda/lib/python3.6/importlib/_bootstrap.py in _gcd_import(name, package, level)

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/conda/lib/python3.6/importlib/_bootstrap.py in _find_and_load(name, import_)

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/conda/lib/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/conda/lib/python3.6/importlib/_bootstrap.py in _load_unlocked(spec)

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/conda/lib/python3.6/importlib/_bootstrap.py in module_from_spec(spec)

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/conda/lib/python3.6/importlib/_bootstrap_external.py in create_module(self, spec)

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/conda/lib/python3.6/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

ImportError: /global/common/software/desi/edison/desiconda/20180512-1.2.5-img/aux/lib/python3.6/site-packages/tractor/_ceres.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN5ceres7ProblemD1Ev

During handling of the above exception, another exception occurred:

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-6a5831c53850> in <module>()
----> 1 from tractor import ceres

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/aux/lib/python3.6/site-packages/tractor/ceres.py in <module>()
     15         except ImportError:
     16             return importlib.import_module('_ceres')
---> 17     _ceres = swig_import_helper()
     18     del swig_import_helper
     19 elif _swig_python_version_info >= (2, 6, 0):

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/aux/lib/python3.6/site-packages/tractor/ceres.py in swig_import_helper()
     14             return importlib.import_module(mname)
     15         except ImportError:
---> 16             return importlib.import_module('_ceres')
     17     _ceres = swig_import_helper()
     18     del swig_import_helper

/global/common/software/desi/edison/desiconda/20180512-1.2.5-img/conda/lib/python3.6/importlib/__init__.py in import_module(name, package)
    124                 break
    125             level += 1
--> 126     return _bootstrap._gcd_import(name[level:], package, level)
    127 
    128 

ModuleNotFoundError: No module named '_ceres'

cori-gcc-py27 for CONFIG

Hi,
I tried to install desiconda on cori, and the command:
CONFIG=cori-gcc-py27 PREFIX=$SCRATCH/software make imaging
gives me error:
make: *** No rule to make target 'conf/cori-gcc-py27', needed by 'imaging'. Stop.

should there be some file in the conf folder called cori-gcc-py27?

"Illegal Instruction" when using 20190311-1.2.7-img software stack on edison

On edison I can't run the imaging pipeline. Here's a MWE--

cd /global/project/projectdirs/cosmo/work/legacysurvey/dr8b/runbrick-decam
export LEGACY_SURVEY_DIR=/global/project/projectdirs/cosmo/work/legacysurvey/dr8b/runbrick-decam
export LEGACYPIPE_DIR=$LEGACY_SURVEY_DIR/code/legacypipe
source $LEGACYPIPE_DIR/bin/legacypipe-env
export PYTHONPATH=$LEGACYPIPE_DIR/py:$PYTHONPATH
python $LEGACYPIPE_DIR/py/legacypipe/runbrick.py --brick 2449p065 --outdir test

For me, the output is

runbrick.py starting at 2019-03-13T05:13:49.529269
legacypipe git version: dr8-test10-158-gb89a292e
Command-line args: ['/global/project/projectdirs/cosmo/work/legacysurvey/dr8b/runbrick-decam/code/legacypipe/py/legacypipe/runbrick.py', '--brick', '2449p065', '--outdir', 'test']
Slurm cluster/job/array: none / none / none

Starting process 31253 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 416 MB, VmSize: 416 MB, VmRSS: 70 MB, VmData: 113 MB, maxrss: 0.071904 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage mask_junk
Runstage tims
Running stage tims
Running stage tims at 2019-03-13T05:13:51.175777
Illegal instruction

Restrict installation of bokeh to 2.4.

Conda now installs bokeh 3.1 by default; this is not compatible with software such as Nightwatch. We may want to specify

bokeh=2.4

in the requirements.

Weird missing symbol when compiling a python extension

Hi,

Not sure if this is desiconda related, but...

I'm investigating a bug in fitsio and I wanted to check whether the master version of fitsio still shows the problem. I checked out that code, ran "python3 setup.py install --home=$(pwd)" and set my PYTHONPATH to pick up that directory. (python3 is /global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/bin/python3), and it compiles cleanly, but then:

cd && python -c "import fitsio"
Traceback (most recent call last):
File "", line 1, in
File "/global/homes/d/dstn/fitsio/lib/python/fitsio/init.py", line 9, in
from . import fitslib
File "/global/homes/d/dstn/fitsio/lib/python/fitsio/fitslib.py", line 29, in
from . import _fitsio_wrap
ImportError: /global/homes/d/dstn/fitsio/lib/python/fitsio/_fitsio_wrap.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __intel_cpu_feature_indicator

Now,

ldd /global/homes/d/dstn/fitsio/lib/python/fitsio/_fitsio_wrap.cpython-36m-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffc837ce000)
libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00002b6308d2b000)
libm.so.6 => /lib64/libm.so.6 (0x00002b6308f9b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b6309298000)
libc.so.6 => /lib64/libc.so.6 (0x00002b63094b5000)
libidn.so.11 => /usr/lib64/libidn.so.11 (0x00002b6309856000)
libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x00002b6309a8a000)
libssl.so.1.0.0 => /global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/libssl.so.1.0.0 (0x00002b6309cb6000)
libcrypto.so.1.0.0 => /global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/libcrypto.so.1.0.0 (0x00002b6309f2c000)
libgssapi_krb5.so.2 => /global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/libgssapi_krb5.so.2 (0x00002b630a36c000)
liblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x00002b630a5b5000)
libldap-2.4.so.2 => /usr/lib64/libldap-2.4.so.2 (0x00002b630a7c4000)
libz.so.1 => /global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/libz.so.1 (0x00002b630aa10000)
/lib64/ld-linux-x86-64.so.2 (0x00002b6308728000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002b630ac27000)
libkrb5.so.3 => /global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/./libkrb5.so.3 (0x00002b630ae2b000)
libk5crypto.so.3 => /global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/./libk5crypto.so.3 (0x00002b630b0ff000)
libcom_err.so.3 => /global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/./libcom_err.so.3 (0x00002b630b31b000)
libkrb5support.so.0 => /global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/./libkrb5support.so.0 (0x00002b630b51e000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00002b630b729000)
libsasl2.so.3 => /usr/lib64/libsasl2.so.3 (0x00002b630b940000)

I think this symbol is supposed to be in some icc library?? It looks like the fitsio module is getting compiled with gcc.

make hpsspy and mpi4py optional/weak dependencies?

Upgrading desiconda on desi-7 is currently unsuccessful because of what appear to be default dependencies on hpsspy (which we definitely don't need outside of NERSC) and mpi4py.

Using the NERSC defaults is obviously part of the problem. It might be the whole problem. Ideally, we can fix this by adding a few KPNO-specific scripts to the conf folder. I will play with the configuration and see how far I can get.

Naming convention for datatran installs

Could we have future datatran installs of desiconda use the same naming convention as cori & edison? For example, the cori & edison both have 20180130-1.2.4-spec, but datatran has 20180102-1.2.2, without the -spec. This makes it more difficult to create a common desimodules module file.

Also, while you're at it, could you please install 20180130-1.2.4-spec on datatran.

Bootstrap script does not use desiutil bootstrap method

The scripts/bootstrap-desi.sh script should use the desiBootstrap.sh script to actually install desiutil so that subsequent packages can be installed with desiInstall. This will prevent desiutil from being installed as both a pip package and a desiInstalled package.

This is similar to #53 .

problem importing skimage with 20190311-1.2.7-img module

See below for the Traceback when importing skimage with 20190311-1.2.7-img. There is no issue when using 20180512-1.2.5-img.

ioannis@edison07:~% desiconda_version=20190311-1.2.7-img
ioannis@edison07:~% module use /global/common/software/desi/cori/desiconda/$desiconda_version/modulefiles
ioannis@edison07:~% module load desiconda
ioannis@edison07:~% 
ioannis@edison07:~% ipython
Python 3.6.6 |Anaconda, Inc.| (default, Oct  9 2018, 12:34:16) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import skimage                                                                                                   
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-aa248544dc2e> in <module>
----> 1 import skimage

/global/common/software/desi/cori/desiconda/20190311-1.2.7-img/conda/lib/python3.6/site-packages/skimage/__init__.py in <module>
    165         _raise_build_error(e)
    166     # All skimage root imports go here
--> 167     from .util.dtype import (img_as_float32,
    168                              img_as_float64,
    169                              img_as_float,

/global/common/software/desi/cori/desiconda/20190311-1.2.7-img/conda/lib/python3.6/site-packages/skimage/util/__init__.py in <module>
      6 from .apply_parallel import apply_parallel
      7 
----> 8 from .arraycrop import crop
      9 from ._regular_grid import regular_grid, regular_seeds
     10 from .unique import unique_rows

/global/common/software/desi/cori/desiconda/20190311-1.2.7-img/conda/lib/python3.6/site-packages/skimage/util/arraycrop.py in <module>
      6 
      7 import numpy as np
----> 8 from numpy.lib.arraypad import _validate_lengths
      9 
     10 

ImportError: cannot import name '_validate_lengths'

include tensorflow and keras as weak dependencies in DESI software

Several groups are building pipeline afterburners on top of tensorflow and keras (Ly-a and Time Domain, for example). It would be useful to have these packages available in the next release of desiconda.

bump galsim to `v1.4.4`

I think this might be the last in the 1.4 series, so is I think worth moving to. @rmjarvis can hopefully comment

Potential MKL Conflicts

In our CMB stack at NERSC, we have seen silent memory corruption, segfaults, etc in a specific scenario:

1. Numpy built with anaconda provided MKL.

2. A compiled python extension which uses a different MKL provided by Intel compilers.

Basically the two MKL versions have some incompatibility, and only one can be loaded in a process. Which version gets loaded depends on whether "import numpy" comes before or after the import of the compiled extension. The two options that we have found that work are:

1. Build compiled code with Intel compilers and MKL. For python, use conda-forge packages without MKL.

2. Build compiled code with gcc and OpenBLAS. For python use packages that have MKL.

In our CMB stack, most of our heavy computational work is in compiled code, so we went with option (1). For desispec, I think only specex is using much heavy compiled code, so option (2) would make more sense.

We should enforce only the 2 choices above when installing.

Environment variable to point to desiconda "aux" directory.

Currently the environment variable DESICONDA points to the anaconda install directory, e.g., /global/common/cori/contrib/desi/desiconda/20170613-1.1.4-spectro/code/desiconda/20170613-1.1.4-spectro_conda. However, I can't find a similar variable that points to the equivalent spectro_aux directory. This would be very useful for people needing to compile code against, say the cfitsio library.

pre-build matplotlib font cache and astropy config stuff

In desihub/desispec#494, @tskisner said:

For docker images, I do pre-build the font cache (and also the astropy config stuff). I could easily do that for regular desiconda installs too.

Yes, please do.

user envs from desiconda 20180130-1.2.4-spec are broken

Turning this email into a ticket for tracking:
I'm trying to create a user conda environment based on desiconda 20180130-1.2.4-spec to install some non-DESI packages and am getting Permission denied errors. e.g.

module use /global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/modulefiles/
module load desiconda
conda create -n test
source activate test
conda install pandas
...
PermissionError(13, 'Permission denied')

The same works if I start with NERSC python/3.6-anaconda-4.4 instead. How can we fix this so that users can start with desiconda to create their own environments to install additional packages?

Associating with desihub project 18.3 in the hopes that this can be fixed in-place before the final release in March 2018.

installation failure on perlmutter

@sbailey tried installing the (still under development and being tested) update branch of desiconda on perlmutter and it didn't work out of the box, ending with:

  checking for MPI compile and link ...
  /opt/cray/pe/craype/2.7.12/bin/cc -shared -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /global/common/software/desi/users/sjbailey/perlmutter/desiconda/20211215-2.0.0/conda/include -fPIC -O2 -isystem /global/common/software/desi/users/sjbailey/perlmutter/desiconda/20211215-2.0.0/conda/include -O3 -fPIC -pthread -fPIC -I/global/common/software/desi/users/sjbailey/perlmutter/desiconda/20211215-2.0.0/conda/include/python3.9 -c _configtest.c -o _configtest.o
  nvc-Error-Unknown switch: -Wno-unused-result
  nvc-Error-Unknown switch: -fwrapv
  failure.
  removing: _configtest.c _configtest.o
  error: Cannot compile MPI programs. Check your configuration!!!
  ----------------------------------------
  ERROR: Failed building wheel for mpi4py
Failed to build mpi4py
ERROR: Could not build wheels for mpi4py, which is required to install pyproject.toml-based projects
ERROR installing pip packages; exiting

the full log is at /global/cfs/cdirs/desi/users/sjbailey/desiconda2/perlmutter-install.log (accessible from perlmutter-p1.nersc.gov or dtn01.nersc.gov while cori is down).

Uncaught failure when attempting to install hpsspy

This issue is to document two issues, first with the way hpsspy is installed, and second with the sourced script that attempts to perform the installation.

1. In conf/pip-pkgs.sh, Line 4, hpsspy is installed with an unusual --no-binary :all: option, which is odd, since hpsspy is pure Python. The file appears to have been created by @marcelo-alvarez. Could you please explain why hpsspy is installed this way?
   • In fact --no-binary :all: doesn't work, which leaves hpsspy uninstalled. See below for error messages
   • Installing hpsspy with an ordinary install appears to work fine. See below.
2. conf/pip-pgkgs.sh only checks the status of the final pip command, so if and when the installation of hpsspy fails, conf/pip-pkgs.sh will just keep running. It only detects failure to install mpi4py. So the installation of threadpoolctl could also fail, and we wouldn't know it. set -e should be set for the entire top-level install.sh script, and tested to make sure that failures do in fact cause the install to crash.

Installation logs:

desi@login28[1027]$ pip install --dry-run --no-binary :all: hpsspy:

DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
Collecting hpsspy
  Using cached hpsspy-0.6.1.tar.gz (39 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-jbmxplzz/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-jbmxplzz/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-jbmxplzz/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 50, in <module>
        File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/importlib/__init__.py", line 126, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
        File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
      ModuleNotFoundError: No module named 'hpsspy'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

desi@login28[1028]$ pip install --dry-run hpsspy

Collecting hpsspy
  Using cached hpsspy-0.6.1-py3-none-any.whl (31 kB)
Requirement already satisfied: pytz in /global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/site-packages (from hpsspy) (2022.7)
Would install hpsspy-0.6.1

qdo crash with most recent software stack 20190311-1.2.7-img

[snip]
Starting qdo job on nid00535 at Tue Mar 12 06:37:58 PDT 2019
QDO environment variables:
QDO_DIR=/global/common/software/desi/cori/desiconda/20190311-1.2.7-img/aux/lib/python3.6/site-packages/qdo-0.8.dev1-py3.6.egg
QDO_DB_USER=desirun_admin
QDO_DB_HOST=nerscdb03.nersc.gov
QDO_SCRIPT=./dr8-runbrick-decam.sh
QDO_BATCH_PROFILE=cori
QDO_BACKEDUP_PATH=/global/project/projectdirs/cosmo/work/legacysurvey/dr8b/runbrick-decam/code/legacyzpts/bin:/global/project/project
dirs/cosmo/work/legacysurvey/dr8b/runbrick-decam/code/legacypipe/bin:/global/common/software/desi/cori/desiconda/20190311-1.2.7-img/a
ux/bin:/global/common/software/desi/cori/desiconda/20190311-1.2.7-img/conda/bin:/global/common/sw/cray/cnl6/haswell/git/2.15.1/gcc/4.
8/vfffcvo/bin:/opt/gcc/7.3.0/bin:/usr/common/software/bin:/usr/common/mss/bin:/usr/common/nsg/bin:/usr/common/software/bin:/usr/commo
n/mss/bin:/usr/common/nsg/bin:/opt/cray/pe/mpt/7.7.3/gni/bin:/opt/cray/rca/2.2.18-6.0.7.0_33.3__g2aa4f39.ari/bin:/opt/cray/alps/6.6.4
3-6.0.7.0_26.4__ga796da3.ari/sbin:/opt/cray/job/2.2.3-6.0.7.0_44.1__g6c4e934.ari/bin:/opt/cray/pe/craype/2.5.15/bin:/opt/intel/compil
ers_and_libraries_2018.1.163/linux/bin/intel64:/opt/ovis/bin:/opt/ovis/sbin:/usr/syscom/nsg/sbin:/usr/syscom/nsg/bin:/opt/cray/pe/mod
ules/3.2.10.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:/global/cs
cratch1/sd/desiproc/code/qdo/bin:/global/cscratch1/sd/desiproc/code/qdo/bin:/global/cscratch1/sd/desiproc/code/qdo/bin:/global/cscrat
ch1/sd/desiproc/code/qdo/bin
QDO_NAME=dr8b-test-hsc-ngc-decam
QDO_BACKEND=postgres
QDO_DB_NAME=desirun
QDO_DB_PASS=32j2j253ss
QDO_WORKERS_PER_JOB=256
QDO_OPTS=--timeout 60 --jitter 64.00
QDO_MPIRUN=srun -n 256 --ntasks-per-node=4 --cpus-per-task=8 --exclusive --cpu_bind=cores --threads-per-core=1
srun -n 256 --ntasks-per-node=4 --cpus-per-task=8 --exclusive --cpu_bind=cores --threads-per-core=1 /global/common/software/desi/cori
/desiconda/20190311-1.2.7-img/aux/lib/python3.6/site-packages/qdo-0.8.dev1-py3.6.egg/bin/qdo do dr8b-test-hsc-ngc-decam --timeout 60 
--jitter 64.00 --script ./dr8-runbrick-decam.sh
srun: fatal: Can not execute /global/common/software/desi/cori/desiconda/20190311-1.2.7-img/aux/lib/python3.6/site-packages/qdo-0.8.d
ev1-py3.6.egg/bin/qdo
/var/spool/slurmd/job19656433/slurm_script: line 35: 16032 Aborted                 $QDO_MPIRUN $QDO_DIR/bin/qdo do $QDO_NAME $QDO_OPT
S --script "$QDO_SCRIPT"
[snip]

avoid spike in temp files during installation

Our current desiconda installations have a ~200k spike in temporary files during an installation, which can cause inode quota problems even if the final installation would fit. Some tips from NERSC to try to avoid this:

CONDA_PKGS_DIRS=$(mktemp -d) conda install ...

They also recommend running this at the end:

conda clean --yes --all

In conf/conda-pkgs.sh we currently run rm -rf $CONDADIR/pkgs/* which is similar but not identical and would need to be updated for these.

I have not actually tested these.

python3.5 weakref exception

I encountered the following (harmless?) error running interactively on cori today:

Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x2af1c27df378>
Traceback (most recent call last):
  File "/global/common/cori/contrib/desi/desiconda/20170613-1.1.4-spectro/code/desiconda/20170613-1.1.4-spectro_conda/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable

The same code did not produce this error yesterday.

Make desiconda installs world-readable

The desiconda software needs to be world readable to allow any NERSC user to use DESI software to interact with public data. We are already doing this by hand for existing installs, but future installs should unconditionally install world-readable, since the base desiconda install only contains public, third-party packages already.

Customizing Your DESI Python Environment

I am following the instructions at https://desi.lbl.gov/trac/wiki/Pipeline/GettingStarted/NERSC#CustomizingYourDESIPythonEnvironment. I am running into an error here with regards to PackagesNotFound.

(base) akim@cori04:~> conda create -p ${HOME}/mydesi --copy --file ${DESICONDA}/pkg_list.txt
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

• openssl==1.0.2r=h14c3975_0
• sortedcontainers==2.1.0=py_0
• locket==0.2.0=py_2
• lz4-c==1.8.3=he1b5a44_1001
• msgpack-python==0.6.1=py36h6bb024c_0
• heapdict==1.0.0=py36_1000
• ca-certificates==2019.6.16=hecc5488_0

Current channels:

• https://repo.anaconda.com/pkgs/main/linux-64
• https://repo.anaconda.com/pkgs/main/noarch
• https://repo.anaconda.com/pkgs/r/linux-64
• https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

Intel compiler and boost

Compiling and linking against boost serialization produces errors like this:

../../src/libharp/.libs/libharp.so: undefined reference to `boost::serialization::array_wrapper c
onst boost::serialization::make_arrayboost::serialization::collection_size_type(float const*, boost::seriali
zation::collection_size_type)'

This was reported and fixed in the upstream boost source:

https://svn.boost.org/trac/boost/ticket/12516

but the fix did not make it into the current stable release (1.63.0). We already support a mechanism for patching boost, so should apply this fix when building boost.

update mpl_toolkits in desiconda/img

An issue arose when using the imaging desiconda build with desiutil.plots.init_sky (full traceback below) and I think it's because this build uses a fairly new version of matplotlib (2.2.2) but an older version of mpl_toolkits (which contains basemap).

The specific error is

 'AxesSubplot' object has no attribute 'set_axis_bgcolor'

which occurs because the set_axis_bgcolor method was changed to set_facecolor (see, e.g., here).

Note that I'm using the desiconda/img environment because I'm working on imaging stuff, but I want to also be able to use the great plotting tools in desiutil. This is not (yet) a problem with the spectroscopic desiconda build because we're still using version 2.1.2 of matplotlib.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-21-5549bf9d8c1b> in <module>()
----> 1 qa_binned_radec(parent)

/global/cscratch1/sd/ioannis/repos/LSLGA/LSLGA/qa.py in qa_binned_radec(cat, nside, png)
     36 
     37         warnings.simplefilter('ignore')
---> 38         basemap = init_sky(galactic_plane_color='k', ax=ax);
     39         plot_sky_binned(ra, dec, weights=weight, 
     40                         max_bin_area=hp.nside2pixarea(nside, degrees=True),

/global/cscratch1/sd/ioannis/repos/desihub/desiutil/py/desiutil/plots.py in init_sky(projection, ra_center, galactic_plane_color, ra_labels, dec_labels, ax)
    466             labels = [0, 0, 1, 0]
    467         m.drawmeridians(
--> 468             ra_labels, labels=labels, labelstyle='+/-')
    469     if dec_labels is not None:
    470         m.drawparallels(

/global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/python3.6/site-packages/mpl_toolkits/basemap/__init__.py in drawmeridians(self, meridians, color, linewidth, zorder, dashes, labels, labelstyle, fmt, xoffset, yoffset, ax, latmax, **kwargs)
   2719                         if t is not None: linecolls[lon][1].append(t)
   2720         # set axes limits to fit map region.
-> 2721         self.set_axes_limits(ax=ax)
   2722         # remove empty values from linecolls dictionary
   2723         keys = list(linecolls.keys()); vals = list(linecolls.values())

/global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/python3.6/site-packages/mpl_toolkits/basemap/__init__.py in set_axes_limits(self, ax)
   3178                 limb1 = self.drawmapboundary(fill_color='none')
   3179                 # draw another filled patch, with no boundary.
-> 3180                 limb2 = self.drawmapboundary(linewidth=0)
   3181                 self._mapboundarydrawn = limb2
   3182         # for elliptical map, always turn off axis_frame.

/global/common/software/desi/cori/desiconda/20180512-1.2.5-img/conda/lib/python3.6/site-packages/mpl_toolkits/basemap/__init__.py in drawmapboundary(self, color, linewidth, fill_color, zorder, ax)
   1621         # if fill_color is string 'none', really don't fill.
   1622         if fill_color is None:
-> 1623             fill_color = ax.get_axis_bgcolor()
   1624         elif fill_color == 'none' or fill_color == 'None':
   1625             fill_color = None

AttributeError: 'AxesSubplot' object has no attribute 'get_axis_bgcolor'

Update to new version of pip

The desiconda env includes pip (good!) but an old version:

% pip list
...
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

This issue is to update to a newer version. I don't know of any missing new features that we need, but at least it would silence this warning.

Note that version 10.0.1 is not as old as it sounds (released Apr 2018) because pip has an unusual recent version history.

Conda-installed ld incompatible with Cray system one

See discussion in desihub/fiberassign#243

Upstream anaconda tracking this in:

ContinuumIO/anaconda-issues#11152

I suggest we add a config option CONDA_LD which if set to yes will leave this alone and otherwise will move the conda-installed ld utility out of the way before proceeding.

Please install desiconda 20180518-1.2.5-spec on edison

Edison has 20180512-1.2.5-spec, but not 20180518-1.2.5-spec

fitsverify missing with desiconda/2.0.0

The fitsverify executable used to be part of a desiconda installation, but now it is missing. I suspect that it used to come along for the ride with the installation of some other product (cfitsio, fitsio, astropy, ...) but now requires a standalone installation. Maybe. Please investigate and add fitsverify to our standard desiconda installations.

If the installation doesn't have any other side effects (like changing the version of cfitsio), I'd also like to post-facto install it to /global/common/software/desi/cori/desiconda/20211217-2.0.0/conda (path corrected)

https://heasarc.gsfc.nasa.gov/docs/software/ftools/fitsverify/

Please install psycopg2 in desiconda

I need psycopg2 installed in the desiconda infrastructure.

make default matplotlib backend TkAgg instead of WebAgg

For some reason, our default matplotlib backend is WebAgg instead of something reasonable like TkAgg:

[cori02 ~] grep ^backend /global/common/software/desi/cori/desiconda/20170920-1.2.0-spec/conda/lib/python3.6/site-packages/matplotlib/mpl-data/matplotlibrc
backend      : WebAgg

The spectro pipeline resets the backend anyway, but WebAgg is a bad backend for interactive work and we should have a better default if we can figure out the recipe for doing that automatically for new installs.

In the meantime, users can override this with their ~/.config/matplotlib/matplotlibrc file, though ideally we shouldn't rely upon a recipe that requires every user to do something custom to work.

add dask-distributed

While I'm thinking of it... add dask-distributed to desiconda with our next update; we may want to use dask some day.

Ensure that lib directories are added to LD_LIBRARY_PATH

Make sure that the conda lib directory is added to LD_LIBRARY_PATH in addition to the aux lib directory. See #13.