desihub / desimodules Goto Github PK

We intend to provide public support for desimodules releases 22.5 and 23.1 and any subsequent release. For command line, preliminary results show that this technique works for 23.1:

export DESI_ROOT=/global/cfs/cdirs/desi/public/edr
source /global/common/software/desi/desi_environment.sh 23.1

Comparing the environment variables before and after (@weaverba137 used the cosmo collaboration account to test this), these path-related environment variables are defined:

TYCHO_DIR=/global/cfs/cdirs/desi/public/edr/target/tycho_dr2
DESI_SPECTRO_DATA=/global/cfs/cdirs/desi/public/edr/spectro/data
GAIA_DIR=/global/cfs/cdirs/desi/public/edr/target/gaia_dr2
DESI_SPECTRO_SIM=/global/cfs/cdirs/desi/public/edr/spectro/sim
MTL_DIR=/global/cfs/cdirs/desi/public/edr/target/surveyops/mtl
DESI_SPECTRO_CALIB=/global/cfs/cdirs/desi/public/edr/spectro/desi_spectro_calib/0.4.0
SQ_MODEL_FILE=/global/cfs/cdirs/desi/public/edr/target/catalogs/lya/sq_models/BOSS_train_64plates_model.json
TOO_DIR=/global/cfs/cdirs/desi/public/edr/target/ToO
DESI_SPECTRO_REDUX=/global/cfs/cdirs/desi/public/edr/spectro/redux
DESI_SPECTRO_DARK=/global/cfs/cdirs/desi/public/edr/spectro/desi_spectro_dark/v2209
SCND_DIR=/global/cfs/cdirs/desi/public/edr/target/secondary
DESI_ROOT=/global/cfs/cdirs/desi/public/edr
SKYBRICKS_DIR=/global/cfs/cdirs/desi/public/edr/target/skybricks/v3
DESI_SURVEYOPS=/global/cfs/cdirs/desi/public/edr/survey/ops/surveyops/trunk
DESI_TARGET=/global/cfs/cdirs/desi/public/edr/target
ZCAT_DIR=/global/cfs/cdirs/desi/public/edr/spectro/redux/daily
CMX_DIR=/global/cfs/cdirs/desi/public/edr/target/cmx_files
URAT_DIR=/global/cfs/cdirs/desi/public/edr/target/urat_dr1
QN_MODEL_FILE=/global/cfs/cdirs/desi/public/edr/target/catalogs/lya/qn_models/qn_train_coadd_indtrain_0_0_boss10.h5
DESI_MOCKS=/global/cfs/cdirs/desi/public/edr/mocks
TARG_DIR=/global/cfs/cdirs/desi/public/edr/target/catalogs
MASK_DIR=/global/cfs/cdirs/desi/public/edr/target/masks
DESI_WWW=/global/cfs/cdirs/desi/public/edr/www
DUST_DIR=/global/cfs/cdirs/cosmo/data/dust/v0_1
DESI_BASIS_TEMPLATES=/global/cfs/cdirs/desi/public/edr/spectro/templates/basis_templates/v3.2

The following environment variables are potentially problematic:

• Defined by desimodules:
  • DESI_SURVEYOPS does not exist (and points to trunk).
    • Note that desitree also defines DESI_SURVEYOPS, which is then overwritten by desimodules, and that value happens to be the same, but could get out of sync.
  • DESI_SPECTRO_CALIB does not exist (but 0.3.1 is there).
    • Note that desispec also defines DESI_SPECTRO_CALIB, which is then overwritten by desimodules. The values are different.
• Defined by desitree:
  • DESI_MOCKS does does not exist.
  • DESI_SPECTRO_SIM does not exist.
  • DESI_WWW does not exist (but we don't really use this anyway).
• defined by desitarget:
  • TYCHO_DIR does not exist.
  • GAIA_DIR does not exist.
  • MTL_DIR does not exist.
  • TOO_DIR does not exist.
  • ZCAT_DIR does not exist (no daily in EDR).
  • SQ_MODEL_FILE does not exist.
  • QN_MODEL_FILE does not exist.
  • CMX_DIR does not exist.
  • URAT_DIR does not exist.
  • MASK_DIR does not exist.
• defined by desispec:
  • DESI_SPECTRO_DARK does not exist (but maybe is planned?).
  • QN_MODEL_FILE and SQ_MODEL_FILE are also defined by desispec and just happen to have the same values as desitarget.
• defined by desisim:
  • DESI_BASIS_TEMPLATES does not exist, but will. Is v3.2 the fuji version?

Consider completely refactoring desimodules so that it just contains the files in

/global/common/edison/contrib/desi/desiconda/startup/modulefiles/desimodules/

rather than having desimodules be a product that itself is installed. i.e. the above directory would become a git clone of desimodules, and when a new release is added, that file is just added to the git repo and committed. This would require some updates to the module files to use $NERSC_HOST instead of hardcoding cori/edison. There might be additional subtleties for datatran and scigate that I haven't realized.

In practice we currently directly edit the files in that directory and then after everything works we retroactively update desimodules/etc/desimodules.modules to match and make a desimodules tag. The fact that desimodules.modules itself is not a valid module file makes testing harder without lots of commit and install cycles, or otherwise risking the fragility of copy-and-paste editing.

Currently it is not possible to obtain the desimodules version from the install directory, which is stored in the $DESIMODULES environment variable: DESIMODULES=/global/common/software/desi/cori/desiconda/startup/modulefiles/desimodules.

A simple solution would be for the desimodules module file to set $DESIMODULES_VERSION.

Although I updated the $DESI_BASIS_TEMPLATES environment variable in desisim/etc/desisim.module last week (see desihub/desisim@0aeddd1#diff-7bf035b8060460e6209f61153102a108), loading the default DESI software on NERSC still points to the v2.5 templates.

ioannis@edison02:~% source /project/projectdirs/desi/software/desi_environment.sh master
ioannis@edison02:~% echo $DESI_BASIS_TEMPLATES
/global/project/projectdirs/desi/spectro/templates/basis_templates/v2.5

Am I missing a step?

The desi-conda Module files overwrite any existing PYTHONPATH definitions. Paths should always set with prepend-path, not setenv.

The main configuration uses desiconda/20220119-2.0.1 on perlmutter, while 23.1 uses desiconda/20230111-2.1.0. This is inconvenient because new packages have to be desiInstalled twice, and it's easy to forget which version the package got installed it. This could lead to more serious problems later.

The desi_environment.c?sh scripts still make reference to scigate, but it is kind of unlikely that they would work. Should we just remove all references to scigate and just have an error message if someone tries to use it?

Another option would be to install a dummy desimodules file /global/common/software/desi/scigate.

I think this issue belongs here, but if not please close and let me know where to re-post.

For various activities, including the SV Data Challenge, desihub/desispec#513, and desihub/desitarget#284 it would be helpful to have an environment variable (e.g., $DUST_DIR) which points to the location of the SFD dust maps.

I just noticed that the following command fails:

dkirkby@cori09:~> source /project/projectdirs/desi/software/desi_environment.sh 18.1
gcc(3):ERROR:105: Unable to locate a modulefile for 'gcc/6.2.0'
...

All numbered releases (17.12, 17.9, ...) fail the same way, but the default (master) works.

Is this a transient problem or are others seeing the same thing?

As mentioned in #36, we can considerably streamline the structure of the desi_environment.(sh|csh) files by dropping the 17.x and 18.x series of modules.

• Should we tag desimodules for everest version 21.7e?
• The .version file still has 21.3; should that also be bumped?
• It looks like this package is ready for the master branch to be renamed, are we ready to do that?

If @sbailey does not know the answer, I'll ask Rollin.

Is there any reason not to add the desi/software/jupyter/kernels/ files to desimodules?

For @sbailey and @tskisner, we need to coordinate on updating desimodules.

• The tags from the summer 2016 pipeline development are now well in advance of those loaded by desimodules.
• Recent NERSC upgrades mean that basically HPCPorts has to be rebuilt on every system.

I've configured the desi environment using:

source /project/projectdirs/desi/software/desi_environment.sh

But when I try to import galsim in ipython I get the following error:

In [2]: import galsim
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-645d464f8d43> in <module>()
----> 1 import galsim

/global/common/cori/contrib/desi/desiconda/20170613-1.1.4-spectro/code/desiconda/20170613-1.1.4-spectro_aux/lib/python3.5/site-packages/galsim/__init__.py in <module>()
     92 
     93 # First some basic building blocks that don't usually depend on anything else
---> 94 from .position import PositionI, PositionD
     95 from .bounds import BoundsI, BoundsD
     96 from .shear import Shear

/global/common/cori/contrib/desi/desiconda/20170613-1.1.4-spectro/code/desiconda/20170613-1.1.4-spectro_aux/lib/python3.5/site-packages/galsim/position.py in <module>()
     20 """
     21 
---> 22 from . import _galsim
     23 from ._galsim import PositionD, PositionI
     24 

ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory

This issue is opened to track a TO-DO item from an email thread on 2016-10-04 between Stephen, Ben, and myself. The DESI conda software stack needs to have 2 install prefixes, one for the conda root environment and one for "extra" software built in a way that is compatible with that root environment. Here are the proposed changes:

1. Have a single modulefile, called "desiconda/" that would fit the defacto "desi" naming convention. This module file would set variables associated with both corresponding install prefixes (since they really should always be used together- the "extra" packages are built using the particular conda root). So the single modulefile would be:

/desi/modulefiles/desiconda/

2. Install software to the matched pair of directories:

/desi/code/desiconda/
/desi/code/desiconda/_extra

Note that the desiconda module will continue to export the DESI_CONDA environment variable pointing to the install prefix of the conda root.

The existing helper scripts for loading DESI kernels in the NERSC Jupyter environment assume that the module command is already available in the environment. However on perlmutter, that is not the case, and the module command must be loaded explicitly.

Here is a hypothetical example which combines a couple of existing scripts:

#!/bin/bash
connection_file=$1
[[ $(declare -F module) ]] || source /usr/share/lmod/lmod/init/bash
module load python
exec python -m ipykernel_launcher -f $connection_file

Note also that the module is now ipykernel_launcher instead of ipykernel.

The equivalent test for the presence of module on tcsh should be:

set _a = `alias module`
if ( "${_a}" == "" ) source /usr/share/lmod/lmod/init/tcsh