desihub / desigal Goto Github PK

View Code? Open in Web Editor NEW

3.0 52.0 1.0 447 KB

DESI Galaxy Science measurements

License: BSD 3-Clause "New" or "Revised" License

Python 95.05% Dockerfile 4.95%

desigal's Introduction

desigal

Introduction

This repository contains code and documentation for the DESI Galaxy and Quasar Physics (GQP) Working Group. For full documentation please visit desigal on Read the Docs.

Installation

This product is installable using pip or it can cloned directly from Github. For example, to install the "0.1.0" tag (or release) do

pip install git+https://github.com/desihub/[email protected]

Alternatively, you can clone the master branch from Github

git clone [email protected]:desihub/desigal.git

and then either install the package to an installation directory of your choice

python setup.py install --prefix=$INSTALL_DIR

or explicitly add the code directory to your $PYTHONPATH environment, e.g.

export PYTHONPATH=/path/to/desigal/py:$PYTHONPATH
export PATH=/path/to/desigal/bin:${PATH}

Product Contents

bin/: Executable scripts.
doc/: High-level documentation (.rst files).
etc/: Small data and configuration files.
py/: Python code.

License

desigal is free software licensed under a 3-clause BSD-style license. For details see the LICENSE.rst file.

desigal's People

Contributors

Stargazers

Watchers

Forkers

dirkscholte

desigal's Issues

Coadd cameras does not return masks

the specutils.coadd_cameras function returns a Nonetype object

Add fastspecfit integration

Need a way to run fastspec on the stacked spectra

Add ( modelled ) weighted averaging

Error in normalize.integrate_flux

@biprateep I encountered an error in the integrate_flux function when the value at the start or the end of a flux window is masked. In that case the interpolation function bounds are shorter than the flux window and an error is returned.

I managed to reproduce the error with this simple example:

wave = np.linspace(0,10,11)
flux = np.linspace(0,10,11)
ivar = flux = np.linspace(0,10,11)
mask = np.ones_like(wave, dtype='bool')
mask[7:9] = False
print(wave[mask])
integrate_flux(wave[mask], flux[mask], ivar[mask], 3., 7.)

Returns: ValueError: A value in x_new is above the interpolation range.

I can 'fix' the issue by setting bounds_error=False, fill_value='extrapolate' in the interpolation of this function but that might be a bit risky (I guess the most extreme extrapolation issues will already be avoided as a max of 20% of the pixels can be masked for the interpolation to even happen).

Do you have any thoughts on what might be a better solution?

Improve the speed of read_spectra function

Here are some of the suggestions, courtesy of Raga Pucha and John Moustakas:

Use the read-only directory: '/dvs_ro/cfs/cdirs/desi/spectro/redux/{specprod}/healpix' instead of /global/cfs/...
You can do the same thing for fastspecfit files as well: /dvs_ro/cfs/cdirs/desi/spectro/fastspecfit/...
In the latest DESI kernel, they have improved the read_spectra() function. You can use "skip_hdus" keyword and it will run faster. Also, you can give a bunch of targets together for a given "SURVEY", "PROGRAM", "HEALPIX"
spec = read_spectra(coadd_file, skip_hdus = ('EXP_FIBERMAP', 'SCORES', 'EXTRA_CATALOG')).select(targets = targetids)
Tip from experience: "DO NOT SKIP MASK HDU". The masks are not applied when you are coadding the spectra across all cameras -- so, keep the MASK HDU for the correct spectra.
When using multiprocessing, test for 128 or 64 processes in this line: pool = Pool(processes = 128) - I have noticed that 256 usually slows things down.
You can use array jobs instead of a single job -- another way to make things run fast and get nodes assigned easier.

Iterative normalization (see code base from fastspec) https://github.com/desihub/fastspecfit/blob/main/py/fastspecfit/templates/templates.py#L437-L515
The continuum normalization (see PR from Siwei et al)

get_spectra sometimes returns duplicate spectra!

Need to investigate:

Is it problem with my loop?
Is it a problem with the db?

Make get_spectra faster!!!!!!!!!

The native implementation of read_spectra is slow even with parallelization. Use memmaps? or something else? To speed up I/O