DESI simulations
License: BSD 3-Clause "New" or "Revised" License
This package contains scripts and packages for simulating DESI spectra. For full documentation, please visit desisim on Read the Docs
desisim is free software licensed under a 3-clause BSD-style license. For details see
the LICENSE.rst file.
Refactor newexp-desi to optionally take fiberassign output + a truth file to generate the targets for the exposure of that tile.
Background: newexp-desi currently works in isolation, generating a new exposure with the right target densities, but it doesn't have any connection to fiber assignment or other exposures of overlapping tiles. The main difficulty is simulating the same target on different exposures on different tiles if it gets re-observed: newexp-desi needs some way of handing off the truth spectrum from one exposure to another.
Suggestion: Keep a cache of templates for every target on each brick. When simulating a new tile, look up all the bricks covered by that tile. If a template cache for a brick already exists, use that. If not, generate a template for every target on that brick (whether or not it was observed by the current tile) and add that to the cache.
newexp-desi outputs simspec files to $DESI_SPECTRO_SIM/$PIXPROD/{night}/simspec-{expid}.fits. I suggest the brick cache should be at $DESI_SPECTRO_SIM/$PIXPROD/bricks/{brickname}-templates.fits.
Template generation currently doesn't let us specify the redshift distribution (see #104), which we need for this feature to be fully functional. But even if we ignore the redshifts from the input truth, it would be useful to get the rest of this infrastructure going and have the ability to simulate multiple observations of the same target based upon fiber assignment output.
Example fiber assignment files at NERSC in /project/projectdirs/desi/target/fiberassign/examples/0.2.3:
The new options would be something like
newexp-desi --fiberassign tile_28598.fits --truth truth-lite.fits [other options...]
Move filters into desimodel so that desispec doesn't have to depend upon desisim. Add/move io code to desimodel to read those filters. If needed, standardize filters into a common format for different telescopes.
For discussion: should basic filter operations (integrate over a filter, calculate a magnitude, ...) also be in desimodel along with the filters, or in desispec? Having those calculations in desimodel feels a bit like feature creep and a non-intuitive place to look for them, but having them in desispec separated from the filters also seems a bit odd. The motivation for putting them in desimodel instead of desispec is because some non-spectral code may want to use them, e.g. mock making (paraphrased comment from Peder Norberg, March 2015). But the association with code for using the filters may be the counter argument.
Discuss.
The DESI platescale is a function of focal plane radius (see $DESIMODEL/data/focalplane/platescale.txt), and thus fibers have a different angular size on the sky depending upon their radius on the focal plane.
Model that.
This affects both the amount of object light that makes it into the fiber and the amount of sky background that makes it into the fiber.
Downstream pipeline steps will need to model this and remove the effective fiber input throughput differences as part of the sky subtraction, flux calibration, fiber flat fielding, etc.
The elg, lrg, star, and wd basis templates in /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/ aren't FITS-standard compliant (the QSO templates are). In particular, HDU2 has an extraneous SIMPLE=T second keyword, when BITPIX is supposed to be the second keyword. These files work with astropy and mrdfits, but cause the more picky cfitsio to fail (and thus fitsio too).
Current code works because astropy.io.fits is our standard fits reader, but the next iteration on templates should fix this as well.
These scripts are currently not tested by travis or included in coverage reports.
I believe the main obstacle is that any test would need to read the following files that currently live in the desimodel svn package:
Loaded 65000 rows from .../desimodel/data/spectra/spec-sky.dat with args {'format': 'ascii'}
Loaded 65000 rows from .../desimodel/data/spectra/spec-sky-bright.dat with args {'format': 'ascii'}
Loaded 65000 rows from .../desimodel/data/spectra/spec-sky-grey.dat with args {'format': 'ascii'}
Loaded 65000 rows from .../desimodel/data/spectra/ZenithExtinction-KPNO.dat with args {'format': 'ascii'}
Loaded 667 rows from .../desimodel/data/sky/solarspec.txt with args {'format': 'ascii.basic'}
Loaded 4233 rows from .../desimodel/data/specpsf/psf-quicksim.fits with args {'hdu': 'QUICKSIM-R'}
Loaded 22411 rows from .../desimodel/data/throughput/thru-r.fits with args {'hdu': 'THROUGHPUT'}
Loaded 4761 rows from .../desimodel/data/specpsf/psf-quicksim.fits with args {'hdu': 'QUICKSIM-B'}
Loaded 24651 rows from .../desimodel/data/throughput/thru-b.fits with args {'hdu': 'THROUGHPUT'}
Loaded 4799 rows from .../desimodel/data/specpsf/psf-quicksim.fits with args {'hdu': 'QUICKSIM-Z'}
Loaded 25531 rows from .../desimodel/data/throughput/thru-z.fits with args {'hdu': 'THROUGHPUT'}
Loaded 27 rows from .../desimodel/data/throughput/fiberloss-perfect.dat with args {'format': 'ascii'}
Loaded 27 rows from .../desimodel/data/throughput/fiberloss-star.dat with args {'format': 'ascii'}
Loaded 27 rows from .../desimodel/data/throughput/fiberloss-lrg.dat with args {'format': 'ascii'}
Loaded 27 rows from .../desimodel/data/throughput/fiberloss-qso.dat with args {'format': 'ascii'}
Loaded 27 rows from .../desimodel/data/throughput/fiberloss-sky.dat with args {'format': 'ascii'}
Loaded 27 rows from .../desimodel/data/throughput/fiberloss-elg.dat with args {'format': 'ascii'}
Loaded 65000 rows from .../desimodel/data/spectra/spec-ABmag22.0.dat with args {'format': 'ascii'}
(The list above was produced by running quickspecsim -c desi --verbose.)
Another problem is that desisim does not have a formal dependency on specsim that I can see. My guess is that we need to updated requirements.txt and etc/travis_env_common.sh to fix this.
Finally, new unit tests need to be created that invoke quickgen and quickbrick. I am not sure how to do this for standalone scripts under bin/, but it would be straightforward if we adopt the entrypoints mechanism recommended here.
Thanks to the redshift data challenge, we now have a first version of quickbrick with validated outputs. This is a good time to benchmark the performance for different template classes and see where we stand:
Quickgen's use of specsim is quite convoluted and exposes a lot of specsim implementation details because specsim does not provide convenient access to the information it needs (see for example desihub/specsim#17 and desihub/specsim#18). This issue is to review this interaction and make changes to specsim and quickgen to reduce the coupling and generally make quickgen more maintainable.
Our usage of flavor and objtypes is getting a little out of hand in desispec. ELG, LRG, QSO seemed straightforward enough, but we are starting to get things like MWS vs. MWS_STAR, STD vs. STD_FSTAR vs. FSTD, etc. This issue is a placeholder to think about the big picture of how this should work and do some cleanup.
When running quickgen on a flavor=flat simspec exposure, it outputs a fiberflat-{brz}0-{expid}.fits file with 5000 spectra in the even if the --nspec option was for fewer spectra. It does use the --nstart option to figure out which spectrograph to start with, but it always rights 5000 spectra there.
e.g. quickgen --simspec {simfile} --fibermap {fiberfile} --nspec 1000 --nstart 2000 should write fiberflat-{brz}1*.fits and fiberflat-{brz}2*.fits, each with 500 spectra. Currently it write only fiberflat-{brz}1*.fits with 5000 spectra.
The equivalent options do work correctly for flavor=science exposures; this problem is only for the fiber flat output.
Feature request from @sbailey:
Contaminant templates: non-XYZ objects that pass the cuts for XYZ. QSOs are probably the most important one to do first.
Creating an issue to track a discussion I just had with @sbailey.
Some tests in the desisim test suite require desimodel, both code and data. This is a serious problem for Travis tests because:
Some solutions that have been discussed so far:
And an additional idea:
The class constructor reads the basis templates from disk but also takes the number of templates to generate as a parameter. This means that the obvious implementation of a loop that generates a batch of QSOs in each iteration ends up re-reading the basis templates each time:
for i in xrange(1000):
qso = QSO(model=100) # reads qso_templates_v1.1.fits
outflux, wave, meta = qso.make_templates()
I suspect the fix is as simple as moving the nmodel parameter from the ctor to the make_templates method, leading to:
qso = QSO() # reads qso_templates_v1.1.fits
for i in xrange(1000):
outflux, wave, meta = qso.make_templates(nmodel=100)
Btw, the make_templates docstring does not describe the wave element of the returned tuple.
Work with Or Graur and the Time-Domain WG to add supernovae spectra (starting with Type Ia's) to galaxy (BGS, LRG, ELG) and---eventually---QSO spectra.
@sbailey newexp-desi is outputing photons in 10^-17 units. Earlier the true flux were also in the units of "10^-17 erg/s/cm2/A' that would go as input also in quicksim. Now both flux and photons are ~ 10^-17 ergs/s/cm2/A. Looks like a unit bug in desimodel.io.throughput ?
In [2]: import astropy.io.fits as fits
In [3]: sim=fits.open('simspec-00000002.fits')
In [4]: sim.info()
Filename: simspec-00000002.fits
No. Name Type Cards Dimensions Format
0 PRIMARY PrimaryHDU 13 ()
1 WAVE ImageHDU 9 (31901,) float64
2 FLUX ImageHDU 9 (31901, 10) float32
3 SKYFLUX ImageHDU 8 (31901,) float32
4 WAVE_B ImageHDU 9 (12326,) float64
5 PHOT_B ImageHDU 8 (12326, 10) float32
6 SKYPHOT_B ImageHDU 7 (12326,) float32
7 WAVE_R ImageHDU 9 (11205,) float64
8 PHOT_R ImageHDU 8 (11205, 10) float32
9 SKYPHOT_R ImageHDU 7 (11205,) float32
10 WAVE_Z ImageHDU 9 (12765,) float64
11 PHOT_Z ImageHDU 8 (12765, 10) float32
12 SKYPHOT_Z ImageHDU 7 (12765,) float32
13 METADATA BinTableHDU 20 10R x 5C [10A, J, E, E, E]
In [5]: sim[2].data[0]
Out[5]:
array([ 3.15100754e-20, 3.10153406e-20, 3.03663754e-20, ...,
2.22941950e-17, 2.20956452e-17, 4.23840662e-17], dtype=float32)
In [6]: sim[5].data[0]
Out[6]:
array([ 1.26388593e-20, 1.24530337e-20, 1.22048011e-20, ...,
1.61152245e-19, 1.33352876e-19, 1.40780001e-19], dtype=float32)
The current set of LRG templates are just >1 Gyr-old simple stellar populations (SSPs) based on theoretical calculations by C. Conroy & collaborators. The current plan for hewing these templates more closely to data is to stack SEQUELS/LRG spectra and then to fit non-negative linear combinations of Charlie's SSPs to those spectra.
We currently treat all fibers as being perfectly aligned such that the fiber aperture loss is identical for all objects of a given time. In practice there will be mis-alignments that will generate different amounts of losses, and thus add noise to our standard star calibration solutions and affect the amount of light received for science targets.
Find a reasonable way to parameterize this and include it in the simulations.
Note: this is somewhat related to, but different than these other issues that cause different amounts of light to make it down the fibers:
I am in the middle of some substantial changes to specsim that move all of the hardcoded desimodel dependencies into a yaml configuration file, with some unavoidable API changes. These changes lay the groundwork for simulating other instruments (eBOSS, for example) and also running Travis tests without desimodel.
This issue is to update desisim's usage of specsim when v0.3 is released with the new API. I believe that the only affected code is bin/quickgen and bin/quickbrick. A version requirement specsim >= 0.3 will also need to be imposed, but I'm not sure how to do that (it looks like specsim is not mentioned at all in the current requirements.txt).
Please note the two warnings generated when attempting to generate brick files. These did not prevent the completion of quickbrick, but they are definitely a couse for concern. The first warning is about an invalid value, the second is about an inconsistent dimension size in the star templates file.
quickbrick -b valid-50 -n 50 --objtype=mix --wavestep=0.2 --downsampling=5 --seed=XXXXX --outdir_truth=.
INFO:io.py:466:read_basis_templates: Reading /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/star_templates_v1.2.fits
/project/projectdirs/desi/software/cori/desisim/0.8.1/lib/python2.7/site-packages/desisim-0.8.1-py2.7.egg/desisim/filterfunc.py:68: RuntimeWarning: invalid value encountered in double_scalars
flux /= np.trapz(self.interp(wave1)*wave1,wave1)
INFO:io.py:466:read_basis_templates: Reading /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/lrg_templates_v1.2.fits
INFO:io.py:466:read_basis_templates: Reading /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/qso_templates_v1.1.fits
INFO:io.py:466:read_basis_templates: Reading /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/elg_templates_v1.5.fits
Simulating ELG template 10/30
Simulating ELG template 20/30
Simulating ELG template 30/30
INFO:io.py:466:read_basis_templates: Reading /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/star_templates_v1.2.fits
/global/cscratch1/sd/kisner/software/hpcports_gnu/astropy-1.0.1_f4385b62-1.0/lib/python2.7/site-packages/astropy-1.0.1-py2.7-linux-x86_64.egg/astropy/io/fits/fitsrec.py:774: UserWarning: TDIM7 value (10,5) does not fit with the size of the array items (10). TDIM7 will be ignored.
actual_nitems, indx + 1))
This has been started in desisim/doc/tex/simulate-templates but needs to be finished.
I noticed two potential problems with how random numbers are generated in the ELG template code, both in templates.ELG.make_templates() [line numbers refer to 497e2e9]:
rand.normal(0.0,0.3) with rand.normal(0.0, 0.3, nchunk).EMSpectrum.spectrum() used to add emission lines to each continuum template always pass the same seed [line 274], so it looks like all calls to np.random.* will return the same values for each simulated emission spectrum.Unfortunately, a test of reproducibility with the same seed does not catch either of these issues since they both lead to spectra that are too reproducible!
In the py/ directory, there are two packages, desisim, and qso_template. The latter hasn't been touched in some time. Is it still necessary to keep it separate, or can it be folded into desisim?
An effort needs to be made to join the current set of QSO templates in desisim with the templates (which extend to intrinsically fainter QSOs) developed by the Lyman-alpha working group. Their work is documented here.
In addition (and in particular) the next set of templates need to include some information about the near-infrared (WISE) flux, so that the appropriate target selection criteria can be applied (even if the near-infrared flux or optical-near-IR color is assigned statistically).
The template generator for STARs need to be a little more flexible to accept more flavors of STAR targets. For example, right now the input gmag distribution is used as the input WD mag distribution and the input rmag for the all other star templates. Can all the target mag distributions be taken from targets.dat in desimodel? Also, can we make the input flavor of star (WD, FSTD, MWS) and argument rather than individual true/false flags? Right now the only way to specify sub-types is WD=True or FSTD=True. For example, I can imagine wanting to have multiple flavors of STAR_MWS so I can get different target cuts and even now the MWS wants a different range than the FSTDs, and I expect the BAD_QSO stars want yet a different mag distribution.
Related, the same objtype used for magnitude limits and TS cuts is passed to specter/throughput.py and used to compute the fiber losses. It would be nice to be able to specify just "star" to specter, since it doesn't need more than that. Right now specter uses the "default input loss" and just generates a warning, but here as well it would be good to distinguish target type from object type.
I'm happy to make these changes, but need input on where some of the default data (like color cuts) should go and I don't want to give grief to all uses of FSTD=True code without warning.
This could be a bit tricky to implement in a meaningfully realistic way, but the basic idea is that the sky won't be perfectly uniform in (x,y) and we should simulate that so that the pipeline has to deal with a non-uniform sky.
While analyzing the packages that desisim depends on, I found a few anomalies that I need to ask about.
xastropy?We propose "quickbrick" similar to what the Lyman-alpha forest group did for QSOs at the Argonne 2015 workshop: go directly from templates to brick files, skipping over the intermediate steps. This is similar to what could be accomplished with newexp-desi + quickgen + desi_make_bricks, but streamlines the process. Its primary use would be outputting brick files for redshift fitting and spectrum-based analysis development. This would enable a quick trick script stack to generate a quick trick brick set. :)
Irritating detail: the brick files are still exposure based; if an object has more than one exposure it gets more than one entry per file, but the final output redshifts only have one row per object. Should the truez file be row-matched to the brick files (and thus have possibly repeated entries) or should it be only one row per object? Should we define that future truez and zbest files put their objects in TARGETID sorted-order so that they can be row-matched?
Notes:
It would use the desisim.templates.*.make_templates() infrastructure to generate template realizations based upon the files in $DESI_BASIS_TEMPLATES. New/updated templates should come in via that infrastructure.
It would use specsim to model the throughput, resolution, and noise, which will use the parameters in desimodel.
Usage: quickbrick [options]
Options:
-h, --help show this help message and exit
-b BRICKNAME, --brickname=BRICKNAME
brickname
-f FLAVOR, --flavor=FLAVOR
comma separated list of elg,lrg,qso,star,sky,bgs
-n NSPEC, --nspec=NSPEC
number of spectra to simulate
-o OUTDIR, --outdir=OUTDIR
output data directory; default .
--outdir_truth=OUTDIR_TRUTH
optional alternative outdir for truth files
When generating synthetic spectral templates, the redshifts are drawn from a uniform distribution between minimum and maximum values defined by the user (with sensible default ranges set by the spectral class). #104 is one alternative.
Another option is to draw redshifts from the expected redshift distribution n(z) for each target class. (Note that desisim.targets.sample_nz already does this---it's just not yet implemented.)
Similarly, the spectra are normalized to an apparent magnitude distribution (in a band which depends on the target class) which is also (unrealistically) uniformly distributed over a user-defined range. We should have the flexibility of drawing from an observationally constrained luminosity function, and then asking whether the object passes the faint-magnitude limit by computing the distance modulus and K-correction.
Note that code to draw from the QSO luminosity function has already been implemented here.
Incorporate an updated/improved set of white dwarf spectral models computed by Boris Gaensicke. Quoting from an email from Boris dated 2016 Feb 18:
I have a hydrogen atmosphere (DA) grid computed with the code from
Koester (2010MmSAI..81..921K) that has a finer sampling (0.25 in log g),
spans a wider range in Teff (6000-90000K), and samples a wider
wavelength range (900A - 30micron). This is essentially an up-to-date version
of the grid that has been used by various groups to analyse SDSS
white dwarfs, and it makes use of the new Stark profiles for the Balmer
lines (Trembley at al. 2009ApJ...696.1755T).
Now that quickgen has been announced to the collaboration, bugs in it become high priority.
The options for simulating spectra above 499 appear to be broken. All of these should work:
quickgen --simspec {simspecfile...} --fibermap {fibermapfile} --nspec 500 --nstart 500 --spectrograph 1
quickgen --simspec {simspecfile...} --fibermap {fibermapfile} --nspec 500 --nstart 500
quickgen --simspec {simspecfile...} --fibermap {fibermapfile} --nspec 1000
quickgen --simspec {simspecfile...} --fibermap {fibermapfile} --nspec 5000
But they all fail with some version of:
Simulating spectrum 500, object type=ELG
Traceback (most recent call last):
File "/project/projectdirs/desi/software/edison/desisim/master/bin/quickgen", line 257, in <module>
nobj[j,i,:waveMaxbin[channel]]=results.nobj[waveRange[channel],i]
IndexError: index 500 is out of bounds for axis 0 with size 500
As a result, quickgen can only simulate spectrograph 0 and not 1-9. Please fix.
0.7 has been tagged but the __version__ string is still 0.6.dev1.
The simulated spectra written out by newexp-desi and quickgen are double, rather than floating-point precision. Is this intentional?
[From @sbailey ]
Christophe, Julien, and I are working on setting up some test data for a redshift fitting data challenge. It seems that the [OII] flux in the new templates is too bright — when I integrate the [OII] doublet I get about a factor of 2.7x brighter than what is reported in the metadata table.
The distribution in the metadata table looks reasonable for DESI, while the templates themselves have whopping [OII] doublets that make it really really easy for the redshift fitters. I suspect that the universe is not that kind.
import numpy as np
from desisim.templates import ELG
wave = np.arange(5000, 9800.1, 0.2)
flux, _, meta = ELG(nmodel=10, wave=wave).make_templates()
for i in range(len(meta)):
z = meta['REDSHIFT'][i]
ii = ((3727-2)*(1+z) < wave) & (wave < (3730+2)*(1+z))
OIIflux = np.sum(flux[i,ii]*np.gradient(wave[ii]))
print i, z, meta['OIIFLUX'][i], OIIflux
Once #71 is done to enable newexp-desi --flavor bright, generate a canonical set of bright time simulations and advertise those for use in future pipeline development (e.g. data challenges).
While we're at it, do the same for an updated set of dark time exposures too.
related to desispec issue 51, the following keywords are missing from pixsim output pix*.fits files: exposure.telra, exposure.teldec, exposure.tileid
There will be sky spectrum residuals in the target spectra because the fiber flat field will be inaccurate.
This will be due to non uniformities of the dome screen intensity because of the finite amount of calibration lamp used, and their potential relative variation of intensity.
Adding a random fraction of the sky spectrum per fiber (with an optional parameter) in quickbrick (and other simulation tools) would help investigate how this could be handled by redshift fitters (bright sky line mask, simultaneous fit of sky residuals with target model ...)
Requested feature from @sbailey:
Ability to pass in a redshift array and get back a set of templates at those redshifts. This is needed for generating simulated data to match an input mock catalog.
pixsim-desi --cosmics cosmics.fits adds cosmics to a simulation using a library of BOSS dark images, e.g. those in /project/projectdirs/desi/spectro/templates/cosmics/v0.2/ at NERSC. It propagates the mask from those files, which is overly clean since there was no signal to confuse the cosmics identification algorithm.
Update pixsim-desi to use desispec.cosmics.reject_cosmic_rays(image) to re-mask the cosmics after adding them to the signal, so that the output simulated mask is realistic given the current state of our cosmics masking algorithms.
quickbrick is barfing because desimodel/data/desi.yaml specifies exptime_dark and exptime_bright whereas quickbrick only expects exptime:
https://github.com/desihub/desisim/blob/master/bin/quickbrick#L59
I can fix this by adding a bright vs dark optional input to quickbrick, which would also be an opportunity to include default bright-sky conditions to the simulated output spectra.
I've begun porting the code I previously wrote in IDL for building spectral templates (of all classes except QSOs) to python in desisim. I'm starting with the ELGs as the "hardest" class because I want there to be flexibility in how the emission-line spectrum is constructed, without losing the (empirical) relationship between the emission-line spectrum (especially [OII]) and the underlying stellar continuum. Eventually this code (or an intermediate piece of code) will need to interface with desisim.targets() for generating mock datasets.
This will involve changes to some code and the desisim dependencies.
Once #70 (add BGS-like templates) is done, add newexp-desi --flavor bgs and bright options.
--flavor bgs would simulate just Bright Galaxy Survey targets, while bright would do a mix of BGS and MWS targets based upon the target densities in $DESIMODEL/data/targets/targets.dat .
Use exptime_bright from desimodel.io.load_params() for exposure time for flavors mws, bgs, and bright.
When running quickbrick to simulate ELG spectra, it appears that for some seed values, make_templates is getting trapped in an infinite loop:
quickbrick -b elg-100 --objtype=elg -n 100 --wavestep=0.2 --downsampling=5 --seed=36278
INFO:io.py:466:read_basis_templates: Reading /Users/balland/software/DESI/templates/elg_templates_v1.5.fits
(nobj gflux rflux zflux)
0 0.389182203653 0.889768319563 2.82101159881
0 0.34244854018 0.520201495332 1.83512049751
0 3.2059061274 3.65286151342 8.91696554622
1 0.596939871811 0.690797688129 1.07839839904
2 0.730462842891 1.22903017783 2.84984074649
2 0.44975232066 0.488796456556 0.945432319503
3 1.25288209136 1.54069514849 2.29774914482
4 1.20394097408 1.7500975788 3.30527697546
5 1.9869155923 2.19026235619 5.25214423389
6 3.46967750653 3.53729113706 5.63974807984
7 0.500691078679 0.938085739886 4.22415601743
7 0.434556737135 0.548816630125 1.26927907769
8 0.734986473212 1.05794442062 3.1308010843
8 0.736668451755 1.37582535989 3.78068489899
8 0.719109088305 0.695289532948 1.29016218049
9 2.66904210178 3.20630371357 9.18626840421
9 1.80427167496 2.53900820452 7.9647101176
9 1.68421749473 3.10725360778 12.0505634408
9 0.753069301385 1.72120610228 4.16594234824
9 0.735655517961 0.737320745965 2.26880454757
9 0.549928377833 0.935599046027 2.83142791768
9 1.88236666985 2.45955548667 10.6525302436
9 1.61184959441 2.16935519679 12.4670519826
9 1.03290594521 1.34613820197 8.10347802548
9 1.23061306005 2.92369667412 19.2532084199
9 1.21513103363 1.58595420827 30.2256374531
9 1.49554906193 2.45919176657 78.9215522665
...
9 0.448388329099 0.661248172651 4.73202638011e+28
9 9.77195130461e+23 1.93529094339 2.7037030858e+29
9 0.566638308873 0.655562775011 9.51792409584e+28
9 0.783621573571 1.59963579161 3.65527790031e+29
9 2.96939642325e+24 1.71422942333 8.28233438182e+29
9 0.640745134036 0.919955489612 2.73154386466e+29
9 1.12847911628 1.56528651667 1.62688762064e+30
9 0.285305984927 0.548127254121 1.30263573083e+29
...
zflux is keeping on increasing (on average), the color cuts are never met, so that grzmask stays to ‘False' , nobj is never incremented, and make_templates never exits the 'while nobj<=(nmodel-1):’ loop.
The problem seems to be related to the fact that the seed given to quickbrick is passed to EMSpectrum.spectrum() from make_templates(). If a different seed is passed to EMSpectrum.spectrum(), (or no seed is passed), the problem disappears.
It's difficult to run pixel level simulations on a laptop, because there are too many environment variables to set, some hard-coded path, and some inconsistencies in path , and too many python packages to have this running.
desiutil,desispec,specter,specsim,desisim,desimodel(git),desimodel(svn)
export PIXPROD=0
export PRODNAME=0
export DESI_SPECTRO_SIM=/home/guy/Projets/DESI/analysis/pixelsims
export DESI_ROOT=/home/guy/Projets/DESI/analysis/pixelsims
export DESI_SPECTRO_DATA=/home/guy/Projets/DESI/analysis/pixelsims
There is a semi hard-coded path in desisim/obs.py
infile = os.getenv('DESI_ROOT')+'/spectro/templates/calib/v0.2/arc-lines-average.fits
After successfully running
newexp-desi --flavor=arc --expid=0 --nspec=25
newexp-desi --flavor=flat --expid=1 --nspec=25
I got another path problem when running
pixsim-desi --night=20160126 --expid=0 --cameras=b0 --nspec=25 --ncpu=12
pixsim-desi --night=20160126 --expid=1 --cameras=b0 --nspec=25 --ncpu=12
in desispec/io/fibermap.py
IOError: cannot open/home/guy/Projets/DESI/analysis/pixelsims/20160126/fibermap-00000000.fits
The only fix I found is with a symbolic link
ln -fs 0/20160126 20160126
How did we ever live without them?
@moustakas : the desisim tests still require old $DESI_ELG_TEMPLATES etc environment variables. I'm not sure if this is because some corner of the code still needs them, or if just the tests themselves need to be updated. Can you look into this and fix the code and/or tests to use the new $DESI_BASIS_TEMPLATES system and get our tests back to a working state?
python setup.py test
...
======================================================================
ERROR: test_read_templates (desisim.test.test_io.TestIO)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./py/desisim/test/test_io.py", line 119, in test_read_templates
flux, meta = io.read_templates(wave, objtype, nspec=3)
File "./py/desisim/io.py", line 439, in read_templates
raise ValueError("ERROR: $"+key+" not set; can't find "+objtype+" templates")
ValueError: ERROR: $DESI_ELG_TEMPLATES not set; can't find ELG templates
----------------------------------------------------------------------
newexp-desi --flavor science to --flavor dark--flavor options for
bgs (Bright Galaxy Survey only)mws (Milky Way Survey only)bright mix of BGS and MWSdark option for specific cases elg, lrg, qso so that if you are testing only one type of object, you don't have to simulate the others.All cases should still include standard star and sky fibers included.
Build the templates on-the-fly by calling the relevant functions in desisim.templates rather than using the static (on-disk) templates.
Add a template class appropriate for simulating Bright Galaxy Survey (BGS) targets. A short term fallback hack could be to use the existing LRG class with a custom zrange, zmagrange, and nocolorcuts=True.
A recent run of quickbrick (tag 0.8.2) produced this output:
quickbrick -b valid-mix-50 -n 50 --objtype=mix --wavestep=0.2 --downsampling=5 --seed=xxxxx --outdir_truth=yyy
INFO:io.py:466:read_basis_templates: Reading /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/star_templates_v1.2.fits
/project/projectdirs/desi/software/cori/desisim/0.8.2/lib/python2.7/site-packages/desisim-0.7.dev287-py2.7.egg/desisim/filterfunc.py:68: RuntimeWarning: invalid value encountered in double_scalars
flux /= np.trapz(self.interp(wave1)*wave1,wave1)
INFO:io.py:466:read_basis_templates: Reading /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/lrg_templates_v1.2.fits
INFO:io.py:466:read_basis_templates: Reading /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/qso_templates_v1.1.fits
INFO:io.py:466:read_basis_templates: Reading /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/elg_templates_v1.5.fits
INFO:io.py:466:read_basis_templates: Reading /project/projectdirs/desi/spectro/templates/basis_templates/v1.1/star_templates_v1.2.fits
/global/cscratch1/sd/kisner/software/hpcports_gnu/astropy-1.0.1_f4385b62-1.0/lib/python2.7/site-packages/astropy-1.0.1-py2.7-linux-x86_64.egg/astropy/io/fits/fitsrec.py:774: UserWarning: TDIM7 value (10,5) does not fit with the size of the array items (10). TDIM7 will be ignored.
actual_nitems, indx + 1))
Traceback (most recent call last):
File "/project/projectdirs/desi/software/cori/desisim/0.8.2/bin/quickbrick", line 4, in <module>
__import__('pkg_resources').run_script('desisim==0.7.dev287', 'quickbrick')
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 699, in run_script
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1617, in run_script
File "/global/project/projectdirs/desi/software/cori/desisim/0.8.2/lib/python2.7/site-packages/desisim-0.7.dev287-py2.7.egg/EGG-INFO/scripts/quickbrick", line 279, in <module>
fx.append(fits.ImageHDU(qsim.wavelengthGrid.astype(np.float32), name='_SOURCEWAVE', header=header))
NameError: name 'fx' is not defined
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.