desisim.simexp.simulate_spectra not varying exposure time about desisim

[moustakas]

Thanks for the ticket, @humnaawan, I can try to look into this.

[andreufont]

Thanks @moustakas for looking into this. At least in the Lyman alpha WG none had played with these observing conditions before, so I'm not too surprised that there are hiccupts. I hope these are easy to fix!

[alxogm]

Hi all, thanks @moustakas for taking a look.

For reference, in the Y1 simulations we make, we do vary the exposure time, and it does change the spectra. It uses the same routine desisim.simexp.simulate_spectra Humna is trying to use.

I see in quickquasars we start with a reference set of obsconditions, which we then modify, here is the line we start with,

desisim/py/desisim/scripts/quickquasars.py

Line 874 in d9b552f

obsconditions = reference_conditions[args.program.upper()]

we then just modify the exptime for each target as needed, and I think the same applies for the other arguments in obsconditions. So maybe is issue at the dictionary level?

[andreufont]

@moustakas - any update on this? We will discuss this project at the KP6 telecon later today.

[moustakas]

Apologies, I have not had a chance to look yet. But @humnaawan can you include an end-to-end example of what you're doing, including the input spectra you're passing to simulate_spectra? Thanks!

[humnaawan]

no worries @moustakas! here's the code that should produce the issue:

import numpy as np
from astropy.io import fits
from astropy.table import Table
import desisim.simexp

# read in the simpec file: wave, flux, obsconditions
simspec_fname = '/global/cfs/cdirs/desi/users/awan/20191206/00000344/simspec-00000344.fits'
simspec_hdul = fits.open(simspec_fname)
wave_simspec = simspec_hdul['WAVE'].data
flux_all_fibers = simspec_hdul['FLUX'].data
obsconditions = simspec_hdul['OBSCONDITIONS'].data
obsconditions_dict = [dict(zip(obsconditions.columns.names, row)) for row in obsconditions][0]

# read in the fibermap
fibermap_fname = '/global/cfs/cdirs/desi/users/awan/20191206/00000344/fibermap-00000344.fits'
fibermap = Table.read(fibermap_fname, format='fits')
fibermap_by_petal = fibermap.group_by('PETAL_LOC')

# get the flux for one petal
petal_ind = 0
fibermap_this_petal = fibermap_by_petal.groups[petal_ind]
flux_this_petal = flux_all_fibers[fibermap_this_petal['FIBER'].quantity, :] # assuming row index = fiber number

# set up the simulator object without any observing conditions
simulator = desisim.simexp.simulate_spectra(wave=wave_simspec,
                                            flux=flux_this_petal,
                                           )
out0 = simulator.camera_output

# now set up the simulator object with specific observing conditions
simulator = desisim.simexp.simulate_spectra(wave=wave_simspec,
                                            flux=flux_this_petal,
                                            obsconditions=obsconditions_dict
                                           )

out1 = simulator.camera_output

# compare the outputs
for cam in range(3):
    print(f'things match: {np.unique(out0[cam] == out1[cam])}')

in case its helpful, i've put a copy of the notebook with all the code above at /global/cfs/cdirs/desi/users/awan/notebooks/. please lmk if this code is not able to reproduce the issue with you (i.e. out0 is exactly the same as out1).

[sybenzvi]

@humnaawan, I get the same results as you with the current main branches of desisim and specsim.

However, I decided to see if the full sim_spectra function in desisim.scripts.quickspectra responds correctly to changes in EXPTIME or is insensitive as you reported before. For me the script behaves reasonably. I increased the EXPTIME to an absurd number to make differences in the fluxes obvious, and I do see a difference. I'm loading the desiconda master package when running this.

from desisim.scripts.quickspectra import sim_spectra
from desispec.io import read_spectra
...

obsconditions_dict['EXPTIME'] = 100000.0

# Run specsim, generate realizations of the flux model, and output to a FITS file
sim_spectra(wave=wave_simspec,
            flux=flux_this_petal, 
            program='dark', 
            obsconditions=obsconditions_dict, 
            spectra_filename='out1.fits')

spec1 = read_spectra('out1.fits')

# plot stuff...

[alxogm]

@humnaawan I think the issue is that the simulator object is not being cleared by just reusing the variable with different arguments. I added this line to your notebook

desisim.specsim._simulators.clear()
before calling the simulator for the second time and that cleared everything and returned different spectra. Can you try it also fix the issue for you?

[humnaawan]

@sybenzvi thanks! so i was largely following how spectra are created in quickspectra which is why i was stumped at what i was seeing; the reason why i'm working with the simulator object directly is that we need to handle instrument response in more detail than is done in quickspectra.

adding desisim.specsim._simulators.clear() as @alxogm suggested works though so that's great. maybe this should be mentioned somewhere since the documentation for desisim.simexp.simulate_spectra gives no impression that the simulator object needs to be cleared for things to not be wrong.