gbrammer / msaexp Goto Github PK

Collecting drizzle<1.15.0,>=1.14.3 (from jwst->msaexp) Using cached drizzle-1.14.3.tar.gz (103 kB) Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [49 lines of output] C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\config\expand.py:134: SetuptoolsWarning: File 'C:\Users\Bruno\AppData\Local\Temp\pip-install-otd0rc0_\drizzle_b6176f633f4846d88005e2b8a12a415d\licenses' cannot be found return '\n'.join( running egg_info writing drizzle.egg-info\PKG-INFO writing dependency_links to drizzle.egg-info\dependency_links.txt writing requirements to drizzle.egg-info\requires.txt writing top-level names to drizzle.egg-info\top_level.txt reading manifest file 'drizzle.egg-info\SOURCES.txt' Traceback (most recent call last): File "C:\Users\Bruno\AppData\Local\Programs\Python\Python310\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 353, in main() File "C:\Users\Bruno\AppData\Local\Programs\Python\Python310\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 335, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "C:\Users\Bruno\AppData\Local\Programs\Python\Python310\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 118, in get_requires_for_build_wheel return hook(config_settings) File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\build_meta.py", line 355, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=['wheel']) File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in get_build_requires self.run_setup() File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\build_meta.py", line 341, in run_setup exec(code, locals()) File "", line 38, in File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools_init.py", line 103, in setup return distutils.core.setup(**attrs) File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools_distutils\core.py", line 185, in setup return run_commands(dist) File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools_distutils\core.py", line 201, in run_commands dist.run_commands() File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools_distutils\dist.py", line 969, in run_commands self.run_command(cmd) File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\dist.py", line 989, in run_command super().run_command(command) File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools_distutils\dist.py", line 988, in run_command cmd_obj.run() File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 318, in run self.find_sources() File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 326, in find_sources mm.run() File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 548, in run self.add_defaults() File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 593, in add_defaults self.read_manifest() File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\command\sdist.py", line 214, in read_manifest self.filelist.append(line) File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 485, in append path = convert_path(item) File "C:\Users\Bruno\AppData\Local\Temp\pip-build-env-4xk_b5iv\overlay\Lib\site-packages\setuptools_distutils\util.py", line 139, in convert_path raise ValueError("path '%s' cannot be absolute" % pathname) ValueError: path '/home/runner/work/drizzle/drizzle/src/cdrizzleapi.c' cannot be absolute [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip.

Implement option to use background_ slits to subtract a background directly from science slits if the 2D pixels line up.

Add capability for a wavelength-dependent width of the extraction profile, e.g., for the prism. Is it written down anywhere what this variation is? WebbPSF?

Hi Gabe,

For EGS, meta_file.get_transforms function seems to fail to reproduce accurate mapping of the slits.

The polynomial coefficients for EGS Dec are quite off. i.e. expected declinations are ~ +52.8 but c0_0=67.74.
Compared to values in COSMOS and UDS they all result in a significantly larger rms (see screenshot).

Thus, the PolygonSkyRegion computed to match positions in the sky does not resemble to correct positions in the sky.

When extracting spectra to perform background subtraction at the slit level NirspecPipeline.extract_spectrum for some sources the dimensions between slits seems to be off by 1 which causes extractions to fail.

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Input In [17], in <cell line: 5>()
      1 ### check in nrs2
      3 key_nrs2 = '2565_32130' 
----> 5 _data = gratings['clear-prism'][1].extract_spectrum(key_nrs2, skip=[], yoffset=0, prof_sigma=0.6, trace_sign=-1, 
      6                               fit_profile_params=None)

File ~/opt/anaconda3/envs/grizli39/lib/python3.9/site-packages/msaexp/pipeline.py:1356, in NirspecPipeline.extract_spectrum(self, key, slit_key, prof_sigma, fit_profile_params, pstep, show_sn, flux_unit, vmax, yoffset, skip, bad_dq_bits, clip_sigma, ntrim, get_slit_data, verbose, center2d, trace_sign, min_dyoffset, **kwargs)
   1354 _dq = (_sbg.dq & bad_dq_bits) == 0
   1355 try:                    
-> 1356     _bkg[_dq] += _sbg.data[_dq]
   1357     _bkgn[_dq] += 1
   1359 except ValueError:
   1360     #print('background failed', j)

IndexError: boolean index did not match indexed array along dimension 1; dimension is 142 but corresponding boolean dimension is 143

ipdb>  np.shape(_dq)
(30, 143)
ipdb>  np.shape(_sbg.data)
(30, 143)
ipdb>  np.shape(_bkg)
(30, 142)

This is coming because

ipdb>  pipe[0].slits[0].data.shape
(35, 425)
ipdb>  pipe[1].slits[1].data.shape
(35, 425)
ipdb>  pipe[2].slits[2].data.shape
(35, 426)

using
grizli version = 1.8.4.dev4+ga5d2ece
msaexp version = 0.6.8.dev8+gf5c84b2.d20230403
jwst version = 1.10.0

For some slits, ValueErrors are raised when drizzling (pull trace below)

Likely to do with the WCS arguments from the slit model doesn't match with the 2D slit sizes when reconstructed?

      8                       single=True,
      9                       blendheaders=True,
   (...)
     15                       pscale_ratio=1.0,
     16                       pscale=None)
     18 ### nrs 1
---> 19 hdul_nrs1 = msaexp.utils.drizzle_2d_pipeline(slits_nrs1, drizzle_params=DRIZZLE_PARAMS,
     20                                         fit_prf=True,
     21                                         outlier_threshold=5,
     22                                         prf_center=None, prf_sigma=0.6, center_limit=4, 
     23                                         standard_waves=1, pixfrac=2.0)
     26 _fig = msaexp.utils.drizzled_hdu_figure(hdul_nrs1, xlim=None)
     28 fig, axes = plt.subplots(2,1,figsize=(10,4), sharex=True)

File ~/opt/anaconda3/envs/jwst/lib/python3.10/site-packages/msaexp/utils.py:1107, in drizzle_2d_pipeline(slits, output_root, standard_waves, drizzle_params, include_separate, **kwargs)
   1103 if (standard_waves > 0) & ('wave_array' not in kwargs):
   1104     # Get fixed wavelength grid
   1105     waves = get_standard_wavelength_grid(slits[0].meta.instrument.grating, 
   1106                                 sample=standard_waves)
-> 1107     _data0 = drizzle_slits_2d(slits,
   1108                               drizzle_params=drizzle_params,
   1109                               wave_array=waves, **kwargs)
   1110 else:                                
   1111     _data0 = drizzle_slits_2d(slits,
   1112                               drizzle_params=drizzle_params,
   1113                               **kwargs)

File ~/opt/anaconda3/envs/jwst/lib/python3.10/site-packages/msaexp/utils.py:782, in drizzle_slits_2d(slits, build_data, drizzle_params, **kwargs)
    779     kwargs['pscale_ratio'] = drizzle_params['pscale_ratio']
    781 if build_data is None:
--> 782     _data = build_regular_wavelength_wcs(slits, **kwargs)
    783     target_waves, header, data_size, output_wcs = _data
    784 else:

File ~/opt/anaconda3/envs/jwst/lib/python3.10/site-packages/msaexp/utils.py:509, in build_regular_wavelength_wcs(slits, pscale_ratio, keep_wave, wave_scale, log_wave, refmodel, verbose, wave_range, wave_step, wave_array, ypad, get_weighted_center, center_on_source, **kwargs)
    507 grid = wcstools.grid_from_bounding_box(bbox)
    508 _, s, lam = np.array(d2s(*grid))
--> 509 sd = s * refmodel_data
    510 ld = lam * refmodel_data
    511 good_s = np.isfinite(sd)

ValueError: operands could not be broadcast together with shapes (30,434) (30,437) 

Following on issue #30 @gbrammer sllitlet_id being used here to create file names seems to not match what is expected when background subtraction is done in 2D extractions.

In a 3 point nod pattern, the msa meta data file has 3 seperate sllitlet_ids. It looks like the files are named using one of the 3, in an example case in 2565 the files are names as
jw02565200001_03101_00002_nrs1_phot.252.2565_18440.fits
The object 18440 is found in slitlets 56, 154, and 252.

In
pipe.set_background_slits()

indices = [slit.slitlet_id
               for slit in self.pipe[self.last_step][0].slits] 

returns the lower number 56. If the index [1] is chosen instead of [0] then it will be 154 and so forth.

So this causes load_slit_data() to fail because condition
if (counts[0] > 0) & (np.allclose(counts, np.min(counts))):

fails (the file is not there, it is looking for 056 while the file is named as 256).

If a object is not identified in the MSA and (perhaps) not extracted through the standard identification of slits, it would be good to be able force extract a spectrum by specifying the slit in question and a different slit of background subtraction.

In

Gives rise to

--> 291     if len(self.files == 0):
    292         self.files = glob.glob(file.replace('cal.fits', 
    293                                             f'{step}.*.fits'))
    295 self.files.sort()

TypeError: object of type 'bool' has no len() 

Currently the slit-level background subtraction assumes something like the 3-point dithers along the slit and that all exposures are at different positions. That isn't necessarily true for the general case where multiple exposures can be taken at a given dither position.

The weight masking functions in make_optimal_extraction occasionally corrupt the 1D extractions at very high S/N. The earlier default behavior max_wht_percentile=98 masked pixels where the weight was above the 98th percentile, in order to flag occasional pixels that had extremely high weights for some unknown reason. This would occasionally clip the centers of bright targets.

This change implemented a new masking scheme with the parameter max_med_wht_factor=10, which would mask pixels that have weights larger than 10 times the median (nonzero) weight in the 2D array. That seems to mostly fix the issue.

Here's an example comparing the weighting schemes:

import matplotlib.pyplot as plt
from msaexp import spectrum

# A bright UNCOVER object
file = 'https://s3.amazonaws.com/msaexp-nirspec/extractions/uncover-4m1-v1/uncover-4m1-v1_prism-clear_2561_42213.spec.fits'

spec = spectrum.SpectrumSampler(file)

new = spec.redo_1d_extraction(bkg_offset=6,
                              max_wht_percentile=None, # Turn off this mask
                              max_med_wht_factor=20)

_ = spec.drizzled_hdu_figure(unit='flam')

_ = new.drizzled_hdu_figure(unit='flam')

fig, ax = plt.subplots(1,1,figsize=(9,4))

ax.plot(spec['wave'], spec['flux']*spec['to_flam'], label='Original, max_wht_percentile=98', color='r', alpha=0.5)
ax.plot(new['wave'], new['flux']*spec['to_flam'], label='max_med_wht_factor=10', color='k', alpha=0.5)

ax.legend()
ax.set_xlim(0.6, 5.2)
ax.grid()
ax.semilogx()

msaexp seems to produce corrupted 2D profiles in utils.make_optimal_extraction

This generates 'phantom' emission lines in 1D where the 2D looks fine

This happens at low SNR, so is a different bug from the previous masking bug

importlib.reload(msaexp)
importlib.reload(drizzle)

hdul = fits.open('uncover_2561_msa2-10646-prism-clear.d2d.fits')
prf_fwhm = 1.9
outhdu = msaexp.drizzle.extract_from_hdul(hdul,
                                              prf_sigma=prf_fwhm/2.35, fix_sigma=False,
                                              prf_center=None, fix_center=False,
                                              verbose=True, fit_prf=True,
                                              )


hdul.info()
sci2d = outhdu['SCI'].data
prf2d = outhdu['PROFILE'].data
wht2d = outhdu['WHT'].data
plt.imshow(prf2d,interpolation='nearest')

When calling the pipeline.NirspecPipeline, specifying source_ids leads to the following error: local variable 'six' was referenced before assignment.

The issue was traced to the pad_msa_metafile function and happens when this line is executed - this_source_ids = np.unique(msa.shutter_table['source_id'][six])

If source_ids are not specified inpipeline.NirspecPipeline, the code runs normally.

The erros for the optimally extracted spectra seems to be underestimated compared to the 'real' expectation. i.e. from the ETC or looking ash the error values from single exposures from the pipeline. The extract_spectrum function seems to be where the real science happens here.

We did a few experiments with some of 2565 data to see what was going on.

Removing any background subtraction we get 3 spectra with good continuum.

The flux levels are not similar between the 3 dithers due to object entering issues, so we've fit a polynomial to 2 of the dithers to normalise the flux and errors to the first dither position. The same correction is also applied to the error spectrum.

Once this normalisation is applied, the flux variation between the individual pixels can be thought to be driven by the noise value. So the RMS of the flux values in the 3 dithers (show in solid black below) can be compared with the normalised errors of the 3 dither positions. They agree well (some sigma clipping applied to remove CR noise etc)

Now the figure below shows the S/N for the 3 discussed above + also with background subtraction as handled by msaexp. It is a bit busy, but basically the 3 dither positions with no background subtraction are in the solid rgb colours. The 6 A-B iterations are shown by dashed colours. The black dashed in S/N = 30 to guide the eye.

The solid black is the optimally extracted spectrum produced by the extract_spectrum function.

The S/N is ~ sqrt(6) higher compared to the individual error levels. However, since the background errors in A-B and A-C are correlated, it should be only sqrt(3) higher.

The extract_spectrum function stacks all 6 iterations together and takes the sum with optimal ivar scaling. So basically the 6 background subtractions are considered to be independent. Like

Is our understanding here correct?

Using the the latest version of the MSAEXP (0.6.6) with H-grating data produces background slitlets that have differing shapes for their data and their error (see attached image). This causes the extraction from drizzled files to fail. Any suggestions on how to address this would be great! Thank you again for the great code to reduce the data.

The background that was subtracted from the 2D spectra is available in the d2d products, but is not propagated through to the spec.fits products. Add these in.