desi-tiles.fits updates about desimodel

[djschlegel]

Stephen- I did follow the FITS convention for unsigned INTs, by storing as TFORM='I'
with the TSCAL,TZERO keywords set to define the offsets between unsigned and signed int.
But that is certainly awkward and prone to error, so could switch OBSCONDITIONS to signed int?

The padding of PROGRAM with trailing blanks is a problem when generating FITS files
with MWRFITS. Let me know if you need this fixed in the file, and I might need some help.

I've also finally computed the STAR_DENSITY entries and will add those to the file.

[sbailey]

I posted this last night but either it didn't succeed or I posted it to the wrong issue:

The OBSCONDITIONS issue appears to be yet another astropy bug. Reading in the original table has the type as 'i2' (rather than 'u2'), but pulling out just that column strangely converts it to 'f8', as does converting it to an astropy.table.Table:

In [1]: import desimodel.io
   ...: t = desimodel.io.load_tiles()
   ...: 

In [2]: t.dtype
Out[2]: dtype((numpy.record, [('TILEID', '>i4'), ('RA', '>f8'), ('DEC', '>f8'), ('PASS', '>i2'),
('IN_DESI', '>i2'), ('EBV_MED', '>f4'), ('AIRMASS', '>f4'), ('STAR_DENSITY', '>f4'),
('EXPOSEFAC', '>f4'), ('PROGRAM', 'S6'), ('OBSCONDITIONS', '>i2')]))

In [3]: t['OBSCONDITIONS'].dtype
Out[3]: dtype('float64')

In [4]: tx = Table(t)

In [5]: tx['OBSCONDITIONS'].dtype
Out[5]: dtype('float64')

In [6]: tx.dtype
Out[6]: dtype([('TILEID', '>i4'), ('RA', '>f8'), ('DEC', '>f8'), ('PASS', '>i2'), ('IN_DESI', '>i2'),
('EBV_MED', '>f4'), ('AIRMASS', '>f4'), ('STAR_DENSITY', '>f4'), ('EXPOSEFAC', '>f4'),
('PROGRAM', '<U6'), ('OBSCONDITIONS', '<f8')])

A signed 32-bit integer might be more robust, even if the existing file should be ok. The file is small enough that 16 vs. 32 bits for this column doesn't matter.

[weaverba137]

I think this is another gotcha from using astropy.io.fits.getdata(). When I use the recommended method for opening FITS files, I don't see a problem with OBSCONDITIONS.

>>> import os
>>> from astropy.io import fits
>>> tilefile = os.path.join(os.environ['DESIMODEL'], 'data', 'footprint', 'desi-tiles.fits')
>>> with fits.open(tilefile) as hdulist:
...     tiles = hdulist[1].data
... 
>>> tiles
FITS_rec([ (1, 304.11000000000001, 16.57, 0, 0, 0.18588032, 1.0678819, 0.0, 3.3637536, 'DARK', 1),
       (2, 306.47000000000003, 15.109999999999999, 0, 0, 0.1337724, 1.0762441, 0.0, 2.4738233, 'DARK', 1),
       (3, 308.78999999999996, 13.630000000000001, 0, 0, 0.088750921, 1.0855807, 0.0, 1.9014858, 'DARK', 1),
       ...,
       (57618, 253.88, -0.080000000000000002, 9, 0, 0.22006717, 1.2213459, 0.0, 4.8983951, 'EXTRA', 0),
       (57619, 256.31999999999994, 1.3999999999999999, 9, 0, 0.15967406, 1.2016389, 0.0, 3.3238409, 'EXTRA', 0),
       (57620, 256.32999999999993, -1.53, 9, 0, 0.33952063, 1.2421075, 0.0, 10.347806, 'EXTRA', 0)], 
      dtype=(numpy.record, [('TILEID', '>i4'), ('RA', '>f8'), ('DEC', '>f8'), ('PASS', '>i2'), ('IN_DESI', '>i2'), ('EBV_MED', '>f4'), ('AIRMASS', '>f4'), ('STAR_DENSITY', '>f4'), ('EXPOSEFAC', '>f4'), ('PROGRAM', 'S6'), ('OBSCONDITIONS', '>i2')]))
>>> tiles['OBSCONDITIONS']
array([1, 1, 1, ..., 0, 0, 0], dtype=uint16)
>>> tiles['OBSCONDITIONS'].dtype
dtype('uint16')

[weaverba137]

Also, I'm not seeing the trailing space in the PROGRAM column.

>>> tiles['PROGRAM']
chararray(['DARK', 'DARK', 'DARK', ..., 'EXTRA', 'EXTRA', 'EXTRA'], 
      dtype='|S6')
>>> tiles['PROGRAM'][0]
'DARK'

[weaverba137]

Aha, the actual cause of float64 appearing is onlydesi=True. If you use load_tiles() with onlydesi=False, the type is correctly set to uint16. It may also have to do with the order in which load_tiles() is called with different values of onlydesi.

[weaverba137]

This is really weird, but this problem really does seem to depend on the order in which load_tiles() is called. This snippet succeeds:

import numpy as np
from desimodel.io import load_tiles
tiles1 = load_tiles(False)
assert tiles1['OBSCONDITIONS'].dtype is np.dtype(np.uint16)
tiles2 = load_tiles(True)
assert tiles2['OBSCONDITIONS'].dtype is np.dtype(np.uint16)
tiles3 = load_tiles(False)
assert tiles3['OBSCONDITIONS'].dtype is np.dtype(np.uint16)
tiles4 = load_tiles(True)
assert tiles4['OBSCONDITIONS'].dtype is np.dtype(np.uint16)

But all I have to do is change tiles1 = load_tiles(True) and it fails on the first assert.

[sbailey]

Under the hood in desimodel.io.load_tiles() it is caching the data from the file so that it is only read once. The onlydesi filter is done on the fly every time. I'm mystified why/how that filtering could change the OBSCONDITIONS dtype, however, and why the order would matter.

[weaverba137]

Indeed, and not only does it appear to depend on the order, I'm also finding inconsistent results when running the snippet above alone, versus running a similar test in test_io.py, i.e. in a formal unit test.

[weaverba137]

I'm starting to see some light. My current theory goes like this:

1. When load_tiles() is first called, it loads the data just fine.
2. However, just loading the data is not sufficient to trigger some of the internal mechanisms that astropy.io.fits uses to handle unsigned integer columns (and scaled columns in general). Some sort of access to either the data itself or the object's dtype is required for this.
3. In particular, slicing the array prior to actually accessing any of the data fails to trigger this mechanism.

[weaverba137]

OK, tests are passing in my git checkout, though Travis tests still need the updated file as mentioned in #31.

[sbailey]

You should be good to go using the test-0.5.0 branch.

[sbailey]

I stripped the training whitespace from the PROGRAM column with the following python code snippet:

import numpy as np
from astropy.io import fits

t, hdr = fits.getdata('desi-tiles.fits', 1, header=True)
t['PROGRAM'] = np.char.strip(t['PROGRAM'])
fits.writeto('blat.fits', t, header=hdr, clobber=True)

There were some minor changes to the comments on required header keywords from IDL vs. astropy.io.fits, but otherwise the files were identical except for the PROGRAM column having whitespace or not. I renamed blat.fits to desi-tiles.fits and committed to svn.

I also verified that the int32 OBSCONDITIONS column remains an int when read in and manipulated via astropy.

Remaining small to do on this ticket: update desimodel.io.load_tiles to exclude PROGRAM='EXTRA' tiles by default.