desihub / desimeter Goto Github PK

I'm looking at FVC expnum 66575, and it has EXTNAME keys:

> listhead /tmp/fvc-00066575.fits | grep EXTNAME
EXTNAME = 'FVC     '
EXTNAME = 'C0003   '           / name of this binary table extension
EXTNAME = 'F0004   '
EXTNAME = 'C0004   '           / name of this binary table extension
EXTNAME = 'F0005   '
EXTNAME = 'C0005   '           / name of this binary table extension
EXTNAME = 'F0006   '
EXTNAME = 'C0006   '           / name of this binary table extension
EXTNAME = 'F0007   '
EXTNAME = 'C0007   '           / name of this binary table extension
EXTNAME = 'F0008   '
EXTNAME = 'C0008   '           / name of this binary table extension
EXTNAME = 'F0009   '
EXTNAME = 'C0009   '           / name of this binary table extension
EXTNAME = 'F0010   '
EXTNAME = 'C0010   '           / name of this binary table extension
EXTNAME = 'F0011   '
EXTNAME = 'F0000   '
EXTNAME = 'C0000   '           / name of this binary table extension
EXTNAME = 'F0001   '
EXTNAME = 'C0001   '           / name of this binary table extension
EXTNAME = 'F0002   '
EXTNAME = 'C0002   '           / name of this binary table extension
EXTNAME = 'F0003   '
EXTNAME = 'C0037   '           / name of this binary table extension
EXTNAME = 'C0024   '           / name of this binary table extension

ie, all mixed up. It looks like desi_fvc_proc with the --sequence option, however, assumes they are in order, and writes out the first image into output file *-F0000.csv.

There is duplicate code for loading metrology data in zb.py and poly2d.py.

I suggest moving this to a higher level. Implement in one place directly in the base class.

To correctly go from (x,y,z) space (e.g. FVC pixels) to positioner coordinates (e.g. Q,S), we must first align that (x,y,z) space as nearly as possible to the CS5 coordinate system. We therefore need a procedural definition of CS5 using tangible measured values. I posted a note to DESI-5435 (also attached here for convenience), which offers one suggestion.

The note also includes some background, data, and explanations describing how well we know the petals' as-mounted radial positions based on CMM data done at LBNL. That may be useful info.

2019-01-03 Comment on operational defn of CS5.pdf

Adjust the fiducials coordinates by stacking the measured coordinates from FVC images.

GEAR_CALIB_T and GEAR_CALIB_P are currently 1.0 in all cases on the Mayall. There may come a date at which robots have variable values set for these. This will affect the internally-tracked POS_T and POS_P angles (which is fine).

However, when running fit_posparams to deduce new values for SCALE_T and SCALE_P, the internally-tracked angles must be corrected for whatever GEAR_CALIB_* values were being used at the time the robot was commanded. Otherwise, the comparison of delta(commanded) vs delta(measured) will come out wrong.

Fix:
1.fitter.py --> Add additional arguments for vectors of GEAR_CALIB_T and GEAR_CALIB_P.
2. In dynamic mode, pre-adjust the dt_int and dp_int data back to as if GEAR_CALIB_* == 1 at time of move.
3. Then run the best-fit.

A configuration file with path to the various calibration files is needed.
Especially for the FVC bias file.

If we want to test using desimeter transforms (alternate to platemaker) on the instrument, I think the easiest thing might be to run an "xy test" with a few suggested modifications, described below. Nothing is easy, exactly, but I guess this is least pain. The system is pretty intricate --- I'm sure we'd want Duan or Kevin to do this. Anyway I took a pass through the code and think these are the changes needed. All code links below are at current revisions as of writing.

pecs.py

• c.f. https://desi.lbl.gov/trac/browser/code/focalplane/plate_control/trunk/pecs/pecs.py?rev=129541
• Line 62: insert a new case like if fvc == 'desimeter':
  • Might need Klaus here to give us a platemaker-less FVC proxy that just returns images. c.f. https://desi.lbl.gov/trac/browser/code/online/DOSlib/trunk/python/DOSlib/proxies.py?rev=129583
  • Or perhaps David R could give us some code to more directly set up the FVC.
• Line 182: make a conditional like if fvc == 'desimeter':
  • Put in code here that triggers an exposure, sends the FITS file to desimeter.fvc2fp() function, gets back QS values, and then shoehorns them into format that the "meapos" dataframe is supposed to contain. (Who here wishes we were running in a properly typed language... raise your hands! I don't have the heart to trace down how meapos is structured at the moment.)

xytest.py

• c.f. https://desi.lbl.gov/trac/browser/code/focalplane/plate_control/trunk/pecs/xytest.py?rev=129521
• Line 173 - 200: make this compatible with multi-petal input targets file (see format description below)

Input targets file

• The current format is kind of awkward.
  • It looks like: https://desi.lbl.gov/trac/browser/code/focalplane/fp_settings/test_settings/xytest_cmx_063064-0.csv?rev=128750
  • Has 1000 columns
  • Requires a unique file per petal.
  • Petal ID is encoded in file name string.
  • Device location ids are encoded mixed with coordinate system.
• Better format would have headers:
  • TARGET_NUMBER
  • PETAL_LOC
  • DEVICE_LOC
  • uv_type
  • u
  • v
  • For example with 10 petals, ~500 positioners / petal, and 10 test targets, the csv file would have 50,000 rows.
    • You could break this up into smaller files by petal, or smaller files by target number. Seems pointless to me, but you could do that. I prefer one file == one test. If you do break it up into smaller files, you need to change how these are stated in the config file.
  • For description of valid uv_type for move requests, please see posschedule.py.
    • request_target() function, comments in lines 42-53.
    • https://desi.lbl.gov/trac/browser/code/focalplane/plate_control/trunk/petal/posschedule.py?rev=129559
• Easiest place to put the targets csv is the same folder as the xytest config file, i.e. code/focalplane

Test settings config file

• Based off of template: https://desi.lbl.gov/trac/browser/code/focalplane/fp_settings/test_settings/xytest_template.cfg?rev=129521
• We should add a parameter:
  • fvc2fp_mode
  • valid options would be: 'platemaker' or 'desimeter'
  • or perhaps could add further suboptions if you want to try different flavors of desimeter, like maybe 'desimeter_7paramfit' vs 'desimeter_42paramfit'. Whatever.
• Fill in all petals to be tested, like ['01'], ['02], etc
  • mode = 'all' --> will test all positioners on that petal with CTRL_ENABLED
  • alternately to test only a specific list of positioners...
    • mode = 'can'
    • posids = [posid_str1, posid_str2, posid_str3, ...]
• input_targs_file = path to your csv
  • this may require some adjustment (see above)
• target_type
  • I think this should be eliminated and just use coordinate system specified in the uv_type of input file. It's redundant and just one more setting to get wrong at time of test.
• num_corr_max --> probably set this to 3 for purposes of test
• targ_max_radius --> set to None for clarity (it will be ignored, since using csv of targets)
• targ_min_radius --> set to None for clarity (it will be ignored, since using csv of targets)
• pause_interval --> I'm not sure what our latest info on cooling is, but 590 sec is the nominal design limit. We can probably go a bit faster.

• Derive field models from acquisition exposures (using gfa_reduce)
• Compare measured field rotation with the design value in the fiber assign file
• Derive an hexapod rotation correction
• Compare with the actual hexapod rotation applied during commissioning

PR #22 implemented radec2tan, but we'll also need tan2radec for analysis of where fibers were on the sky, given their measured locations on the FVC and FP.

basedir=/project/projectdirs/desi/spectro/data/20200107/00038674
desi_fvc_proc \
  --infile $basedir/fvc-00038674.fits.fz \
  --expected $basedir/coordinates-00038674.fits \
  --outfile spots-00038674.csv

PETAL_LOC=0 DEVICE_LOC=533 is matched, but with PINHOLE_ID=0 for all 4 spots.

x = Table.read('spots-00038674.csv')                                                                                                  
x['XPIX', 'YPIX', 'PETAL_LOC', 'DEVICE_LOC', 'PINHOLE_ID'][x['LOCATION'] == 533]                                                      

<Table length=4>
       XPIX               YPIX        PETAL_LOC DEVICE_LOC PINHOLE_ID
     float64            float64         int64     int64      int64   
------------------ ------------------ --------- ---------- ----------
 2689.077254236983  474.1507123657714         0        533          0
2688.0201166683487 479.83004331316874         0        533          0
 2693.642353089485 482.22714056577547         0        533          0
2686.9455325543513 485.50391201761533         0        533          0

i.e. it seems to be successfully matching those pinholes to the correct device location (they aren't just unmatched spots) but the pinhole IDs aren't being set. Why?

These are my suggestions for broad direction of getting "the analysis stuff currently in PECS" into desimeter.

General requirements

• Analysis runs completely independent from the instrument.
• Past analyses are instantly reproducible, at any time.
• All code in Github, with total visibility to the collaboration.
• Any data cached to disk must be in simple, standard formats (CSV for tables, JSON for structures, etc).

Data selection

Need some script(s) for grabbing move data off the onlinedb, and then caching it. See PR #71

Suggested new analysis tools

1. summarize_pos_errors

   • analyzes positioning performance data for a broad collection of positioners
   • capable of merging night data and multiple xytests etc --- not just one "xytest" (!)
   • generates a table like DESI-5500, summarizing targeting errors over a broad swath of positioners
     • counts of target errors at quantiles: 1σ, 2σ, 3σ, 4σ, 5σ, 100%
     • see script there, xytestdata_summarize_v2.py
   • generates error histogram for each submove
     • Num targets, max err, rms err, mean err, median err printed on it.
   • option to do the summaries in various groupings:
     • by petal
     • by canbus
     • everything-that-was-tested-that-day

2. plot_pos_targmeas

   • generates a plot that overlays targets and measured points for a given positioner
   • does this overlap with anything already in desimeter?

3. plot_pos_arcmeas

   • select arc calibration data for a given positioner
   • do so by parsing notes in posmovedb
   • include ability to find / merge results from multiple calibration runs (!)
   • generate a plot of a given set of arc calibration points for a given positioner
   • does this overlap with anything already in desimeter?

4. fit_posparams_arc

   • same underlying selection function as plot_pos_arcmeas
   • do circle fits + mechanical assumptions to calculate calibration params

• Run gfa_reduce on the guide CCD image
• Fit astrometry (desimeter "field model")
• Save field model result
• Send back pointing offsets, hexapod rotation, guide star pixel coordinates to the ICS.

Compute field model on many exposures, predict fiber position on focal plane and compare with platemaker coordinates? files.
First goal is to fit shift, rotation, dilatation and correlate differences with ADC angles , parallactic angle and zenith angle ...

Develop a script and routine "fit_acquisition_exposure" that

• use as input a guide catalog with centroids in pixel coordinates (x_gfa_meas,y_gfa_meas), matched to GAIA ra dec (for instance /global/cscratch1/sd/schlafly/desi/gfa_reduce_202001/guide-00038408_catalog-00000.fits, with table columns xcentroid,ycentroid,ra_gaia,dec_gaia)
• transform ra_gaia,dec_gaia -> x_tan,y_tan using transform.radec2tan
• x_tan,y_tan -> x_fp,y_fp using transform.tan2fp.tan2fp
• x_fp,y_fp ->x_gfa_pred, x_gfa_pred using fp2gfa (to be implemented)
• fit a transformation from x_gfa_pred, x_gfa_pred to x_gfa_meas, y_gfa_meas with
  tel_ra, tel_dec, field_rotation, scale factor and maybe compression along parallactic angle

In ptl2fp.py, there is a utility function for getting petal alignment data:

The file path is hard-coded for grabbing the alignment data. This will cause versioning confusion in either of the following future cases:

1. Running desimeter on the spare petal at LBNL.
2. Running desimeter if any petals are replaced or retracted / inserted at the Mayall.

To avoid stateful confusion, I suggest that we have multiple configurations in petal-alignments.yaml. Identify each one by a unique key.

And ptl2fp() function will need an additional, required argument providing this key.

Should be "NotImplementedError".

And again at:

If you don't mind, I'd like to fix these myself. Just to learn the process of branching, making pull requests, etc in github.

Add matching of positioner DEVICE_LOC to FVC spots based upon their expected locations. IIUC, current code only matches static fiducials, but not moving positioners.

I think there may be places in desimeter where we should add checks that the current petal-alignments.yaml matches whatever was in the online constants DB at the time of the measurement.

For example when we are doing best fits to discover OFFSET_X, OFFSET_Y based on historical data, then this may be important.

I think the simplest and cleanest test is:

1. Get OBS_X, OBS_Y and PTL_X, PTL_Y from the online DB. Here the PTL values were generated in the online system by applying the Constants DB transformation params to OBS.

2. Use desimeter / petal-alignments.yaml to do the analagous conversion into PTL. (Note OBS == what desimeter calls X_FP, Y_FP.)

3. Check that PTL values from (1) and (2) should exactly match.

This comes out of a comment thread in #98 (comment).

Analyse ADC rotation data to validate the ray tracing model used in tan2fp transform.

DESI-5421 metrology is currently missing GFA data for petals 3 and 7, due to missing measurements that weren't taken in the lab. Add placeholders to desimeter/data/fp-metrology-patch.csv using the average of the metrology from the other petals in petal-local coordinates transformed for petals 3 and 7 offsets and rotations.

From Aaron:
4 of the exposures fail in desi_fit_guide_star_coordinates with an error "ValueError: A value in
x_new is above the interpolation range.". These are expid = 44840,
44885, 44895, 44900. They're all right near zenith.

There are a number of exposures on night 20200118 for which I obtain desimeter field center Dec coordinates that seem to be off by ~70 arcseconds. One example is EXPID = 41797. The specific command I'm using is:

desi_fit_guide_star_coordinates -i /project/projectdirs/desi/users/ameisner/GFA/reduced/v0005/20200118/00041797/gfa-00041797_catalog.fits --fits-header /project/projectdirs/desi/spectro/data/20200118/00041797/gfa-00041797.fits.fz -o gfa-00041797.json
WARNING:desi_fit_guide_star_coordinates:29::no TARGTRA in header of HDU 0, try HDU 1
INFO:fieldmodel.py:92:read_guide_stars_catalog:reading guide stars in /project/projectdirs/desi/users/ameisner/GFA/reduced/v0005/20200118/00041797/gfa-00041797_catalog.fits
INFO:fieldmodel.py:102:read_guide_stars_catalog:selection stars for which we have a good match (< 2.0 arcsec)
INFO:fieldmodel.py:131:fit_tancorr:Use 'mjd_obs' in catalog, MJD=58867.243140375576
INFO:fieldmodel.py:149:fit_tancorr:Use LST=93.87341566663235
ERROR:gfa2fp.py:31:gfa2fp:PETAL_LOC 3 GFA metrology missing
WARNING:fieldmodel.py:258:all_gfa2fp:missing metrology
ERROR:gfa2fp.py:31:gfa2fp:PETAL_LOC 7 GFA metrology missing
WARNING:fieldmodel.py:258:all_gfa2fp:missing metrology
INFO:fieldmodel.py:182:fit_tancorr:RMS coord. residual = 1.45 arcsec
INFO:fieldmodel.py:183:fit_tancorr:Rotation angle (field rot ZP) =0.044 deg
INFO:fieldmodel.py:184:fit_tancorr:Pointing correction dHA=-0.00 arcsec, dDec=-0.00 arcsec
INFO:fieldmodel.py:185:fit_tancorr:Scales sxx=1.0009 syy=1.0008 sxy=0.0000
wrote gfa-00041797.json

The desimeter output file gfa-00041797.json has (ra, dec) = (153.48500768926948, 15.662657681144372). This is using desimeter version 1d0a473.

Separately, I wrote my own code to determine the field center based on the same GFA reductions being passed to desimeter, and I get field center (RA, Dec) coordinates of (153.485567, 15.643475).
This is basically the same as what desimeter gets in RA but offset by ~70 arcseconds in Dec. The field center coordinates I derive with my own code agree very closely with the desired field center (SKYRA, SKYDEC) = (153.485423, 15.643504), which makes sense because this exposure was taken immediately after applying a pointing correction mount offset.

From this same pointing model sequence, there are 16 exposures where my field center (RA, Dec) values agree with desimeter, and these all have desimeter RMS_ARCSEC values of ~0.15-0.35:

EXPID = 41785 , RMS_ARCSEC = 0.18693942
EXPID = 41910 , RMS_ARCSEC = 0.27438225
EXPID = 41913 , RMS_ARCSEC = 0.27071138
EXPID = 41916 , RMS_ARCSEC = 0.19972174
EXPID = 41924 , RMS_ARCSEC = 0.24017903
EXPID = 41932 , RMS_ARCSEC = 0.22779664
EXPID = 41934 , RMS_ARCSEC = 0.18618517
EXPID = 41936 , RMS_ARCSEC = 0.26857016
EXPID = 41948 , RMS_ARCSEC = 0.27877184
EXPID = 41960 , RMS_ARCSEC = 0.19437467
EXPID = 41962 , RMS_ARCSEC = 0.34004326
EXPID = 41966 , RMS_ARCSEC = 0.27551664
EXPID = 41968 , RMS_ARCSEC = 0.24442977
EXPID = 41973 , RMS_ARCSEC = 0.23566265
EXPID = 41975 , RMS_ARCSEC = 0.19593430
EXPID = 41977 , RMS_ARCSEC = 0.25727198

On the other hand, there are 13 exposures from this pointing model sequence where desimeter gives field center results offset by ~70 arseconds in Dec. These cases always have intermediate values of RMS_ARCSEC in the ~1.2-1.5 range, so I'm wondering if that is providing some sort of clue as to what's happening.

EXPID = 41797 , RMS_ARCSEC = 1.4516730
EXPID = 41810 , RMS_ARCSEC = 1.2984165
EXPID = 41823 , RMS_ARCSEC = 1.4075735
EXPID = 41835 , RMS_ARCSEC = 1.3585141
EXPID = 41847 , RMS_ARCSEC = 1.4500220
EXPID = 41859 , RMS_ARCSEC = 1.3889484
EXPID = 41871 , RMS_ARCSEC = 1.3919272
EXPID = 41883 , RMS_ARCSEC = 1.4293239
EXPID = 41895 , RMS_ARCSEC = 1.4196646
EXPID = 41898 , RMS_ARCSEC = 1.4414393
EXPID = 41900 , RMS_ARCSEC = 1.2369366
EXPID = 41902 , RMS_ARCSEC = 1.4176247
EXPID = 41904 , RMS_ARCSEC = 1.3151747

My GFA reduction outputs for all of these exposures (in both of the above lists) indicate successful WCS calibrations for every single guide camera (CONTRAST > 2), so I don't think that's the problem.

Comparison of fiber to target offsets predicted by desimeter from the FVC images and field model with the measured offsets from the dither analysis.

Currently, get_posmov_fvc_data matches each pos move db entry to FVC coordinates of positioner, for each positioner of each petal. It does the matching via timestamps. It then outputs a CSV file per positioner, containing those results + a bunch of other columns from the posmovedb.

Coming soon:

• Saving EXPOSURE_ID and EXPOSURE_ITER to the database. Should simplify matching.
• Saving OBS_X, OBS_Y, PTL_X, PTL_Y, PTL_Z to the database. These are from platemaker and from postransforms at the time of measurement.
• I want to:
  • Shift all calibrations and xytest analysis out of PECS and into desimeter.
  • Use desimeter or platemaker results interchangeably.
  • Provide tools in desimeter that can do the analysis "instantly" for operators on the mountain.

To handle this, I suggest that we:

1. Include the new columns in get_posmove_fvc_data.
2. Factor out the part where it grabs data from posmovedb.
3. Factor out the part where it saves a particular CSV file format.
4. Make a new (similar) script for grabbing data without having to run desimeter.

So both get_posmov_fvc_data and the new script---let's call it get_posmov_data---can use (2) and (3) as well.

Does that sound like a good approach? Or would it be easier to just extend get_posmove_fvc_data with more options?

I get only the right half of the focal plane matched when running

desi_fvc_proc --zbfit -i fvc-00052009.fits.fz --extname F0002 --field-model fm-00052009-20200730.json --expected coordinates-00052009.fits -o fvc-00052009-20200817.csv

with the new desiconda install, loaded with,

module use /global/common/software/desi/${NERSC_HOST}/desiconda/20200801-1.4.0-spec/modulefiles
module load desiconda

It works fine with the old desiconda install

module use /global/common/software/desi/${NERSC_HOST}/desiconda/20190804-1.3.0-spec/modulefiles
module load desiconda

So it's either a different interpretation of the code between different python versions, or some bug with an I/O module.

OFFSET_T,P are calibration parameters defined by the relation EXT_T,P = INT_T,P + OFFSET_T,P , where INT_T,P are the "intrinsic" angles in the frame of the positioners ; and EXT_T,P represent angles in the frame of the petal. The issue is that they serve 2 different purposes in the current calibration code:

1. OFFSET_T,P define the angles of the hard stops in the petal frame because the rehoming procedure attributes fixed INT_T,P angles for the hard stops of the positioners.
2. OFFSET_T,P are used in a calibration fit as zero points, or constant offsets, between the "measured" angles and the "expected" angles INT_T,P. As such, they compensate for any drift between the true angles of the positioners and the parameters INT_T,P. Indeed INT_T,P are not measurements, they are values initialized after rehoming the positioners, and then incremented in the code for each move, with their values saved in the DB. The problem is that the INT_T,P values do not perfectly track the true angles, errors get accumulated, and the values of INT_T,P have to be recalibrated from time to time.

So the calibration fit should always fit for offsets, but distinguish whether to attribute this measured offset to a measurement of the actual angle of the hard stops (i.e. OFFSET_T,P), or a correction to INT_T,P.
The former case should be considered only when the calibration includes a rehoming of the positioners, AND if it is expected that the drifts of INT_T,P would be negligible (in particular for a non-linear response, the situation is more complicated).

Applying corrections to OFFSET_T,P in a calibration sequence that does not include rehoming, or when many moves have been performed since rehoming so would lead to incorrect angles for the hard stops.

Using same example as #19:

basedir=/project/projectdirs/desi/spectro/data/20200107/00038674
desi_fvc_proc \
  --infile $basedir/fvc-00038674.fits.fz \
  --expected $basedir/coordinates-00038674.fits \
  --outfile spots-00038674.csv

there are several positioners that are matched to more than one spot, sometimes very far away, e.g.:

from astropy.table import Table
from collections import Counter
x = Table.read('spots-00038674.csv')
for (petal, loc, pinhole), n in Counter(zip(x['PETAL_LOC'], x['DEVICE_LOC'], x['PINHOLE_ID'])).items(): 
    if n>1: 
        print(petal, loc, pinhole, n) 

0 533 0 4
0 509 0 2
0 409 0 2
0 224 0 2
0 205 0 2
0 157 0 2
8 350 0 2
8 248 0 2
7 24 0 2
6 159 0 2
3 494 0 2
7 377 0 2
4 134 0 2
7 443 0 2
7 397 0 2
6 437 0 2

The first case (PETAL_LOC=0 DEVICE_LOC=533) is the fiducial reported in #19, but the others are positioners, e.g.

x['XPIX', 'YPIX', 'DEVICE_LOC', 'PINHOLE_ID'][x['LOCATION'] == 509] 

<Table length=2>
       XPIX               YPIX       DEVICE_LOC PINHOLE_ID
     float64            float64        int64      int64   
------------------ ----------------- ---------- ----------
2667.6594170561475 540.0779076216791        509          0
 2695.654126737584 565.4560307807191        509          0

While writing unit tests for PR #7, I found that findfiducials is missing some pinholes that it previously found:

from pkg_resources import resource_filename
import numpy as np
from astropy.table import Table
from desimeter.findfiducials import findfiducials

testspotfile = resource_filename('desimeter', 'test/data/test-spots.csv')
origspots = Table.read(testspotfile)

spots = origspots.copy()
spots.remove_columns(['LOCATION', 'PINHOLE_ID', 'X_FP', 'Y_FP',
                              'X_FP_METRO', 'Y_FP_METRO'])

spots = findfiducials(spots)
test_pinholes = list(zip(origspots['LOCATION'], origspots['PINHOLE_ID']))
found_pinholes = list(zip(spots['LOCATION'], spots['PINHOLE_ID']))

missing = set(test_pinholes) - set(found_pinholes)
print('Missing pinholes: {}'.format(missing))

Results in

Missing pinholes: {(4321, 2), (8011, 4), (4011, 4), (1011, 4), (9011, 4), (4150, 4), (3011, 4)}

Note that DEVICE_LOC=11 is the most problematic, as is PINHOLE_ID=4 (central pinhole).

Produce readme for the workflow from fvc images and posmovedb data --> positioner parameter best-fits and pass/fail histories.

C.f. comments linked below:
#63 (comment)

When calling ptl2fp, with an argument of zptl=None, a dummy vector is generated for ptlz full of zeros:

But this does not match the curved focal surface. I would have thought we need to instead:

1. Calculate r_ptl = np.hypot(xptl, yptl).
2. Estimate z_ptl using a function like R2Z_lookup() in posconstants.py.
   (https://desi.lbl.gov/trac/browser/code/focalplane/plate_control/trunk/petal/posconstants.py#L66)

Easy to add this r2z (and z2r) to the desimeter transforms package if desired.

I might not be understanding some subtlety of how the "ptl" coordinates are meant to be interpreted.

Add transforms between sky RA,dec and focal plane x,y using GFA WCS solutions to tie down the telescope pointing, hexapod offset/rotation, and atmospheric effects.

get_posmoves fails for me. See console output copied below. I believe the cause is:

• using date-min arg
• turned on with-calib flag
• but no calibration rows available since that date

[msdos@beyonce desimeter-master]$ /software/products/desimeter-master/bin/get_posmoves --host XXXX --port XXXX --password XXXX --petal-ids 1 --date-min 2020-08-06 --with-calib -o ~/jhsilber/posmovedata
select * from posmovedb.positioner_calibration_p1 where pos_id='M08275' order by time_recorded
Traceback (most recent call last):
  File "/software/products/desimeter-master/bin/get_posmoves", line 99, in <module>
    j=np.where(tstamp_calib<t)[0][-1]
IndexError: index -1 is out of bounds for axis 0 with size 0
[msdos@beyonce desimeter-master]$ python -m pdb /software/products/desimeter-master/bin/get_posmoves --host XXXX --port XXXX --password XXXX --petal-ids 1 --date-min 2020-08-06 --with-calib -o ~/jhsilber/posmovedata
> /software/products/desimeter-master/bin/get_posmoves(3)<module>()
-> import os,sys
(Pdb) b 99
Breakpoint 1 at /software/products/desimeter-master/bin/get_posmoves:99
(Pdb) l
  1     #!/usr/bin/env python
  2
  3  -> import os,sys
  4     import psycopg2
  5     import numpy as np
  6     import matplotlib.pyplot as plt
  7     from astropy.table import Table
  8     import datetime
  9     import argparse
 10
 11     from desimeter.util import parse_fibers
(Pdb) c
select * from posmovedb.positioner_calibration_p1 where pos_id='M08275' order by time_recorded
> /software/products/desimeter-master/bin/get_posmoves(99)<module>()
-> j=np.where(tstamp_calib<t)[0][-1]
(Pdb) l
 94
 95                 for k in new_keys :
 96                     posmoves[k]=list()
 97
 98                 for t in tstamp_posmovedb :
 99 B->                 j=np.where(tstamp_calib<t)[0][-1]
100                     for k1,k1 in zip(new_keys,calib.keys) :
101                         posmoves[k1].append(calib[k2][j])
102
103             # save, using astropy.Table io
104
(Pdb) t
1585169115.612538
(Pdb) tstamp_calib
array([1.58516912e+09, 1.58688938e+09, 1.58689045e+09, 1.58692164e+09,
       1.58692168e+09, 1.58692171e+09, 1.58697402e+09, 1.58990817e+09,
       1.58990819e+09, 1.58990820e+09, 1.58990978e+09, 1.58990983e+09,
       1.58990984e+09, 1.58990986e+09, 1.58991030e+09, 1.58993004e+09,
       1.58993006e+09, 1.58993007e+09, 1.58993028e+09, 1.58993196e+09,
       1.58993205e+09, 1.59019717e+09, 1.59019719e+09, 1.59019721e+09,
       1.59051586e+09, 1.59068536e+09, 1.59068539e+09, 1.59068541e+09,
       1.59072267e+09, 1.59258753e+09, 1.59258755e+09, 1.59258757e+09,
       1.59355202e+09, 1.59371648e+09, 1.59413956e+09, 1.59415106e+09,
       1.59415182e+09, 1.59424257e+09])
(Pdb) np.where(tstamp_calib<t)
(array([], dtype=int64),)
(Pdb) np.where(tstamp_calib>t)
(array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
       35, 36, 37]),)

New focal plane metrology (petal alignments) provided by Steve is needed for offline FP calibration analysis.

json copied to text below with updated information:
{
"type": "group",
"name": "focal_plane_metrology",
"version": 200,
"elements": [

        {   
            "name":         "0",
            "version":      76298,
            "constants":    {
"petal_id":4,
"petal_offset_x":0.1224,
"petal_offset_y":0.0528,
"petal_offset_z":0.0,
"petal_rot_1":0.0,
"petal_rot_2":0.0,
"petal_rot_3":-1.885369086

}
},

        {   
            "name":         "1",
            "version":      76299,
            "constants":    {
"petal_id":5,
"petal_offset_x":0.0105,
"petal_offset_y":0.0005,
"petal_offset_z":0.0,
"petal_rot_1":0.0,
"petal_rot_2":0.0,
"petal_rot_3":-1.256580211

}
},

        {   
            "name":         "2",
            "version":      76300,
            "constants":    {
"petal_id":6,
"petal_offset_x":-0.0265,
"petal_offset_y":0.0415,
"petal_offset_z":0.0,
"petal_rot_1":0.0,
"petal_rot_2":0.0,
"petal_rot_3":-0.628353415

}
},

        {   
            "name":         "3",
            "version":      76301,
            "constants":    {
"petal_id":3,
"petal_offset_x":-0.048,
"petal_offset_y":-0.0299,
"petal_offset_z":0.0,
"petal_rot_1":0.0,
"petal_rot_2":0.0,
"petal_rot_3":0.000150777

}
},

        {   
            "name":         "4",
            "version":      76302,
            "constants":    {
"petal_id":8,
"petal_offset_x":-0.0196,
"petal_offset_y":0.0459,
"petal_offset_z":0.0,
"petal_rot_1":0.0,
"petal_rot_2":0.0,
"petal_rot_3":0.628496576

}
},

        {   
            "name":         "5",
            "version":      76303,
            "constants":    {
"petal_id":10,
"petal_offset_x":-0.0794,
"petal_offset_y":-0.0113,
"petal_offset_z":0.0,
"petal_rot_1":0.0,
"petal_rot_2":0.0,
"petal_rot_3":1.256632524

}
},

        {   
            "name":         "6",
            "version":      76304,
            "constants":    {
"petal_id":11,
"petal_offset_x":-0.0412,
"petal_offset_y":0.003,
"petal_offset_z":0.0,
"petal_rot_1":0.0,
"petal_rot_2":0.0,
"petal_rot_3":1.885139556

}
},

        {   
            "name":         "7",
            "version":      76305,
            "constants":    {
"petal_id":2,
"petal_offset_x":0.0904,
"petal_offset_y":-0.102,
"petal_offset_z":0.0,
"petal_rot_1":0.0,
"petal_rot_2":0.0,
"petal_rot_3":2.51321797

}
},

        {   
            "name":         "8",
            "version":      76306,
            "constants":    {
"petal_id":7,
"petal_offset_x":0.0056,
"petal_offset_y":-0.0267,
"petal_offset_z":0.0,
"petal_rot_1":0.0,
"petal_rot_2":0.0,
"petal_rot_3":3.14148172

}
},

        {   
            "name":         "9",
            "version":      76307,
            "constants":    {
"petal_id":9,
"petal_offset_x":-0.014,
"petal_offset_y":0.0264,
"petal_offset_z":0.0,
"petal_rot_1":0.0,
"petal_rot_2":0.0,
"petal_rot_3":3.769961549

}
}

]

}

desimeter is already used for the focal plane calibration. The interface is in DOSlib.proxies.FVC. But the interface for the fiber positioning for sky observations has to be developed.

Need to read HEXAPOD rotation angle in guide tables, not in the primary header because it can be different after the acquisition exposure.

That's the case for instance for exposure 20200206/00047450 .

fit_posparams currently calculates best-fits for all positioners, then saves 3 monolithic results files ("static", "dynamic", and "merged"). This has large memory overhead, and means that all processed data up to that point is lost in event of a crash or error.

1. Save a results file per positioner, as they are processed.
2. Remove the redundant triple tables --- get all data into one table per positioner.
3. Update plotting script to read in the individual tables rather than monolith

Add transforms between sky RA,dec and focal plane x,y based upon an optics model and instrument configuration, i.e. a blind prediction without using GFA astrometry, using hexapod settings, dateobs, RA, dec (and from which precession, airmass, and hour-angle can be derived). More sophisticated versions might use air temperature, air pressure, and focal plane temperature.

There is a residual scale factor between focal plane and sky coordinates that vary from exposure to exposure. It is likely part of this variation could be predicted as a function of the telescope pointing (HA,Dec).

When running analyze_bright_image, I get the following error in fvc2fp.py:

Traceback (most recent call last):
  File "/software/products/desimeter-master/bin/analyze_bright_image", line 62, in <module>
    print(transfo.tojson())
  File "/software/products/desimeter-master/py/desimeter/transform/fvc2fp.py", line 91, in tojson
    params['meandistance'] = self.meandistance
AttributeError: 'FVC2FP' object has no attribute 'meandistance'

Command to produce this is:
analyze_bright_image --back-illuminated back1.fits --front-illuminated front1.fits -o pos1.csv

FITS files to run this example are at:
front1.fits
back1.fits

Study why desi_fvc_proc failed on 43046 or 43047 at KPNO:
INFO:findfiducials.py:279:findfiducials:matched 0 pinholes from 0 fiducials
metrology not matched.

Currently desimeter.transform.gfa2fp incorrectly treats the GFAs as being parallel to the xy plane, instead of being flat in 3D space that is more approximately aligned with the curved focal surface. This results in ~30 um inconsistencies in the transforms.

Fix that.

(Copying this over from #192 so we don't forget)

One CI test is failing on python 3.8 but passing on 3.6 and 3.7:

FAIL: test_tan2radec (desimeter.test.test_radec2tan.TestRADEC2TAN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/circleci/project/py/desimeter/test/test_radec2tan.py", line 139, in test_tan2radec
    assert(np.abs(ddec*3600.)<0.001) # 1 milli-arcsec

Handle proper motions in desi_fit_guide_stars

posparams/plotter.bin_errors() generates a dict classifying positioners by best-fit error over time. For example of how to call load up data and send it to bin_errors(), see plot_posparams script.

Alternately, if you've already run plot_posparams, can just load up the .json file it outputs (which is exactly this result from bin_errors()).

Use this binned data to output some useful reports. In particular:

1. Most recent pass/fail status for each robot, by error threshold.
2. Which robots improve (error diminishes) when binned in "dynamic" mode.

For item (2), this is saying when SCALE_T and SCALE_P are allowed to float. This would indicate likelihood of a potentially usable "linear" type error. To do this classification, need to distinguish "big" changes in SCALE_T or SCALE_P --- a real effect --- from "small" --- just a slightly better fit.

test_findfidicials is now failing, starting with the matching changes in PR #40 (but alas I didn't notice before merging).

The test uses a spots file generated by desimeter on FVC data with the Canon lens, but updating the test to use py/desimeter/data/canon-lens-fvc2fp.json as the input transform didn't work, which surprises/concerns me.

PR #41 includes updating the test to use canon-lens-fvc2fp.json and flags the test as an expected failure, so turn that off when working on this issue.

Ideally the test would confirm that we can reliably find fiducials on both canon-lens and new singlet-lens FVC data.

Add support for new FVC singlet lens, using fiducial-only data in /global/cfs/cdirs/desi/cmx/fvc/newlens-20200204/ as a starting point.

At minimum desimeter/data/default-fvc2fp.json needs to be updated. Better would be to devise a date-versioned transform such that older FVC data can still be processed transparently, while also supporting a different default transform for new FVC data.

Bonus: I think the original bootstrapping procedure used a platemaker-derived reference file; it would be preferable for desimeter to have its own independent bootstrap method.

We need a thermal-mechanical model. To give a sense of scale, metrology is generally done at ~22C (I need to verify from actual reports), whereas at Mayall the focal plane is nominally operated at 11C. It is mostly aluminum (~24 ppm/C), hence metrology data will be too large by a factor 2.64e-4, or 214 um across the 812 mm diameter.

I believe this needs to occur prior to performing the fit, e.g. sometime before this line:

Thermal model should be done in a separate module, not spaghettied into zb or poly2d or the like. Also, note the duplicated metrology-loading code in the fit functions. See issue #26.

One option is we move the metrology loading code directly into the base class. And immediately after loading the metrology, apply the thermal model. I'm still wrapping my head around how you guys want your code structured, and where you think interfaces should lie. But anyway, probably better not to bury metrology data prep in specific implementations of fit().

Perhaps best would to make a clean module called like "metrology.py", and it does all the loading and prep of that data, including thermal model etc.

The thermal model may be simple (single fixed temperature at 11C, lacking any real knowledge) or more complex (looking up current temperature sensors---there are many---at the focal plane, either from the online DB, from archived DB, or real-time requests to the hardware. Or even by analyzing FVC image dot separations, if we do a physical test program to correlate them with FP temperature).

Also note that the GIFs will deform somewhat differently physically from everything else, since each pair of GIFs is mounted to a titanium plate on the GFA (which in turn is retrained by an aluminum housing). It's ~10 um effect at our operating temperature. See DESI-0998.

Use the fitted positioner theta axis coordinates from a series of arc calibration FVC images to refit the petal offsets and rotation AND adjust the coordinates of fiducials because there is evidence that they have systematic errors of >=10 microns.