Skip alignment when bestSolutionID in AstromDB is 'FIT_SVM_catalog' about drizzlepac

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

The first task is to test the assertion the best solution obtained from the AstrometryDB is applied as the primary WCS.  (Use a fresh ipppssoot dataset - process the RAW files yourself so there is no WCS in the FLC/FLT headers).  Ideally, the best solution in the dataset is a FIT_SVM solution so you can see if the runastrodriz tries to override with a FIT_REL solution.

The second task in the code which needs to be verified is if an alignment to GAIA is actually performed if the ASTROMETRY_COMPUTE_APOSTERIORI environment variable is set to ON or the "-g" option is set on the command line to runastrodriz.  A dataset which has a WCS of FIT_SVM_"NotGAIAeCR3Catalog" should be used for this test.  The issue here is there should be a way to override the specification of this ticket in the case where the "best" solution is FIT_SVM_NotGAIACatalog.  It is possible this would be the case. 

The software team must discuss how to proceed if it is found the current overrides do not work. Note: if an override does work, but the new WCS is not found to be as good as a FIT_SVM, we probably want an escape dump the less good solution.

Note the force_alignment parameter to verify_guiding in runastrodriz.py is not actually used in the routine.

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

A run-down of the current WCS solutions for an example dataset: j8vo01jyq_flc.fits.

 

The primary WCS from a fresh download from the archive is: IDC_4bb1536oj-FIT_REL_GAIAeDR3

 

Processing the flc through the regular pipeline results in the same WCS.

 

Running fresh j8vo01jyq_flc.fits and j8vo01jzq_flc.fits through the SVM pipeline yields a WCS of: IDC_4bb1536oj-FIT_SVM_GAIAeDR3

 

Using the updateObs function from STWCS loads the following WCS options:

OPUS, IDC_0461802ej, OPUS-GSC240, IDC_0461802ej-GSC240, OPUS-HSC30, IDC_0461802ej-HSC30, IDC_0461802ej-FIT_REL_GAIADR2, j8vo01jyq_flt_IDC_0461802ej-HSC30-hlet.fits, IDC_0461802ej-FIT_SVM_GAIADR2

 

The following are added as headerlets extensions within the flc file (OPUS, IDC_4bb1536oj, IDC_4bb1536oj-GSC240, IDC_4bb1536oj-FIT_REL_GAIAeDR3), however, the primary WCS listed in the extension 1 WCSNAME is IDC_4bb1536oj-GSC240. 

 

While there exists a better a posteriori WCS solution, it seems to choose the an a priori solution. It also  looks like an SVM WCS is recognized by the astrometry database, but is not added as a headerlet.

ADDED NOTE: It looks like WCS are not added from the astrometry database because they have different IDCTAB values (0461802ej) than the values in the header for the file (4bb1536oj). I will try to look at a file that does not have this discrepancy. 

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Steve Goldman Please recall that the FIT_SVM WCS solutions were not added to the astrometry database due do a bug in software (CAOM or Mast catalog?) which did not trigger to ingest the HAP (e.g., FIT_SVM_*) headerlets to update the database.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

I have some questions regarding your investigation this far.

The primary WCS from a fresh download from the archive is: IDC_4bb1536oj-FIT_REL_GAIAeDR3```
It the above is just an FYI for an ipppssoot or SVM FLC/FLT or DRC/DRZ?

 

Processing the flc through the regular pipeline results in the same WCS.```
By "processing" do you mean "runastrodriz"? By processing an ipppssoot_raw.fits using "calacs.e" and then with "runastrodriz", you would have a freshly processed result with internal headerlets from the solutions in the astrometry database.

 

Running fresh j8vo01jyq_flc.fits and j8vo01jzq_flc.fits through the SVM pipeline yields a WCS of: IDC_4bb1536oj-FIT_SVM_GAIAeDR3```
OK.  This is an upgrade from the runastrodriz processing.

 

Using the updateObs function from STWCS loads the following WCS options:
OPUS, IDC_0461802ej, OPUS-GSC240, IDC_0461802ej-GSC240, OPUS-HSC30, IDC_0461802ej-HSC30, IDC_0461802ej-FIT_REL_GAIADR2, j8vo01jyq_flt_IDC_0461802ej-HSC30-hlet.fits, IDC_0461802ej-FIT_SVM_GAIADR2```
Using "updatewcs" loads up the existing WCS database solutions for this image.

 

The following are added as headerlets extensions within the flc file (OPUS, IDC_4bb1536oj, IDC_4bb1536oj-GSC240, IDC_4bb1536oj-FIT_REL_GAIAeDR3), however, the primary WCS listed in the extension 1 WCSNAME is IDC_4bb1536oj-GSC240. ```
Did you check the file before and after use of updatewcs to make sure exactly what was happening?  Part of this exercise is to make sure the code is working as expected.
 

While there exists a better a posteriori WCS solution, it seems to choose the an a priori solution. It also  looks like an SVM WCS is recognized by the astrometry database, but is not added as a headerlet.```
Two items here.  I already mentioned the first in that the most of the FIT_SVM solutions were not properly ingested into the astrometry database due to a software bug outside of our control.  However, that may not be the case with your dataset.  The second item is the version of the IDCTAB calibration reference file.  The headerlet extensions are based on the IDC_4bb1536oj IDCTAB.  The astrometry database should have loaded the IDC_0461802ej solutions also into headerlets.

Sorry, but I cannot tell the exact sequencing of the solutions in the original files processed.  This is why to be very clear, it would be good to start with "calxxx.e" and check the output FLT/FLC file.  Then execute "runastrodriz" and check the HDU of the SCI as well as the headerlet extensions of the FLT/FLC file. You can also see the WCS solutions read from the astrometry database in the *trl.txt file.  These should match what you see in the FLT/FLC file and will either be in the SCI header or a headerlet depending upon the IDCTAB. 

All of these steps are to make sure it is understood what the software is doing now – before a change is made to ensure the "best" WCS solution is never overwritten in the file header.  The "best"solution template (just a suggestion) of enumerated strings or possibly regular expressions.

Should we talk about this ticket?

[stscijgbot-hstdp]

Comment by Jennifer Mack on JIRA:

Hi!  I'm very interested in this discussion. Can you include me if you decide to meet? Varun and I have been doing some testing and can share those results.  My schedule is pretty open except for 11am Wed.

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

There were some errors in the previous comments, and so we can disregard them. After discussing things with Michele it has become clear that the important thing to test here is whether an "SVM. . ." WCS is overwritten during runsinglehap SVM processing. While the name of the WCS solution in the header will be the same, the numbers of the actual solution may change. This is because the SVM WCS solutions are produced with the same name, even if they were produced with changes to various things (e.g. drizzlepac bug fixes).

It is also important to start with data created by HSTCAL in order to trigger new WCS solutions to be created with the correct IDCTAB which can also be specified in this step.

We should have a discussion about using indices after the SVM WCS solutions in order to avoid confusion between different versions; it would be useful to include warren in those discussions.   We should also at some point review the current documentation and update it if necessary.

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

This ticket will be on hold until we can create a plan for implementing the proposed enhancement. We will need to first determine exactly how the best WCS is determined, figure out how that information is passed to drizzlepac (this may need to include passing the best_solution_id from the astrometry database in the future), and then confirmation that the WCS solutions are updated. A hurdle to this work is the recent CAOM issues in ingesting SVM WCS solutions. Another hurdle is ensuring WCS solutions are treated as unique within drizzlepac. A more detailed run-down of the steps involved with this work will be circulated and upload in a separate page or comment. 

[stscijgbot-hstdp]

Comment by Jennifer Mack on JIRA:

Astrometry Database Documentation from Brian:

https://innerspace.stsci.edu/display/CSB/AstrometryDB+Documentationhttps://innerspace.stsci.edu/display/CSB/AstrometryDB+Documentation

 

Current plan: 
Matt Burger will push all HLET & FL?_*_HLET files to the astrometry database with consulation from Brian
Talya will test that it worked.
Brian will give a thumbs up when it looks good on his end.
Brian will update logic to set the bestSolutionID in the database to use the latest SVM solution.

 

Regarding catalog fitting:
   - We will keep the eDR3 solutions (since DR3 is identical) and update the WCS for DR4. 
   - HSCv4.0 is coming soon and can be added added to the logic for catalog priority (currently eDR3 > GSC242 > 2MASS)