spacetelescope / drizzlepac

AstroDrizzle for HST images.

Drizzlepac

Nightly regression test results are available only from within the STScI network at this time. https://plwishmaster.stsci.edu:8081/job/RT/job/drizzlepac/

The use of this software on HST data is described at:

http://drizzlepac.stsci.edu/

A complete description of the documented interfaces in the code itself can be found at:

http://drizzlepac.readthedocs.io

Installation

Conda (Recommended)

Drizzlepac is installed when you install the stenv conda environment (a replacement for astroconda). Select your desired release and follow the instructions on the stenv installation page.

Install with pip

$ pip install git+https://github.com/spacetelescope/drizzlepac.git

The option --no-use-pep517 MAY be required in order to correctly build the C extensions with pip versions up to 22.2, after commenting out the build-backend from the pyproject.toml config file.

Support for installing using pip is still evolving, so use of this command is provided on an experimental basis for now.

From Source

Clone this repository

$ git clone https://github.com/spacetelescope/drizzlepac
$ cd drizzlepac

Build the documentation

Note: If you intend to use drizzlepac's embedded help feature from within an interactive python or ipython session, we recommend you do not skip this step.

$ cd doc/
$ make html

Install DrizzlePac

$ python setup.py install

drizzlepac's Issues

Path to directory changed to lower case

Using drizzlepac version 2.1.17, has anyone else noticed some weird things happening when writing the blot image? It seems that the path to the file gets lower-cased. In the output below darkTesting got changed to darktesting. Did a .lower() get called in somewhere by mistake?

reference sky value for image  /grp/hst/acs7/dborncamp/darkTesting/ff/jcqtb9w7q_flc.fits  is  25.5566139221

reference sky value for image  /grp/hst/acs7/dborncamp/darkTesting/ff/jcqtb9weq_flc.fits  is  25.5818347931



Saving output median image to:  abells1063F435_med.fits

==== Processing Step  Create Median  finished at  14:57:25.748 (15/08/2017)

==== Processing Step  Blot  started at  14:57:25.757 (15/08/2017)

    Blot: creating blotted image:  /grp/hst/acs7/dborncamp/darkTesting/ff/jcqtb1rkq_flc.fits[sci,1]

Using default C-based coordinate transformation...

-Generating simple FITS output: /grp/hst/acs7/dborncamp/darktesting/ff/jcqtb1rkq_sci1_blt.fits

Writing out image to disk: /grp/hst/acs7/dborncamp/darktesting/ff/jcqtb1rkq_sci1_blt.fits

Error when aligning FITS files to Ref Catalog

ESO_Guide_Star_Coords_J2000.txt

I have 4 FITS files that I am trying to align to a reference catalog. After tweakreg has found the sources in each image and moves its focus to the reference catalog, I get an error in drizzlepac/catalogs.py which says: TypeError: 'NoneType' object has no attribute 'getitem'. This error occurs on line 263. I did nothing in regards to setting the flux limits so I do not know why the program entered the 'if' statement to begin with. I've attached the reference catalog I am using.

Error message has bad syntax

A user was trying to run some WFPC2 images through astrodrizzle and got the message below. They complained that the syntax in the error message was wrong (missing quotes in the updatewcs call). The message should also be updated to use astropy.io.fits instead of iraf tasks. Also, typing 'c' gives you a message that 'c' is not defined.

A 'DGEOFILE' keyword is present in the primary header but 'NPOLFILE' keyword was not found.
            This version of the software uses a new format for the residual distortion DGEO files.
            Please consult the instrument web pages for which reference files to download.
            A small (new style) dgeofile is needed ('_npl.fits' extension) and possibly a
            detector to image correction file ('_d2i.fits' extension).
            The names of these files must be added to the primary header either using the task XXXX
            or manually, for example:

            hedit u33z0401t_c0m.fits[0] npolfile fname_npl.fits add+
            hedit u33z0401t_c0m.fits[0] d2imfile fname_d2i.fits add+

            where fname_npl.fits is the name of the new style dgeo file and fname_d2i.fits is
            the name of the detector to image correction. After adding these keywords to the
            primary header, updatewcs must be run to update the science files:

            from stwcs import updatewcs
            updatewcs.updatewcs(u33z0401t_c0m.fits)

            Alternatively you may choose to run astrodrizzle without DGEO and detector to image correction.

            To stop astrodrizzle and update the dgeo files, type 'q'.
            To continue running astrodrizzle without the non-polynomial distortion correction, type 'c':

Source filtering in Tweakreg incorrectly interprets 'cps' units

fluxuntis parameter in tweakreg is used to indicate units of input catalog fluxes. When fluxunits is 'mag', code assume the values are negative and so inverts comparisons used for filtering (compared to the case when fluxunits is the default 'counts'). However, the logic in the code does not take into account another supported unit: 'cps'. This is because the code checks whether fluxunits is 'counts' and if not - it assumes it is 'mag'. Therefore, if user supplies a catalog with fluxes with 'cps' units, code assumes the catalog's fluxes are magnitudes.

The offending block of code can be found here: https://github.com/spacetelescope/drizzlepac/blob/master/lib/drizzlepac/catalogs.py#L306:

            # apply flux limit clipping
            if limits['fluxunits'] == 'counts':
                minindx = self.radec_full[2] >= fluxmin
                maxindx = self.radec_full[2] <= fluxmax
            else:
                minindx = self.radec_full[2] <= fluxmin
                maxindx = self.radec_full[2] >= fluxmax

The problem can be fixed by checking if flux units are in 'mag' and assume they are either 'counts' or 'cps' if not in 'mag'.

Tweakreg doubles the number of sources in the reference catalog when expand_refcat is True

CC: @stsci-hack This is a second bug (for the first bug see #86) reported by Jennifer Mack related to the use of expand_refcat option in tweakreg. This bug causes tweakreg to crash when expand_refcat is True and regardless of enforce_user_order value (before #86 it was crashing when enforce_user_order was True).

The reported error is:

IndexError: boolean index did not match indexed array along dimension 0;
dimension is 4096 but corresponding boolean dimension is 2048

Essentially, code attempts to select some elements of a vector using a boolean array of incorrect length.

My detective work has uncovered that this bug was introduced in r45805 - see imgclasses.py line 379. Alternatively, see GitHub link 859cb00#diff-62a80b7810f430706a201f2fce2070fdR379

Essentially the code (before r45805) was:

if len(xycat) > 2:
    fluxlist.append(xycat[2])
    idlist.append(xycat[3])
else:
    fluxlist.append([999.0]*len(skycat[0]))
    idlist.append(np.arange(len(skycat[0])))

and after r45805 it became

if len(xycat) > 2:
    fluxlist.append(xycat[2])
    idlist.append(xycat[3])
if len(skycat) > 2:
    fluxlist.append(skycat[2])
    idlist.append(skycat[3])
else:
    fluxlist.append([999.0]*len(skycat[0]))
    idlist.append(np.arange(len(skycat[0])))

This second then-block doubles the number flux and ID values added to fluxlist and idlist lists.

@stsci-hack It is not clear why this middle block has been introduced at all. I am inclined to remove it but I would like to get your opinion if this removal could affect something else.

test_lanczos_with_grid failed on nott dev

I accidentally messed this up in fixing #76. My bad! The file of interest was in a "modified" mode and I accidentally committed it. Then I panicked and reverted the commit but only on nott dev. Not sure how reverting that commit would have changed it as my svn-fu is not very strong...

https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=107533042

TweakReg figures not closed? (when interactive mode turned off)

Hi I came across a problem when make a plot after run the TweakReg task:
When plot a new figure, say, a scatter plot using pylab, which is not related to TweakReg, the TweakReg data (vector plot/hist plot) go on top of the new figure. I was wondering is it possible turn the matplotlib settings back to normal at the end of TweakReg task? Many thanks!

example code:

from astropy.io import ascii
import pylab as pl
tweakreg.TweakReg(files=fs_fits, configobj=f_cfg, refimage=f_ref, catfile='mtxy.lst',
                              refcat=self.f_cat, updatehdr=updatehdr, wcsname=wcsname)
xys = ascii.read(self.outdir+'shifts.txt', data_start=1)
pl.errorbar(xys['col2'], xys['col3'], xerr=xys['col6'], yerr=xys['col7'], fmt='o', alpha=0.8)
pl.savefig('test.png')
pl.close()

The test.png was supposed to be scatter plot of the frame shifts, however, it turns out be a vector plot of TweakReg task on top of the scatter plot.

versions:

• TweakReg Version 1.4.3(01-Jul-2016)
• Python Version 2.7.13

Incorrect pixel center used when computing CRPIX

As mentioned in #62, setting CRPIX to NAXIS/2 is not really correct because the "center" is at the center of the pixel and not at its corner. The "center" issue is explained in https://fits.gsfc.nasa.gov/users_guide/users_guide/node52.html For example, if we have an image containing a single pixel, then the index of that pixel is at the center of the pixel and thus NAXIS/2 would give 1/2=0.5 instead of 0 (using Python indexing). Secondly, FITS WCS is not using Python indexing but rather 1-based indexing so that the CRPIX of a single pixel image should be 1 instead of 0. Succinctly, correct (for FITS WCS) formula for a CRPIX at the middle of an image should be: (NAXIS+1)/2. This is also explained in section 2.1.4 of Greisen & Calabretta 2002

CC: @stsci-hack

VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future

Examples:
https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=38489285
https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=38489286

[WARNING ] .../drizzlepac/updatehdr.py:321: VisibleDeprecationWarning:
using a non-integer number instead of an integer will result in an error in the future
  a = inv[0.0]

This is a test

Test issue.

Possible logical/computational error in createMedian.py

CC: @stsci-hack

The code in a couple of IF blocks is never executed and so I wonder if it should be removed or whether this is a mistake and for some "units" the desired code is not executed... See https://github.com/spacetelescope/drizzlepac/blob/master/lib/drizzlepac/createMedian.py#L169:

        if lthresh is not None:
            if proc_units.lower() == 'native':
                if native_units.lower() == "counts":
                    lthresh = lthresh * det_gain
                    if native_units.lower() == "counts/s":
                        lthresh = lthresh * img_exptime
        if hthresh is not None:
            if proc_units.lower() == 'native':
                if native_units.lower() == "counts":
                    hthresh = hthresh * det_gain
                    if native_units.lower() == "counts/s":
                        hthresh = hthresh * img_exptime

Code that follows if native_units.lower() == "counts/s" is never executed because native_units cannot be both "counts" AND "counts/s" at the same time. I wonder what was the intention here and what is the correct thing to do...

Tests failed at "if errorobj" in util.py

See https://ssb.stsci.edu/pandokia/pandokia.cgi?query=treewalk&context=dev%3A3.5.4&host=nott&project=drizzlepac&test_run=conda-conda_tests_py3_2017-10-24-10%3A30%3A01&status=E

Trailer file written to:  astrodrizzle.log

  File "/srv/iraf/bldtmp/pO5n/p/envs/dev/lib/python3.5/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/srv/iraf/bldtmp/pO5n/p/envs/dev/lib/python3.5/unittest/case.py", line 605, in run
    testMethod()
  File "/srv/iraf/bldtmp/pO5n/p/envs/dev/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/srv/rt/betadrizzle/vary_perf/wfc3_binned_asn/inmemory.on_parallel.off/run.py", line 70, in test
    astrodrizzle.AstroDrizzle(input_files, mdriztab=False, configobj=cfg_fname)
  File "/srv/iraf/bldtmp/pO5n/p/envs/dev/lib/python3.5/site-packages/drizzlepac/astrodrizzle.py", line 123, in AstroDrizzle
    run(configObj, wcsmap=wcsmap)
  File "/srv/iraf/bldtmp/pO5n/p/envs/dev/lib/python3.5/site-packages/drizzlepac/util.py", line 228, in wrapper
    if errorobj:
None

AstropyDeprecationWarning: Accessing an HDU after an HDUList is closed

Example test: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=36464238

[WARNING ] /srv/iraf/bldtmp/porcelain.g23AA24Tm8/porcelain/envs/dev/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py:1094:
AstropyDeprecationWarning: Accessing an HDU after an HDUList is closed, where
that HDU was no read while the HDUList was open
is deprecated.  That is, you did something like:

    >>> hdulist.close()
    >>> print(hdulist[2].header)

even though hdulist[2] had not been read yet.  Instead
do:

    >>> print(hdulist[2].header)
    >>> hdulist.close()

or open the file with lazy_load_hdus=False to read all
the HDUs into memory immediately upon opening the file.

  """), AstropyDeprecationWarning)

TweakReg crashing trying to create undistorted ref_wcs from already undistorted image

Tweakreg is crashing with the following Traceback:

  File "/user/vbajaj/anaconda2/envs/astroconda/lib/python2.7/site-packages/drizzlepac/util.py", line 219, in wrapper
    func(*args, **kwargs)
  File "/user/vbajaj/anaconda2/envs/astroconda/lib/python2.7/site-packages/drizzlepac/tweakreg.py", line 373, in run
    cat_origin=cat_src, **kwargs)
  File "/user/vbajaj/anaconda2/envs/astroconda/lib/python2.7/site-packages/drizzlepac/imgclasses.py", line 1120, in __init__
    self.wcs = utils.output_wcs([self.wcs], undistort=True)
  File "/user/vbajaj/anaconda2/envs/astroconda/lib/python2.7/site-packages/stwcs/distortion/utils.py", line 44, in output_wcs
    outwcs = make_orthogonal_cd(ref_wcs)
  File "/user/vbajaj/anaconda2/envs/astroconda/lib/python2.7/site-packages/stwcs/distortion/utils.py", line 126, in make_orthogonal_cd
    scale = (wcs.idcscale) / 3600.  # HST pixel scale provided
TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'

The input image is an undistorted (UVIS drc) image, and the images are being aligned to an already existing reference catalog. From what I can tell, it's crashing because it can't find the value of the idcscale attribute for the reference WCS. Since I am aligning to an external catalog, I believe the reference WCS is just the WCS read from the first image in the input image list. However, the input image list is all drc files, so they are undistorted and therefore have wcs.idcscale = None.

However, this only happens when I run tweakreg on an image that I have already tweaked once before. If I just pick a "fresh" drizzled image and run it, I get no issues. Specifically, I have to already have had to run tweakreg using the fitgeometry='general' to get this error. If tweakreg was previously run using fitgeometry='rscale', this does not happen

I think this is might come from an issue with an issue in drizzlepac/imgclasses.py, specifically line 1114:

                    if self.wcs.instrument == 'DEFAULT':
                        raise ValueError("Distorted non-HST reference images "
                                         "are not supported.")
                    log.warn("\nReference image contains a distorted WCS.\n"
                             "Using the undistorted version of this WCS.\n")
                    self.wcs = utils.output_wcs([self.wcs], undistort=True)

I think the check _is_cd_unitary(wcs) within _is_wcs_distorted is the issue here. I figure the current distortion solution does not fully account for axis-dependent scale and rotation. When TweakReg is run with a high fidelity catalog to align the images (GAIA), xrot, yrot, xscale, and yscale get applied to the CD matrix. My understanding of the linear algebra isn't too great, but I figure this makes the matrix non-unitary, causing TweakReg to detect the image as 'distorted'. Im not sure what the best way forward is, because I guess the images are technically distorted (or the pixels arent square at least?), but I'm not sure if that should really matter in terms of using this WCS as a reference for other alignments.

FutureWarning: comparison to `None` will result in an elementwise object comparison in the future

Example test: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=36464238

[WARNING ] /srv/iraf/bldtmp/porcelain.g23AA24Tm8/porcelain/envs/dev/lib/python2.7/site-packages/drizzlepac/adrizzle.py:699:
FutureWarning: comparison to `None` will result in an elementwise object comparison
in the future.
  here = _outsci==None and _outwht==None and _outctx==None

Example test: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=36464376

[WARNING ] /srv/iraf/bldtmp/porcelain.g23AA24Tm8/porcelain/envs/dev/lib/python2.7/site-packages/drizzlepac/outputimage.py:468:
FutureWarning: comparison to `None` will result in an elementwise object comparison
in the future.
  if self.outweight and whtarr != None:

[WARNING ] /srv/iraf/bldtmp/porcelain.g23AA24Tm8/porcelain/envs/dev/lib/python2.7/site-packages/drizzlepac/util.py:1066:
FutureWarning: comparison to `None` will result in an elementwise object comparison
in the future.
  assert(dataArray != None), "Please supply a data array for createFiles"

Why is drizzlepac running on Travis CI?

I don't see any .travis.yml file and yet...

https://travis-ci.org/spacetelescope/drizzlepac/builds/214776978

https://travis-ci.org/spacetelescope/drizzlepac/builds/273370222

[TEST] ImportError no module named calc_pixmap

While combing through pandokia back-end logs, I saw this and it is not reported in the daily reports because pandokia cannot pick up such things (don't ask me why).

https://ssb.stsci.edu/steuermann/steuermann.cgi?action=log&name=conda_tests_2017-07-30-10:30:02/nott:conda_testing/dev_tests

COMMAND : 'pdknose --pdk --with-doctest --doctest-tests test_pixmap.py' (for file /srv/rt/betadrizzle/unit/test_pixmap.py) 2017-07-30 10:38:23.180210
E
======================================================================
ERROR: Failure: ImportError (No module named calc_pixmap)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/iraf/bldtmp/pGNj/p/envs/dev/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/srv/iraf/bldtmp/pGNj/p/envs/dev/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/srv/iraf/bldtmp/pGNj/p/envs/dev/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/srv/rt/betadrizzle/unit/test_pixmap.py", line 20, in <module>
    from drizzlepac.calc_pixmap import calc_pixmap
ImportError: No module named calc_pixmap

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)
return from wait, status= 1
COMMAND EXIT: exit 0 2017-07-30 10:38:25.586787
pass=0 fail=0 error=0

Tests

Test

pembry public test failure

https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=104227551

Switch from string.uppercase to string.ascii_uppercase

A compatibility problem with Python 3 as well as the solution has been reported by @cmccully (Thanks!): in Python 3 string.uppercase needs to be replaced with string.ascii_uppercase.

AstropyDeprecationWarning: "clobber" was deprecated in version 1.3

Example test: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=36464376

[WARNING ] /srv/iraf/bldtmp/porcelain.g23AA24Tm8/porcelain/envs/dev/lib/python2.7/site-packages/astropy/utils/decorators.py:470:
AstropyDeprecationWarning: "clobber" was deprecated in version 1.3
and will be removed in a future version. Use argument "overwrite" instead.
  AstropyDeprecationWarning)

[EXP] Switching pandokia tests to Python 3

Any insight on why some drizzlepac tests would fail in Python 3? For example, see https://ssb.stsci.edu/pandokia/pandokia.cgi?query=summary&qid=3142

Make new releases

New releases of drizzlepac and its affected dependencies are needed to propagate the fixes of deprecation warnings to our public build on AstroConda. (So, really the process is GitHub releases + AstroConda recipe updates.)

EDIT: Deprecation warning is not as urgent as critical bug fix, so for packages with only fixes to these warnings, the release can wait but should be done before the deprecated codes are removed completely from their dependencies (e.g., Astropy).

p.s. If you maintain PyPi etc, that needs updating too but I don't care about that.

Drizzle question

Hello,

I have a question regarding the drizzlepac software. I have it installed via the astroconda package and would like to use the adrizzle function within drizzlepac to singly drizzle Keck NIRC2 data to correct distortion, where I have distortion correction images. The original drizzle function allowed the user to specify the following three parameters (I've found all the others that I need),

ir.drizzle.coeffs, 
ir.drizzle.xgeoim, and
ir.drizzle.ygeoim 

When I run help for the adrizzle.drizzle task, it states that -
Aside from the input parameters, this step requires:
* valid input images with SCI extensions
* valid distortion coefficients tables
* any optional secondary distortion correction images

But it doesn't specify what keywords to use to assign them when running.

Matching two stand-alone catalogs (without images containing WCS) in tweakreg

say that I have 2 catalogs (no images, just lists of coordinates, ra, dec). Any idea of how to find shift, rot, scale to "astrometrize" cat2 onto cat1?

Source filtering of reference catalog in TweakReg is broken

Source filtering in tweakreg is implemented in the Catalog.apply_flux_limits() function. The code relies on the PAR_PREFIX class constant to figure out the correct parameters that contain user-supplied flux limits. When user does not set flux limits these parameters are not present in the self.pars dictionary and flux limits are not applied.

The problem is that in the child class RefCatalog this PAR_PREFIX constant is set to 'ref' which is incorrect value for reference catalog parameters: they start with 'r' instead of 'ref' except for the refnbright parameter which stars with 'ref'. This leads to the code not finding user-supplied reference catalog limits and thus not filtering it.

Failing vary_perf/wfc3_binned_asn/* tests in dev

Since 2017-09-21 multiple vary_perf/wfc3_binned_asn/* drizzlepac tests started failing in the sky matching/subtraction step with the following error:

ValueError("Text mode 'w' not supported: files must be opened in binary mode",)

I have traced the source of this error to a recent change by @drdavella made in astropy/astropy#6373 - see the new _normalize_fits_mode(): astropy/astropy@8a21da6#diff-6eeb84ceec5958f743415dd8967222b9R83

Somehow this breaks existing code in the drizzlepac. @drdavella Any ideas of how your modifications to astropy's FITS module have changed the way FITS files can be opened?

TypeError: buffer is too small for requested array

Example test: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=36462995

Trailer file written to:  stis_01_1visit

  File "/Users/shared/iraf_conda/bldtmp/porcelain.phEUcEl55g/porcelain/envs/dev/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/Users/shared/iraf_conda/bldtmp/porcelain.phEUcEl55g/porcelain/envs/dev/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/iraf/rt/betadrizzle/stis/stis_01/stis_01.py", line 72, in stis_01
    astrodrizzle.AstroDrizzle('o*flt.fits',runfile=testname,configobj=parObj)
  File "/Users/shared/iraf_conda/bldtmp/porcelain.phEUcEl55g/porcelain/envs/dev/lib/python2.7/site-packages/drizzlepac/astrodrizzle.py", line 123, in AstroDrizzle
    run(configObj, wcsmap=wcsmap)
  File "/Users/shared/iraf_conda/bldtmp/porcelain.phEUcEl55g/porcelain/envs/dev/lib/python2.7/site-packages/drizzlepac/util.py", line 229, in wrapper
    raise errorobj
TypeError('buffer is too small for requested array',)

setup.cfg needs to state proper stsci.tools version

Because #43 introduced non-backward compatible changes from stsci.tools. If people updates drizzlepac without also updating stsci.tools, they will get ImportError.

Astrodrizzle sky subtraction does not work as advertised

The following does not actually work:

"However, if the sky subtraction step is turned off, AstroDrizzle will still use the sky value recorded in the MDRIZSKY keyword when performing single-image drizzling and cosmic ray identification, as it provides the only indication of the background sky level needed for the statistical computations used to identify cosmic rays."

The value stored in MDRIZSKY is always subtracted, independent of whether the sky subtraction is turned on or off.

RuntimeWarning: invalid value encountered

Example test: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=36464242

[WARNING ] /srv/iraf/bldtmp/porcelain.g23AA24Tm8/porcelain/envs/dev/lib/python2.7/site-packages/drizzlepac/minmed.py:220:
RuntimeWarning: invalid value encountered in sqrt
  __rms_file = np.sqrt(__median_file_weighted + __bkgd_file + __readnoise_file)

[WARNING ] /srv/iraf/bldtmp/porcelain.g23AA24Tm8/porcelain/envs/dev/lib/python2.7/site-packages/drizzlepac/minmed.py:247:
RuntimeWarning: invalid value encountered in less
  __minimum_flag_file = np.where(np.less(__minimum_file_weighted,__median_rms_file),1,0)

[WARNING ] /srv/iraf/bldtmp/porcelain.g23AA24Tm8/porcelain/envs/dev/lib/python2.7/site-packages/drizzlepac/minmed.py:301:
RuntimeWarning: invalid value encountered in less
  self.combArrObj = np.where(np.less(__minimum_file_weighted,__median_rms2_file),

Astrodrizzle sets EXPTIME to 1 in single sci images

A user reported - see Footprints ticket no. 10636 - that astrodrizzle sets EXPTIME keyword to 1 in the header of single sci images. This is an important keyword to have in single sci images and in the past astrodrizzle used to write out correct value of EXPTIME. However, staring with r45809 or, alternatively, this github adaptation, astrodrizzle started setting EXPTIME to 1 for unknown to me reasons.

@stsci-hack What was the reason for setting EXPTIME to 1 in r45809 and can we revert to the original code?

VisibleDeprecationWarning: `rank` is deprecated; use the `ndim` attribute or function instead

Example test: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=36464376

[WARNING ] /srv/iraf/bldtmp/porcelain.g23AA24Tm8/porcelain/envs/dev/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2699:
VisibleDeprecationWarning: `rank` is deprecated; use the `ndim` attribute or
function instead. To find the rank of a matrix see `numpy.linalg.matrix_rank`.
  VisibleDeprecationWarning)

IOError: [Errno 24] Too many open files

I am trying to create a mosaic from ~300 FLTs and I keep getting this error. I have done similar mosaics before using older versions of drizzlepac and have not encountered this error. Setting the "in_memory" to True or False does not change the outcome. BTW, the crash happens at the 234th file.

        wht_type = 'IVM'
        output = 'goodsn-F105W-astrodrizzle-v4.3'
        final_refimage = '/astro/clear/cgosmeyer/ref_files/REF/goodsn_3dhst.v4.0.F125W_orig_sci.fits'
        astrodrizzle.AstroDrizzle(root+'_asn.fits', output=output, runfile = 'astrodrizzle.log', updatewcs = False, wcskey = 'TWEAK',
            proc_unit = 'native', coeffs = True, context = False, group = '', build = False, crbit = 4096, stepsize = 10,
            resetbits = 0, num_cores = None, in_memory = False, restore = False, preserve = False, overwrite = False,
            clean = True, static = False, static_sig = 4.0, skysub = True, skywidth = 0., skystat = '', skylower = None,
            skyupper = None, skyclip = 0, skylsigma = 0.0, skyusigma = 0.0, skyuser = 'MDRIZSKY', skyfile = '',
            driz_separate = False, driz_sep_wcs = False, median = False, blot = False, driz_cr = False,
            driz_combine = True, final_wht_type = wht_type, final_kernel = 'square', final_wt_scl = 'exptime',
            final_pixfrac = 0.8, final_fillval = None, final_bits = 576, final_units = 'cps', final_wcs = True,
            driz_sep_bits = 0, final_refimage=final_refimage)
## -- End pasted text --
INPUT_DICT: {'restore': False, 'final_wht_type': 'IVM', 'final_wcs': True, 'final_wt_scl': 'exptime', 'wcskey': 'TWEAK', 'median': False, 'driz_sep_wcs': False, 'skystat': '', 'final_refimage': '/astro/clear/cgosmeyer/ref_files/REF/goodsn_3dhst.v4.0.F125W_orig_sci.fits', 'static': False, 'skywidth': 0.0, 'skyupper': None, 'overwrite': False, 'final_fillval': None, 'crbit': 4096, 'blot': False, 'proc_unit': 'native', 'skyclip': 0, 'skyusigma': 0.0, 'skylower': None, 'final_pixfrac': 0.8, 'build': False, 'input': 'GOODSN-F105W_asn.fits', 'final_units': 'cps', 'preserve': False, 'driz_separate': False, 'clean': True, 'final_kernel': 'square', 'skysub': True, 'stepsize': 10, 'skylsigma': 0.0, 'runfile': 'astrodrizzle.log', 'final_bits': 576, 'in_memory': False, 'group': '', 'skyfile': '', 'resetbits': 0, 'driz_sep_bits': 0, 'driz_cr': False, 'skyuser': 'MDRIZSKY', 'num_cores': None, 'driz_combine': True, 'context': False, 'coeffs': True, 'output': 'goodsn-F105W-astrodrizzle-v4.3', 'static_sig': 4.0}
Setting up logfile :  astrodrizzle.log

AstroDrizzle Version 2.1.8(08-Feb-2017) started at: 14:42:06.991 (03/03/2017)

==== Processing Step  Initialization  started at  14:42:06.992 (03/03/2017)

##############################################################################
#                                                                            #
# ERROR:                                                                     #
# AstroDrizzle Version 2.1.8 encountered a problem!  Processing terminated   #
# at 14:45:18.884 (03/03/2017).                                              #
#                                                                            #
##############################################################################


   --------------------          --------------------
                   Step          Elapsed time
   --------------------          --------------------

         Initialization          0.0000 sec.

   ====================          ====================

                  Total          0.0000 sec.

Trailer file written to:  astrodrizzle.log

---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/drizzlepac/util.pyc in wrapper(*args, **kwargs)
    217             # finally clause is reached.
    218             try:
--> 219                 func(*args, **kwargs)
    220             except Exception as errorobj:
    221                 raise

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/drizzlepac/astrodrizzle.pyc in run(configobj, wcsmap)
    182         procSteps.addStep('Initialization')
    183         imgObjList = None
--> 184         imgObjList, outwcs = processInput.setCommonInput(configobj)
    185         procSteps.endStep('Initialization')
    186

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/drizzlepac/processInput.pyc in setCommonInput(configObj, createOutwcs)
    191                                             group=configObj['group'],
    192                                             undistort=undistort,
--> 193                                             inmemory=virtual)
    194
    195     # Add original file names as "hidden" attributes of imageObject

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/drizzlepac/processInput.pyc in createImageObjectList(files, instrpars, group, undistort, inmemory)
    323     mt_refimg = None
    324     for img in files:
--> 325         image = _getInputImage(img,group=group)
    326         image.setInstrumentParameters(instrpars)
    327         image.compute_wcslin(undistort=undistort)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/drizzlepac/processInput.pyc in _getInputImage(input, group)
    428             from . import wfc3Data
    429             if _detector == 'UVIS': return wfc3Data.WFC3UVISInputImage(input,group=group)
--> 430             if _detector == 'IR': return wfc3Data.WFC3IRInputImage(input,group=group)
    431
    432     except ImportError:

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/drizzlepac/wfc3Data.pyc in __init__(self, filename, group)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/drizzlepac/wfc3Data.pyc in __init__(self, filename, group)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/drizzlepac/imageObject.pyc in __init__(self, filename, group, inmemory)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/stsci.tools-3.4.1.dev0-py2.7.egg/stsci/tools/fileutil.pyc in openImage(filename, mode, memmap, writefits, clobber, fitsname)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/stsci.tools-3.4.1.dev0-py2.7.egg/stsci/tools/fileutil.pyc in isFits(input)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/stsci.tools-3.4.1.dev0-py2.7.egg/stsci/tools/stpyfits.pyc in wrapped_with_stpyfits(*args, **kwargs)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.pyc in fitsopen(name, mode, memmap, save_backup, cache, **kwargs)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.pyc in fromfile(cls, fileobj, mode, memmap, save_backup, cache, **kwargs)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.pyc in _readfrom(cls, fileobj, data, mode, memmap, save_backup, cache, **kwargs)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/astropy/io/fits/file.pyc in __init__(self, fileobj, mode, memmap, clobber, cache)
    148             self._open_fileobj(fileobj, mode, clobber)
    149         elif isinstance(fileobj, string_types):
--> 150             self._open_filename(fileobj, mode, clobber)
    151         else:
    152             self._open_filelike(fileobj, mode, clobber)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/astropy/io/fits/file.pyc in _open_filename(self, filename, mode, clobber)
    477
    478         if os.path.exists(self.name):
--> 479             with fileobj_open(self.name, 'rb') as f:
    480                 magic = f.read(4)
    481         else:

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/astropy/io/fits/util.pyc in fileobj_open(filename, mode)

IOError: [Errno 24] Too many open files: 'ibohbiddq_flt.fits'
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-9-6c9268a119d1> in <module>()
     10     driz_combine = True, final_wht_type = wht_type, final_kernel = 'square', final_wt_scl = 'exptime',
     11     final_pixfrac = 0.8, final_fillval = None, final_bits = 576, final_units = 'cps', final_wcs = True,
---> 12     driz_sep_bits = 0, final_refimage=final_refimage)

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/drizzlepac/astrodrizzle.pyc in AstroDrizzle(input, mdriztab, editpars, configobj, wcsmap, **input_dict)
    121     # already called 'run()'.
    122     if not editpars:
--> 123         run(configObj, wcsmap=wcsmap)
    124
    125 #

/Users/imomcheva/anaconda/envs/iraf27/lib/python2.7/site-packages/drizzlepac/util.pyc in wrapper(*args, **kwargs)
    227                     # (hope that end_logging didn't change the last exception raised)
    228                     if errorobj:
--> 229                         raise errorobj
    230
    231         return wrapper

IOError: [Errno 24] Too many open files: 'ibohbiddq_flt.fits'

In [10]:

TweakReg is ignoring units of user-provided catalogs

TweakReg is ignoring units (xyunits parameter) of user-provided (through the catfile parameter) catalogs. Specifically, user-provided catalogs are assumed to be in "pixels" even when user sets xyunits to "degrees".

Even though the support for catalogs in degrees is documented in TweakReg's documentation, the support for degrees has never been implemented.

This bug was first reported by Will Waldron (UAH).

Fix failing tests on pandokia

There are a bunch of failing tests on all platforms. For example, see https://ssb.stsci.edu/pandokia/pandokia.cgi?query=treewalk&host=%25&project=drizzlepac&test_run=conda-conda_tests_2017-01-24-10%3A30%3A01

VerifyWarning: Card is too long, comment will be truncated

Example test: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=36464239

[WARNING ] VerifyWarning: Card is too long, comment will be truncated.

Multiple issues related to how astrodrizzle uses custom WCS parameters

astrodrizzle computes "output WCS" (either for the single drizzle or for the final drizzle steps) as follows:

1. Find a "default" WCS:
   • If user provides a "reference" WCS image - use that image's WCS;
   • Otherwise, construct the "default" WCS using stwcs.distortion.utils.output_wcs();
2. If user specifies custom WCS parameters (i.e., using final_rot, final_scale, etc. or driz_sep_rot, driz_sep_scale, etc.), astrodrizzle "merges" user-specified WCS parameters with the default WCS found in the previous step. Sometimes this "merging" requires recomputation of some of the parameters of the default WCS.

A recent footprints ticket number 10154 prompted a review of the code in drizzlepac.wcs_functions.mergeWCS() function which revealed the following problems with the code:

1. Specification of a custom WCS parameter such as final_scale together with a "reference image" results in the recomputation of the output image size. Due to rounding errors, output size may be 1 pixel smaller than expected.
2. Every time output image size is computed (essentially, every time user specifies a custom WCS parameter (except for outnx) AND user provides a reference image whose CRPIX is not at the center of the image of the image (NAXIS/2 as used by the code), the computations are simply incorrect: i) image's "footprint" is rotated about NAXIS/2 instead of CRPIX and ii) "merged" WCS' CRPIX is set to outnx,y/2 thus changing the location of the reference point (which is incorrect as this unintentionally shifts the image).

In addition, setting CRPIX to NAXIS/2 is not really correct because the "center" is at the center of the pixel and not at its corner. For example, if we have an image containing a single pixel, then the index of that pixel is at the center of the pixel and thus NAXIS/2 would give 1/2=0.5 instead of 0 (using Python indexing). Secondly, FITS WCS is not using Python indexing but rather 1-based indexing so that the CRPIX of a single pixel image should be 1 instead of 0. Succinctly, correct (for FITS WCS) formula for a CRPIX at the middle of an image should be: (NAXIS+1)/2. This last part probably deserves its own issue (EDIT: see #63).

CC: @stsci-hack

Failed tests: Two tests have unanticipated changed results

https://ssb.stsci.edu/pandokia/pandokia.cgi?query=summary&qid=5749

Remove astrolib.coords dependency

Drizzlepac no longer uses the astrolib.coords package but still requires it during setup:

https://github.com/spacetelescope/drizzlepac/blob/master/setup.cfg#L62

setup.cfg:

requires-dist =
    stsci.tools
    stsci.convolve
    stsci.image
    stsci.imagemanip
    stsci.imagestats
    stsci.ndimage
    stsci.stimage
    nictools
    stwcs (>=1.1.9)
    fitsblender (>=0.2.2)
    astrolib.coords (>=0.39.4)  # <- this
    astropy (>=1.0)
    numpy (>=1.9)

Subtracted sky is not converted to electrons when computing sigma for combine_type "minmed"

Based on my analysis of the code (done when working on #11), it seems that createMedian step does not convert subtracted sky value to electrons (rather it is left in its original units, i.e., e/s, counts/s, counts, e - so sometimes - when input FLT images are already in electrons - the formula for computing sigma works correctly).

See: https://github.com/spacetelescope/drizzlepac/blob/master/lib/drizzlepac/createMedian.py#L265
and
https://github.com/spacetelescope/drizzlepac/blob/master/lib/drizzlepac/minmed.py#L178

Subtracted sky value comes directly from the value of MDRIZSKY in input image headers:
https://github.com/spacetelescope/drizzlepac/blob/master/lib/drizzlepac/sky.py#L354

TEAL interface does not allow setting custom WCS CRPIX

The code for drizzlepac.wcs_functions.mergeWCS() checks if user specified a custom CRPIX value for the output WCS. However, user parameter dictionary will never contain such a parameter when run with TEAL because TEAL UI (and astrodrizzle.cfg) does not contain CRPIX field.

CC: @stsci-hack

AstropyDeprecationWarning: The rotateCD function is deprecated

Example test: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=36464377

[WARNING ] /srv/iraf/bldtmp/porcelain.g23AA24Tm8/porcelain/envs/dev/lib/python2.7/site-packages/drizzlepac/wcs_functions.py:592:
AstropyDeprecationWarning: The rotateCD function is deprecated and
may be removed in a future version.
  outwcs.rotateCD(_delta_rot)

Moving target subarray images do not combine properly

The HLA has identified a use-case where multiple apertures (subarray formats) were used along with full frame exposures to observe a moving target. Not surprisingly, drizzlepac was unable to correctly use the WCS information to correctly combine these exposures. It is not clear what can be done, but this issue will provide a reminder to review this use-case to see what might possibly be done to generalize even further how these sets of exposures should be combined.

The following details came from a help call filed by R. White (HLA):

_Here are a couple of examples. Visit 17 of proposal 11559 has these 10 WFC3/UVIS exposures:

filename aperture ccdchip
ib4v17plq_flc.fits UVIS-QUAD-SUB 2
ib4v17pmq_flc.fits UVIS-QUAD-SUB 2
ib4v17pnq_flc.fits UVIS-QUAD-SUB 1
ib4v17poq_flc.fits UVIS-QUAD-SUB 2
ib4v17ppq_flc.fits UVIS-QUAD-SUB 2
ib4v17pqq_flc.fits UVIS-QUAD-SUB 2
ib4v17psq_flc.fits UVIS-QUAD-SUB 1
ib4v17puq_flc.fits UVIS1-2K4-SUB 1
ib4v17pvq_flc.fits UVIS-QUAD-SUB 2
ib4v17pwq_flc.fits UVIS-QUAD-SUB 2

These are all subarray exposures of Jupiter. Most of the observations use the UVIS-QUAD-SUB aperture, but that apparently moves from chip to chip depending on the quad filter being used. And the ib4v17puq_flc.fits observation uses the UVIS1-2K4-SUB aperture, which is a much larger subarray. Applying the WCS from one image to another leads to the weird result shown in the attached preview.

Another example is visit 11 of proposal 13675, which has 11 WFC3/UVIS exposures:

filename aperture ccdchip(s)
icoc11fmq_flt.fits UVIS 2 1
icoc11fnq_flt.fits UVIS 2 1
icoc11foq_flc.fits UVIS2-2K2C-SUB 2
icoc11fpq_flc.fits UVIS-QUAD-SUB 1
icoc11frq_flc.fits UVIS2-2K2C-SUB 2
icoc11fsq_flc.fits UVIS2-2K2C-SUB 2
icoc11ftq_flc.fits UVIS-QUAD-SUB 1
icoc11fuq_flc.fits UVIS-QUAD-SUB 1
icoc11fvq_flc.fits UVIS2-2K2C-SUB 2
icoc11fwq_flt.fits UVIS 2 1
icoc11fxq_flt.fits UVIS 2 1

This visit has several different apertures, including some UVIS apertures that are not subarrays (and so have both CCD chips) and some subarrays (with only 1 chip).

For this visit the results are really weird. I'm pretty sure that what happens is that one of the 2-chip observations is used as the reference image for the WCS. It supplies a WCS for both chips. When it gets to one of the images that is a subarray, there is no second chip. The 4th extension in those files is the D2IMARR extension, which is a 64x32 pixel distortion array. Apparently the astrodrizzle code blithely proceeds to apply the WCS to that tiny coefficient image and drizzles it into the output! The results for exposure icoc11foq_flc.fits are shown in the second image._

AstroDrizzle broken with astropy v1.3

After updating astropy in an astroconda environment, AstroDrizzle now dies with the error:
AttributeError: 'module' object has no attribute 'new_table'. The astropy.io.fits.new_table function was deprecated and removed as discussed here.

Some non-standard WCS keywords were excluded

Example test: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=36464239

[WARNING ] Some non-standard WCS keywords were excluded:
A_3_1, A_3_0, B_3_0, B_3_1, B_1_2, B_1_3, B_1_1, B_2_1, B_2_0, B_2_2,
B_ORDER, A_ORDER, B_0_4, B_0_3, B_0_2, A_4_0, B_4_0,
A_1_1, A_1_3, A_1_2, A_0_4, A_0_2, A_0_3, A_2_2, A_2_0, A_2_1
Use the ``relax`` kwarg to control this.

Wcskey 'O' is reserved for the original WCS and should not be deleted.

Currently, astrodrizzle code emits messages such as

[INFO    ] Wcskey 'O' is reserved for the original WCS and should not be deleted.

multiple times during a run (see, e.g., https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=38950982).

The code should be modified not to attempt to delete "original" WCS.

points were outside the output image

I see a lot of "nnn points were outside the output image" warnings. If they can be ignored, then just close this.

Why is Travis CI on?

There is not even a .travis.yml present. As a result, everything fails and it is misleading.

Increase precision of printed scale and skew in tweakreg

Vera Platais requested that tweakreg prints fit scales and skews with higher precision (~10 digits) in the fit's '*.match' files. Also she would like the precision of the shifts and RMS values be reduced to 2 digits after the dot.

I have looked at the code and currently the precision used for the scale is 6 digits which results in scale being printed as 1 when the actual value is 0.9999999. Also, it seems that skew is not printed at all.