spacetelescope / astrocut Goto Github PK

This might need to be updated to include a conditional involving the upper boundaries (in X and Y) of the FFIs. From what I understand, this is catching for cases where the lower limit of the cutout limits falls below 0 in either axis, but we don't have a conditional for when the upper limit goes beyond 2048 or whatever the FFI image size is:

Looking at TESSCut products, the data array includes columns 'POS_CORR1' and 'POS_CORR2' but all the values are zero. For a typical TPF, these columns give the per-image pointing offsets relative to the WCS information, and they are important for matching source locations between TESS and other data sets. Is it possible to include the correct differential velocity aberration data in the TESSCut products? I have code that uses this information to analyze TPFs that I hope to use on TESSCut image arrays as well.

Per the astropy documentation:

When combining intervals and stretches, the stretch object must come before the interval object.

Currently they are applied in the opposite order.

Could this repo get a tag called "mast" so that a URL that looks like https://github.com/search?q=topic%3Amast+org%3Aspacetelescope would allow this to show up as public software relevant for MAST users?

On the s3-support branch for making direct s3 cutouts, seeing an error with this set of input data:

from astrocut import CutoutFactory

astrocut_cutter = CutoutFactory()

cube_file = 's3://stpubdata/tess/public/mast/tess-s0030-4-3-cube.fits'
coordinates = '102.7 -70.5'
cutout_size = (3, 3)
output_path = '/tmp'

astrocut_cutter.cube_cut(cube_file, coordinates, cutout_size, output_path=output_path)

Traceback:

Traceback (most recent call last):
  File "/Users/bfalk/repos/astrocut/astrocut/tests/test_small_s3_cutout.py", line 12, in <module>
    astrocut_cutter.cube_cut(cube_file, coordinates, cutout_size, output_path=output_path)
  File "/Users/bfalk/repos/astrocut/astrocut/cube_cut.py", line 736, in cube_cut
    self._parse_table_info(cube.table.data, verbose)
  File "/Users/bfalk/repos/astrocut/astrocut/cube_cut.py", line 930, in table
    return self.loop.run_until_complete(self.cube_async.table)
  File "/Users/bfalk/miniconda3/envs/astrocut/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/bfalk/repos/astrocut/astrocut/cube_cut.py", line 1018, in table
    self._table = await self.read_table()
  File "/Users/bfalk/repos/astrocut/astrocut/cube_cut.py", line 1032, in read_table
    hdu2_str = await self._read_block(hdu2_offset, length=None)
  File "/Users/bfalk/repos/astrocut/astrocut/cube_cut.py", line 1064, in _read_block
    resp = await self.s3_client.get_object(
  File "/Users/bfalk/miniconda3/envs/astrocut/lib/python3.9/site-packages/aiobotocore/client.py", line 187, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRange) when calling the GetObject operation: The requested range is not satisfiable

Specifying a local file for the cube_file, I don't see this error, and the cutout succeeds.

I'm wondering if this particular sector/file needs some special casing when reading in the table information?

@barentsen wondering if you might be able to take a look at this at some point?

Cutouts are able to be made on FITS Cut out of the box, but future mission like Roman will be housing data in ASDF files. Find a way to support this. ASDF Cut?

Currently there's no error handling for cases where a user feed TicaCubeFactory SPOC FFIs and vice versa. Add some explicit error handling for this.

The astrocut.fits_cut method fails to write the cutout file to the directory specified by the output_dir argument. Instead the cutout file always appears to be written to the directory from which the image was read. A sample test file is attached which demonstrates this (sadly renamed as a '.txt' file to fool stupid GitHub attachment checks).

test_fits_output.py.txt

Below is the output from a successful test run. Note that the line:

OUTPUT_DIR=/tmp/outdir

is output from within _save_multiple method of cutouts.py, indicating that the method is receiving the correct value for the output_dir parameter. However, the line:

co_file=/tmp/imgdir/m13_250.417651_36.458904_100x100_astrocut.fits

is printing the paths (in this case, just one) in the list returned by cutouts.fits_cut and the path is pointing to the directory from which the image was read.

============================= test session starts ==============================
platform darwin -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0

Running tests with Astropy version 3.2.2.
Running tests in astrocut/tests/test_fits_output.py.

Date: 2019-10-27T22:12:47

Platform: Darwin-17.7.0-x86_64-i386-64bit

Executable: /usr/local/src/anaconda3/envs/astrocut/bin/python

Full Python Version:
3.7.4 (default, Aug 13 2019, 15:17:50)
[Clang 4.0.1 (tags/RELEASE_401/final)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Numpy: 1.17.3
Scipy: 1.3.1
Matplotlib: not available
h5py: not available
Pandas: not available
astropy_helpers: 3.2.1
Using Astropy options: remote_data: none.

rootdir: /private/var/tmp/astrocut-test-u09wli0y/lib/python3.7/site-packages, inifile: setup.cfg
plugins: arraydiff-0.3, astropy-header-0.1.1, doctestplus-0.4.0, openfiles-0.4.0, remotedata-0.3.2
collected 1 item

OUTPUT_DIR=/tmp/outdir
astrocut/tests/test_fits_output.py
co_file=/tmp/imgdir/m13_250.417651_36.458904_100x100_astrocut.fits
 <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>

Following the work on PR #56

@falkben has a couple suggestions on how the GitHub Actions workflow can be improved for astrocut:

• cutting out 3.6
• including 3.9 and 3.10
• updating the other dependencies we test against (e.g. which version of astropy and numpy)
• cleaning up the tox.ini file for anything we're not using, which i'm not even sure what we're not using at this point and what we are using.
• seeing why "Python 3.7 with dev version of key dependencies" ALWAYS fails (i know we mark is allowed failure... but if it always fails, it's not useful either)
• fixing python 3.8 windows 32 test

To whom it may concern,

If you are using https://github.com/astropy/ci-helpers in your appveyor.yml , please know that the Astropy project has dropped active development/support for Appveyor CI. If it still works, good for you, because we did not remove the relevant files (yet). But if it ever stops working, we have no plans to fix anything for Appveyor CI. Please consider using native Windows support other CI, e.g., Travis CI (see https://docs.travis-ci.com/user/reference/windows/). We apologize for any inconvenience caused.

If this issue is opened in error or irrelevant to you, feel free to close. Thank you.

xref astropy/ci-helpers#464

In line

Tyler Desjardins mentions that we should consider moving emails from help[at]stsci.edu to point to the web portal where possible and appropriate. For HST (or any non-JWST), it is https://hsthelp.stsci.edu . For JWST, it is https://jwsthelp.stsci.edu . Please update info in setup.py, setup.cfg, documentation, etc as appropriate.

Please close this issue if it is irrelevant to your repository. This is an automated issue. If this is opened in error, please let pllim know!

xref spacetelescope/hstcal#317

APE 17 is accepted. Look out for a "conversion guide" that Tom Robitaille will send out to astropy dev mailing list in the near future.

Address final todo items in test_cube_cut.py

Suggestion from @falkben RE: Pull Request #59

"Can we use class inheritance here? It looks like most of the changes would just be in the class attributes. If we make those changes, I think we can make these changes much simpler and should be easier to maintain."

Also some commentary I initially made in TicaCubeFactory class:

NOTE: Later down the line 
        we might want to generalize CubeFactory so that we no longer need 
        a separate class for the TICA cubes. 
        
        Modifying the __init__ 
        in CubeFactory so that it can switch parameters based on whether 
        we want cubes for TICA or TESS will probably be the route I take.
        Or see how this is generalized when Zcut wraps CubeFactory?

"Cutout contains to data!" in cutout.py. Should be "Cutout contains no data." "To data" occurs in four places.

As recommended by @chamblee-st , it would make the cutout tests more modular

Right now fits_cut and img_cut assume the list of images they are given all share the same WCS and pixel scale, but no actual checking is done. A minor improvement would be to add checking and raise an error is the images don't match. A larger improvement would be to implement optional resampling.

In the astrocut document

https://astrocut.readthedocs.io/en/latest/astrocut/

the example given in the "Making Image Cube" section is incomplete and thus can not execute.

The line

input_files = glob("data/*ffic.fits")

refers to the data subdirectory which does not exist which causes input_files to be an empty list.

Suggest solution: Test to see if data/*ffic.fits files exist before defining input_files. If no such files exist, then create a small set of *ffic.fits images in a directory called data.

I appear to be experiencing issues with TESScut. For example, requesting a cut-out for Pi Men in sector 1 yields a Cutout Error: No FFIs found for input target. Cutout not performed..

Example:

For easier troubleshooting and checks in the future.

Greetings! First, thank you so very much for developing astrocut / tesscut! It's a very useful tool -- I've just started using it, and sailing has overall been very smooth.

Recently though, I believe I encountered a potential bug!

Behavior: using the TESScut web service, when I enter (RA, dec) = (85.8, -1.5) as in the attached screenshot, I am told "Cutout Error: No FFIs found for input target. Cutout not performed." However, the location specified is definitely in sector 6, camera 1, ccd 1 (see image on the right, with the lovely M42). This error seems to be returned for a number of the celestial coordinates on this CCD.

I checked the holdings website, which indicated that sector 6 should be fine to query.

My plan is to write up a CubeFactory + CutoutFactory routine to investigate a bit more closely, since I figure this may be because the backend database might not yet be updated through Sector 6? Nonetheless, this seemed worth reporting, since I am likely not the only one!

Thank you again your work developing and maintaining this project!
--Luke

The Python 3.7 with dev astropy tests are failing, this needs to be fixed before the next astropy release.

Issue to capture small non-urgent things that I notice while adding new functionality:

cube_cut.py

• _get_cutout_limits docstring parameters is wrong (center_coord is no longer a param)

• target_pixel_file = os.path.join(output_path, tpf_name) is indented one too far

• Get rid of the overwrite=True or make it an argument in cube_cut?

• Make cube_cut handle x vs y consistent with new fits_cut functionality (which is correct)

• Update file creation date to reflect cutout date not cube date

Tests

• Remove unused extname argument in add_keywords

General

• Make a utils file and move common functionality there
• Update docs to include info on how to cite
• Refactor cube_cut/make_cube into one file?

Thank you for making and maintaining TESSCut, it is a great service!

I've come across an issue making light curves for moving objects which I think is a function of the name used. The issue might be when forward slashes are in the object name, somewhere in the parsing of that information?

As an example, if I try to make a light curve for the comet C/2014 UN271 with the following (or through the web interface)

https://mast.stsci.edu/tesscut/api/v0.1/moving_target/astrocut?obj_id=C%2F2014%20UN271&y=20&x=20&units=px&sector=30

I get a Cutout Error: Cutout could not be performed response. However, if I use an alternate name for the same object, Bernardinelli-Bernstein, the equivalent command (https://mast.stsci.edu/tesscut/api/v0.1/moving_target/astrocut?obj_id=Bernardinelli-Bernstein&y=20&x=20&units=px&sector=30) works perfectly.

My assumption from this is that it must be something in the object name, and the forward slash is the most plausible suspect to me. Have you come across this before?

Rename CubeFactory to SpocCubeFactory? Or eventually consolidate the cube factory classes into 1 for TICA and SPOC.

When fit_wcs_from_points went into astropy its call pattern changed, the local version in astrocut needs to be updated to match so that astrocut can use the astropy version if it is available and it's own if it isn't.

Additionally testing needs to be done to make sure the accuracy of the fit wcs remains the same.

Add header keyword value quality checks to unit tests for make_cube.py.

There was an incident where the EXT1 header keyword NAXIS2 did not reflect the correct number of FFIs that were added to the cube. We are still unsure of what happened, but as a preventative measure, it would be good to have some checks in place to verify the quality of the primary and image header keyword values.

Hello!

We've come across an issue in the timestamps in TessCut frames (TessCut is a wonderful service, by the way!)

When I look the astrocut file timestamps, there is a digitization that I can see in them when we plot the difference in time between consecutive frames. We think this is a floating point rounding error in the timestamps, somewhere they are being stored as 32-bit floats and since the time values have four digits before the decimal place, this limits the precision to ~10 seconds on any individual data point.

You can see what I mean in figure form at afeinstein20/eleanor#119. The code to reproduce this is below:

tc = fits.open('tess-s0001-4-3_97.842681_-68.044906_10x15_astrocut.fits')
plt.plot(np.diff(86400*tc[1].data['time']), '.')
plt.ylim(1795, 1805)
plt.xlabel('Cadence', fontsize=20)
plt.ylabel('Delta Time (sec)', fontsize=20)

The TASOC light curve times look proper, so we think it's a processing issue along the way rather than the way the telescope is recording data. If you think it's something upstream from astrocut where the bug is then I'm happy to work my way up the chain until we figure out where this is being induced.

Thanks so much!
Ben

Hello,

I'm trying to create a cutout from a cfht megaprime image. I've confirmed the target is on one of the chips, but the software keeps telling me it isn't. I've tried using the extension = 'all' option, but I assume I'm just not handling the multi-extension file correctly.

The target appears on different chips in some of the images, so I was hoping I wouldn't need to determine which one (which extension) I need on each image.

Let me know if you have any suggestions, or if you've seen this issue before.

Thank you and all the best,
Scott

I'm currently trying to run the code from a Win10 machine, Anaconda, in a Jupyter notebook. The image (~340MB) can be obtained from here

from astrocut import fits_cut
from astropy.io import fits
from astropy.coordinates import SkyCoord

input_files = ['714745p.fits.fz.part']

center_coord = SkyCoord.from_name('[HS80] B337')
cutout_size = 500

cutout_file = fits_cut(input_files, center_coord, cutout_size, extension = 'all', single_outfile=True)
print(cutout_file)

This may be a known issue, but I thought it'd be worth highlighting the following caveat:

TESSCut appears to resolve names to their J2000.0 position. For example, Proxima Cen:

This is a caveat because nearby stars such as Proxima Cen have a very large proper motion, so anyone clicking "Download FFI Cutout" after having entered the name of this target may be surprised to find that the object is not in the center of the TPF. This is because its high proper motion since J2000 has moved it slightly out of the cutout! Example for Proxima Cen:

This may not be important to fix, because it only affects a small number of high proper motion targets, but some solutions could be to:

• have the TESSCut service automatically return the J2020.0 position;
• highlight in the interface that the names resolve to J2000.0 positions; or
• issue a warning if a name of a high-proper motion object is resolved.