keplergo / pyke Goto Github PK

View Code? Open in Web Editor NEW

118.0 16.0 35.0 24.19 MB

Easy command line tools for Kepler, K2 & TESS data analysis.

Home Page: http://pyke.keplerscience.org

Python 100.00%

kepler k2 tess astrophysics

pyke's Introduction

PyKE

A suite of Python/PyRAF tools to analyze Kepler data.

For more information and documentation, visit http://keplerscience.arc.nasa.gov/software.html#pyke

Installation with PyRAF

The easiest way to install PyKE is through the astroconda-iraf channel: http://astroconda.readthedocs.io/en/latest/installation.html#legacy-software-stack-with-iraf

After that, run the following commands on a terminal of your preference:

mkiraf
pyraf
kepler

Acknowledgement

If you find this code useful in your research, please consider citing: Title: PyKE: Reduction and analysis of Kepler Simple Aperture Photometry data

Authors:          Still, Martin; Barclay, Tom
Publication:      Astrophysics Source Code Library, record ascl:1208.004
Publication Date: 08/2012

This package was mostly developed by Tom Barclay (@mrtommyb) and Martin Still.

Support

Users are welcome to open issues involving any aspects of this software.

Feel free to contact us also through: [email protected]

pyke's People

Contributors

Stargazers

Watchers

pyke's Issues

[DOC]: rtfd is not building documentation properly

rtfd doesn't like our utils module

ImportError: No module named 'pyke.utils'

As a consequence, numpydocs are not being processed appropriately.

I'm investigating.

All tasks should have a default for `outfile`

outfile is currently required by many tasks. Let's help the user by setting a sensible default, e.g. infile + "-taskname.extension"

Use `--overwrite` instead of `--clobber`

In line with astropy, let's use --overwrite instead of --clobber in all commands and functions. It's a lot less obscure. I certainly could never remember the word clobber!

kepdraw: ploterr is not used

ploterr is not being used in kepdraw.

plotgrid was fixed in 311d745.

PER2DET

Hi,

Please have a look at this part of PER2DET:

https://github.com/KeplerGO/PyKE/blob/master/pyke/kepfunc.py#L298#L313

All these lines are equivalent to:

        for (j, y) in enumerate(DATy):
            for (k, x) in enumerate(DATx):
                xx = x - OBJx[i]
                yy = y - OBJy[i]

Does anybody know why there are some many useless calculations? Is anybody sure that this code is correct?

CC: @barentsen @mirca

Kepextract does not work on a single pixel anymore (?)

From [email protected]:

On Mon, Jun 5, 2017 at 8:53 AM, Mike Reed [email protected] wrote:

Hi Geert,
Any update on this? Our 'big' issue is that kepextract used to work on a single pixel, but at
some point there was an update so it now requires 7 pixels. Our kepmask file used to look like 
b5|||851|139|4,4, but now we have to list 7 pixels, which is too large for this crowded field, and 
not necessary for original Kepler data. We want to extract single pixels, find which pixels really 
have the star (from SC data, we know that it is only 2-3 pixels for each rotation) and then 
extract only those pixels.

Add possibility to change time format in kepconvert

Hi!

I'm currently looking how to convert BJD time to UTC time. I wondering if it could be an option to kepconvert. It'll allow developers to easily import ASCII or CSV files without fearing the conversion in environment without Astropy 😄

Clarify the distinction between command-line tools and Python code

It was pointed out to me by a beginning user that, e.g. in the first tutorial ( http://pyke.readthedocs.io/en/latest/tutorials/aperture-photometry-in-k2.html ), it is not clear that the wget and kepmask are command-line tools that need to be executed in the shell rather than a Python environment. Let's make sure this is always clear!

Add `kepsff` to PyKE3

It would be nice if the first release of PyKE3 can be used to create a detrended K2 lightcurve from the pixel data, i.e. using the steps explained in the original "Walkthrough E" at https://keplerscience.arc.nasa.gov/PyKEprimerWalkthroughE.shtml

The main thing missing to enable this is the kepsff task. How much time would it cost to port this task to PyKE 3? @mrtommyb seems to think there is no difference between the code in martinsff found in his Trappist-1 repo, and the original kepsff that came with PyKE.

kepmask: should be able to deal with negative flux values

When FLUX values have a negative value, they will display as white in kepmask because it takes the log10 by default. To fix this, let's add a pedestal to make sure all values are above zero before taking the log.

Have `kepconvert` produce CSV output

I propose the following improvements to kepconvert:

add a header line to the ascii output;
choose a sensible default for the output file name (e.g. infile+".txt");
set a sensible default for the default conversion (fits2asc?);
add support for csv;
improve the description of the output format in the --help string.

Tutorial needs to include downloads of PRF files.

Tutorial call to kepprfphot requires a directory of prf files. The tutorial needs to give instructions for downloading the files.

$ kepprfphot ktwo200071160-c91_lpd-targ-trimmed_3.fits ktwo200071160-c91_trimmed_3_prfphot --columns 1013 --rows 918 --fluxes 18000 --prfdir ~/kplr2011265_prf --background --clobber --verbose

The exception the routine throws when you try without it is also unhelpful:

KEPPRFPHOT started at: Thu Jun 29 16:51:12 2017
ERROR -- KEPIO.OPENFITS: cannot open ktwo200071160-c91_lpd-targ-trimmed_3.fits as a FITS file
ERROR -- KEPPRFPHOT: is ktwo200071160-c91_lpd-targ-trimmed_3.fits a Target Pixel File?
Traceback (most recent call last):
File "/usr/local/bin/kepprfphot", line 11, in
load_entry_point('pyke', 'console_scripts', 'kepprfphot')()
File "/Users/christina/K2/PyKE/pyke/kepprfphot.py", line 1157, in kepprfphot_main
args.overwrite, args.verbose, args.logfile)
File "/Users/christina/K2/PyKE/pyke/kepprfphot.py", line 244, in kepprfphot
kepmsg.err(logfile,message,verbose)
File "/Users/christina/K2/PyKE/pyke/kepmsg.py", line 16, in err
raise Exception(message)
Exception: ERROR -- KEPPRFPHOT: is ktwo200071160-c91_lpd-targ-trimmed_3.fits a Target Pixel File?

optimize performance of kepoutlier

Add a `kepsky` tool

In a future version (>3.0), we should add a tool to overlay or compare the region of sky covered by a Target Pixel File against survey images (DSS, SDSS, 2MASS...). The use case is to enable a user to get a high-resolution view to e.g. check for blended stars.

This tool has been requested by several people, and an example implementation is available at https://github.com/stephtdouglas/k2-pix

We could call it kepsky or similar.

`keprange` does not work with the Qt5Agg back-end

The interactive clicking feature of keprange does not work when I use the Qt5Agg back-end, though it does work with TkAgg.

@amcody and @gully reported similar problems with the OsX back-end.

We are going to want to either debug this or force the use of TkAgg.

Create new tutorials

This post intends to capture the list of tutorials we'd like to add to PyKE. Ideally, the tutorials would form a nice continuous story.This list is intended to be revised continuously. To make changes to this list, either comment below or edit this post.

Kepler/K2 basics

Data products

How to access Kepler and K2 data?
What is a Target Pixel File (TPF)?
What is a Lightcurve file (LLC)?
What is a Full Frame Image (FFI)?
What other data products do the archives provide?

Working with pixel data

Inspecting and plotting pixel data using kepmask and keppixseries.
Identifying instrumental artifacts in the pixel data

Working with lightcurves

Inspecting and plotting lightcurves
Identifying instrumental artifacts in the lightcurves using the QUALITY flags
How to stitch lightcurves from multiple quarters together?
How to remove long-term systematics from lightcurves using kepcotrend?
How to remove long-term astrophysical trends from lightcurves using kepflatten?
How to convert lightcurves to a different format using kepconvert?

Do-it-yourself photometry using PyKE

Aperture photometry for Kepler targets
Aperture photometry and systematics removal for K2 targets
PSF photometry in a K2 cluster
Separating a background EB from a foreground star

Studying exoplanets using PyKE

Finding planets

How to find planets using kepbls?
How to remove systematics due to the spacecraft?

Validating planets

How to check if an individual transits is due to artifacts or red noise? (transit like test)
How to check for a background EB using the centroiding test? (on target test)
How to check for secondary eclipses using the odd-even test? (stellar eclipse test)

Characterizing planets

How to fit a transit using keptransit? (or PyTransit or BATMAN)

Characterizing stars using PyKE

Seismology

How to plot up a power spectrum
Fitting granulation noise?
How to extract asteroseismic parameters from a light curve?
PS diagrams for a range of stellar types through Kepler, MS to RGB. RGBs in K2.

Rotation

How to extract the rotation period from a lightcurve?

Activity

How to characterize flares?
Fitting star spots
Butterfly diagrams/long term activity

Comments on docs and some warning/error messages

Some very minor edits to docs:

Docs

In install.html after the make html instructions, _build ➡️ build.
install.html directs users to pyke.readthedocs.io; should it direct to pyke.keplerscience.org instead?
In kepextract docstring, consider: "The pipeline" ➡️ "The Kepler pipeline"
In kepextract docstring, consider placing "Light curves for extended sources..." on its own line.
In kepextract docstring: "one of three options::" ➡️ "one of three options:"
In kepextract docstring: "pixels in within" -> "pixels within"
In kepextract docstring: "warnings to the shell" -> "warnings print to the shell"
Possible broken link: "We recommend the tools `ds9"

Warnings

invalid value encountered in double_scalars while running kepextract
tqdm jumps a line when warnings appear. Is there an option to detect and enable Jupyter notebook support similar to astropy Progressbar?
Kepextract prints information (e.g. Aperture photometry...) even in verbose=False mode. Should it?

Agree on a "printing and logging" convention for PyKE

We need a convention on how PyKE should deal with printing feedback to the users and writing logfiles. The current system based on verbose and logfile arguments is not in line with the logging mechanisms used by major scientific packages. I propose the following:

Main principles

There shall be no print statements anywhere.
All user feedback messages shall be delivered via a new keplog.log(level, message) function, which in turn shall make use of either Python's or AstroPy's built-in logging framework, which support different levels of logging (ERROR, WARNING, INFO, DEBUG).

Consequences

The verbose arguments shall be removed in favor of the config mechanisms provided by the logging framework. A convenience function called keplog.set_level can be added to help users set the log level (e.g. keplog.set_level(DEBUG)). The command-line tools can have a user-friendly --silent or --debug option to set the log level to ERROR or DEBUG, respectively.
All logfile arguments shall be removed in favor of the config mechanisms offered by the logging framework. A convenience function called e.g. keplog.set_logfile can be added to help users divert output to a logfile.

Other conventions

Whenever PyKE writes a file, it must be reported to the user via keplog.log(INFO, "Writing " + filename).
Progressbars shall have their desc and disable argument set, i.e.
tqdm(iterable, desc="Informative description", disable=(keplog.get_level() <= INFO))
Command-line tools should catch all exceptions and print them via keplog.log(ERROR, msg).

What do y'all think?

Implement the "walkthrough" examples in the old pyke website

Add a "How to use PyKE in Windows?" tutorial

If you are a pro Windows user that came across this ticket, please feel free to respond and volunteer to help us figure out how to make PyKE accessible in Windows. The Kepler team tends to use OSX and Linux, but is interested in making their tools as widely accessible as possible.

Inconsistency in kepcotrend

function fitfunct_fmin and the likes require date as parameter, but don't do anything with it

Update CLI descriptions in docstrings

Minor notes on the CLI when you execute --help.

In kepwindow --help "Name of input file" is too general. Should say some more concrete information, FITS only, or ascii too?
In keptrim --help the command line argument --kepid refers only to "the Kepler ID number from the Kepler Input Catalog". What about K2 EPIC numbers?
In kepstitch --help, "Append multiple month short cadence and/or multiple quarter long cadence data" should have a qualifier that it's for the original Kepler data only. Also: --outfile OUTFILE is an optional command line argument, which differs from most other CLI behaviors. (I actually prefer the default outfile name strategy.)
Why does kepstddev output an outfile if it just calculates CDPP? Does it modify the header or something?
In kepsff, repeated word "for":

--npoly_arfl NPOLY_ARFL
Order of ploynomial ~~for~~ for arclength-flux calibration
(default: 3)

In kepsff, the name --plotres, does not seem to match the description:

Save hardcopies of the plots?

More explanation needed for how kepprf differs from kepprfphot
keppca must require the CBV files, right? This could be noted in the CLI.
In kephead: "Snippet of keyword name" could be "full or partial keyword name"
In kepft CLI "search period" is a bit misleading, since nothing is being "searched".
In kepfold, many of the CLI optional arguments refer to binning properties, which are only needed if binning is selected. Can that be indicated somehow?
In kepflatten: the description is wrong:

Remove or replace data outliers from a time series

A few suggestions on kepprfphot

The barytime vector in kepprfphot looses precision due to multiple floating-point operations and the printed output is slightly (O(10^-5 d)) different from naive expectations.
A large part of kepprfphot code is devoted to preparing the plots.
A) this should be a separate function because it's logically different task,
B) producing this plots should be an option, or there should be option to turn it off,
C) points A) and B) may apply to other source files as well.
When I read some of the output files using astropy and access the tabulated data, I'm getting error message WARNING: VerifyWarning: Invalid keyword for column 4: Column null option (TNULLn) is invalid for binary table columns of type 'E' (got -1). The invalid value will be ignored for the purpose of formatting the data in this column. [astropy.io.fits.column]. I have not in detail investigated the cause of this issue.

Update the KeplerScienceWebsite to reflect PyKE v3.0

Let's update our science center website to provide instructions on installing and using PyKE3.

Use RestructeredText instead of MarkDown for our README

Pypi only supports ReStructuredText, which is why pyketools on pypi looks silly: https://pypi.python.org/pypi/pyketools

kepconvert error on converting ascii to fits

kepconvert is supposed to be able to convert fits to ascii and ascii to fits. The fits to ascii works fine, but when you try to go the other way there is an error message.

Fits to ascii

kepconvert('kplr008462852-2011073133259_llc.fits','dip.ascii',overwrite=True,conversion='fits2asc', columns='TIME,SAP_FLUX,SAP_FLUX_ERR')

Ascii to fits

kepconvert('dip.ascii','kplr008462852-Q8.fits',overwrite=True,conversion='asc2fits',columns='TIME,SAP_FLUX,SEP_FLUX_ERR',verbose=True)
Produces the error
ERROR -- KEPCONVERT: cannot create light curve extension in kplr008462852-Q8.fits

speeding up kepclip

Big props on how much faster kepextract is on short-cadence data with PyKE 3.0, thanks for that!

I've found another (less significant) bottleneck: kepclip takes A LOT longer than it should if it's just removing entries in a time series (up to a few minutes on a relatively new MacBook Pro). I wonder if you could investigate why kepclip is so SLOW on short-cadence data. kepstitch runs considerably faster for what should be essentially the same arithmetic. Thanks!

Add tutorials using PyKE

Add `kepdraw` to PyKE3

PyKE2 included a tool called kepdraw that was used to see what a lightcurve looks like. It was not ported to PyKE3 as part of #12, but I do think we should consider including this tool in the first release of PyKE3. This is because many users may not be comfortable with making a quick lightcurve plot from scratch, which may keep them from adopting PyKE3.

Add function to decode quality flags to PyKE

A possible implementation is already here: https://gist.github.com/barentsen/9b5a7b51f48e026e3af5

Bad time format keyword in TPF/lightcurve file

I was working on this one: ktwo220142950-c08_spd-targ.fits, when kepconvert failed with the following:

Exception: ERROR -- KEPCONVERT: error converting time to ktwo220142950-c08_spd-targ.lc.txt: format must be one of ['jd', 'mjd', 'decimalyear', 'unix', 'cxcsec', 'gps', 'plot_date', 'datetime', 'iso', 'isot', 'yday', 'fits', 'byear', 'jyear', 'byear_str', 'jyear_str']

Apparently units are "BJD-2454833" in the light curve file generated by kepextract instead of the ones above.

kepmask: imin and imax are not used but can mess up iscale

Hi,
I tried to rescale the TPF figure in kepmask but manually entering imin and imax values had no effect. This is problematic when the automatic scaling runs into trouble for some TPFs, and need forced imin and imax to display properly (default scaling gives "ValueError: minvalue must be less than or equal to maxvalue"). One example is KTWO229228811-C08.

The default logarithmic scaling stays the same. When I tried with linear or squareroot with manual imin/imax values, it gives the following errors. Linear scaling:

line 171, in kepmask
    zmin *= 1.0
TypeError: can't multiply sequence by non-int of type 'float'

Squareroot scaling:

line 168, in kepmask
    zmin = math.sqrt(zmin)
TypeError: a float is required

Release PyKE v3.0.0

Thanks to @mirca's incredibly hard work, the master branch of PyKE is now compatible with Python 3 and no longer depends on PyRAF (cf #12), which were the key goals for "PyKe v3.0" aka "PyKE3". This means we can now start planning the first official release of PyKE3!

Let's start by releasing a "3.0.beta" and announce it to a small audience on social media, followed by an official "3.0.0" a few weeks later announced on the website.

Tasks to complete ahead of the release are likely to include the following:

Review the documentation and its hosting
Edit the keplerscience website to reflect PyKE 3.0
Ensure installation and citation instructions in the README are up to date
Add the kepdraw tool [#18]
Add the kepsff tool [#19]
Add a simple tutorial to the sphinx docs that explains how to get a lightcurve from K2 pixels using kepmask + kepextract + kepflatten + kepsff + kepdraw (cf. https://keplerscience.arc.nasa.gov/PyKEprimerWalkthroughE.shtml)
Decide on a name for PyKE to use in PyPI (pyke, pykep, pykepler are all taken by others). Let's use pyketools?

Move PyKE documentation

The current PyKE doc lives in https://keplerscience.arc.nasa.gov/PyKE.shtml

The goal is to have it on sphinx (with a beautiful, elegant, minimalist theme).

I'm working on this also in #12

Suppress warning in kepdraw

This warning

WARNING: VerifyWarning: Invalid keyword for column 4: Column null option (TNULLn) is invalid for binary table columns of type 'E' (got -1).  The invalid value will be ignored for the purpose of formatting the data in this column. [astropy.io.fits.column]

appears all the time when using kepdraw. I guess we should suppress it somehow.
thoughts? @barentsen

kepclip should accept relative ranges

kepdraw should plot PDCSAP_FLUX by default and show informative msg

Geert's idea.

Offer the possibility to export the data to a time series database

Hi,

I'm new to astronomy analysis, so maybe my need is fulfilled with another software, please let me know 😄

I would like to give a try analyzing Kepler time series with Warp10, an open source time series database that has huge analysis capabilities. For this, I need to export Kepler FITS time series to the Warp10 Warp10 Input Format.

I saw that there is a task called kepconvert, should I add the option there or create a new task for TS export? Other people might want to export to another standard.

This is not a requirement for the first release however.

PRFfit[j, k] += PRFfit[j,k] + splineInterpolation(dy * wy, dx * wx) * flux[i]

It like "x += x + dx". Previous function in the same file has:

PRFfit[j, k] = PRFfit[j, k] + splineInterpolation(dy, dx) * flux[i]

which looks normal. Can you comment on this?

`kepsff` raises ValueError in K2 aperture photometry tutorial

When trying out the K2 aperture photometry tutorial I encounter the following exception:

$ kepsff ktwo202073445-c00_lpd-targ-kepextract-kepclip-kepflatten.fits --datacol DETSAP_FLUX --stepsize 5. --npoly_ardx 4 --sigma_dsdt 10. --overwrite
  0%|                                                                                                                                               | 0/7 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/gb/bin/anaconda/bin/kepsff", line 11, in <module>
    load_entry_point('pyketools', 'console_scripts', 'kepsff')()
  File "/home/gb/dev/PyKE/pyke/kepsff.py", line 774, in kepsff_main
    args.logfile)
  File "/home/gb/dev/PyKE/pyke/kepsff.py", line 701, in kepsff
    instr[1].data.field('SAP_FLUX')[t1:t2] /= cfac
ValueError: operands could not be broadcast together with shapes (244,) (8,) (244,)

kepmask: doesn't work with macosx backend on @gully's machine

We may want to consider enforcing e.g. TkAgg as the backend.