gabriel-p / photpy

Python CCD photometry

photpy

Python CCD photometry.

Docs: http://photpy.rtfd.io/

photpy's Issues

Road to v1.0

List of issues to close before 1.0 can be published:

• Get rid of psfmeasure in fitstats script #17
• Extract the crop process from align_crop and deprecate the alignment #20
• Extract a more general pymatch process from id_standard #19
• Add flip support to pymatch script #6
• Limit the number of stars used in pymatch script #8
• Solve transformation equations #7
• Assign errors to aperture photometry values #16
• Perform linear regression taking errors in x,y into account in fit_standards script #10
• Finish psf_phot script #18
• Duplicate DAOMASTER's main functions #12

Two good resources (source):

The 2017 and 2018 versions of my lecture notes are here:
2017: https://ysbach.github.io/AO_2017/
2018: https://ysbach.github.io/AO_LectureNotes/

Read article on cross-matching using k d trees

See 2. Cross-matching section:

https://sandipanweb.wordpress.com/2017/07/29/some-analysis-with-astronomy-data/

Perform PSF photometry

• Main article: https://photutils.readthedocs.io/en/stable/psf.html

• photutils-datasets

  1. PSF Photometry in Crowded Fields with Photutils
  2. PSF/PRF Photometry on Spitzer Data
  3. PSF photometry on artificial Gaussian stars

• Uncertainties

  1. issue
  2. example

• Generate PSF model from stars in frame? (astropy/photutils-datasets#4)

• Do the *PSFPhotometry classes replace IRAF's allstar?

• Aperture correction?

• Summarized points: GUIDE TO COMPUTING A PSF IN A CROWDED FIELD

Extract the crop process from align_crop and deprecate the alignment

The alignment process should be deprecated in favor of the more general pymatch script (see #19). The crop process should be kept from this script, to see if it can be used.

See about replacing/enhancing the standard star matching function with Astroalign

Astroalign: A Python module for astronomical image registration

• https://arxiv.org/abs/1909.02946
• https://github.com/toros-astro/astroalign

Store coords found by get_data

The coords of "good" stars found by get_data should be stored in a .coo file.

These coords can later be used by the align_crop script, and save the time it takes to obtain them.

How is 'sigma' obtained by fitparams??

The photcal/fitparams IRAF task shows a sigma column when :vshow is used, with values for each star used in the fit. These values are used to obtain the reduced chi, and others statistical parameters.

How are these sigma values obtained? The code seems to write to that column a 1/sqrt(X) and the description of :results (in inlfit, see below) states that this is a weight.

---

fitparams uses INLFIT ("The interactive non-linear least squares fitting package") which uses the standard Levenberg-Marquardt non-linear least squares algorithm to fit the data.

Detailed descriptions of the algorithm can be found in the following two references.

1. Bevington, P.R., 1969, Data Reduction and Error Analysis for the Physical Sciences, Chapter 11, page 235.
2. Press, W.H. et al., 1986, Numerical Recipes: The Art of Scientific Computing, Chapter 14, page 523.

Related data:

• The Levenberg-Marquardt method for nonlinear least squares curve-fitting problems, Henri P. Gavin (2017)

Limit the number of stars used in pymatch script

Currently the triangleMatch() function uses all the stars in both frames to obtain the possible triangles. MATCH uses the brightest 30 which sounds reasonable.

Rename and re-factor id_standard

Make this script more general so that it can act as as replacement to the (DAO)MATCH code.

1. Change name (pymatch?, photmatch?)
2. Accept any number of frames, not just a standard frame. Align these to a given reference frame.
3. Output transformation values to file (to be used by match_transf as done by DAOMASTER to refine the matching)

Add flip support to pymatch script

The findRotAngle() function will (should) fail if one of the frames is flipped with respect to the other.

Extract a more general 'match' process from id_standard

This script should act as as replacement to the (DAO)MATCH code.

• Add flip support #6
• Limit the number of stars used #8
• Solve transformation equations #7
• Accept any number of frames, not just a standard frame. Align these to a given reference frame.
• Output transformation values to file (to be used by match_transf as done by DAOMASTER to refine the matching)

Use coords from fitstats in align_crop

Use the coords already generated by fitstats, if they exist.

See #1

Aperture photometry error for standards

Properly assign error to the magnitudes obtained for the standard stars.

Photutils example: https://photutils.readthedocs.io/en/stable/aperture.html#error-estimation

How is the value returned with the method shown above related to the merr value given by IRAF's phot task?

• IRAF forum thread on phot error: http://iraf.net/forum/viewtopic.php?showtopic=138390
• Document mentioned in that thread: http://www1.phys.vt.edu/~jhs/phys3154/snr20040108.pdf
• Photutils issue open: astropy/photutils#629

Explore linear fitting methods for replicating fitparams

Read

• OLS vs ODR: Least squares regression with errors in both variables: case studies (Zotero)
• ODR: Orthogonal Distance Regression (Zotero)
  • Estimate the standard deviation of fitted parameters in scipy.odr? How to properly calculate the standard deviations for the fitted parameters? (see comments in answer)
• BCES: Linear Regression for Astronomical Data with Measurement Errors and Intrinsic Scatter (Zotero)

Auto assign standards in frame

The standards stars in a given field should be automatically assigned to detected stars in the observed frame.

See algorithm for polygon stretching: https://gist.github.com/hellpanderrr/2b2252e350a0e20c7271

Perform linear regression taking errors in x,y into account (fit_standard)

--> How to fit a model to data, Hogg (2010) <--

(Is this the code used to perform the Bayesian, outlier model, 2D uncertainties fit?)

• Facebook post with lots of advice about linear regression
• My own FB post on the subject. Recommendations:
  • Fitting a simple model to data using MCMC (outliers not modelled independently)
  • Fitting a straight line with outliers (outliers known a priori?)
  • A Bayesian approach to linear regression with errors in both X and Y (linmix), article (apparently does not evaluate outliers)
• Robust Regression with Outlier Detection (PyMC3) (implements the Hogg, Bovy & Lang method using PyMC3)
• Two- and Multi-dimensional Curve Fitting using Bayesian Inference, Steiner (2017), an article that discusses this issue

OLD (below)

---

Explore linear fitting methods for replicating fitparams

• Fitting data when both variables have uncertainties
• AutoCal: A software application for calibrating photometric data, Wium & Soelen 2017 (Zotero)
• Least squares regression with errors in both variables: case studies, Oliveira & Aguiar 2013 (Zotero) (OLS vs ODR, nice illustrative image)
• Orthogonal Distance Regression, Boggs & Roger 1990 (Zotero)
  • Estimate the standard deviation of fitted parameters in scipy.odr? How to properly calculate the standard deviations for the fitted parameters? (see comments in answer)
• BCES: Linear Regression for Astronomical Data with Measurement Errors and Intrinsic Scatter (Zotero)

Examples

1. Calculate uncertainty of linear regression slope based on data uncertainty
2. Fitting a Model to Data (emcee)
3. Line of Best Fit (Jake VanderPlas), emcee

StackOverflow

1. Standard deviation for fitted parameters with scipy.ODR, when data contains error
2. Linear fitting in python with uncertainty in both x and y coordinates
3. Linear fit including all errors with NumPy/SciPy
4. How to do linear regression, taking errorbars into account?
5. Python linear fitting with multiple error bars

Vary reference frame until all images are properly aligned

Currently the align_crop script will use as a reference frame (if none is given) the one with the most stars found.

This frame is not always the best one to use. If one or more frames are detected as possible bad alignments, the reference frame should be switched and the process applied again.

Do this until all frames are aligned, or until no more frames are left. In the latter case, use the run where the most frames were properly aligned.

Solve transformation equations

1. Find all stars in common between both frames

2. Use those stars to solve for the parameters of the general six-parameter transformation:

   x_1 = A + C*x_n + D*y_n
   y_1 = B + E*x_n + F*y_n
   

3. Output transformation values to file (to be used by match_transf as done by DAOMASTER to refine the matching)

This way we are reproducing the tasks carried out by MATCH.

Run DAOStarFinder in id_standard

Currently the id_standard script uses a .coo file from fitstats to fins the scaling, rotation, and translation for the observed standard frame to match the standard stars image.

This is nor correct since after the align_crop script, the coordinates will change. The script needs to run DAOStarFinder to generate its own set of coordinates from an already aligned frame.

Replace psfmeasure task in fitstats

Currently the fitstats script uses the psfmeasure task to isolate bright stars and estimate the frame's FWHM.

This single task makes the package depend on PyRAF (and IRAF), which is not good.

Find (or create) a replacement built on Python.

Duplicate DAOMASTER's main functions

1. The techniques of least squares and stellar photometry with CCDs, Peter B. Stetson (1989)
   Lecture by P Stetson himself. Lots of details about how MATCH and MASTER work at the end.

2. Modern Observational Techniques: Fall 2015, Mike Bolte

   1. Lecture on aperture photometry and PSF-fitting photometry, photometric calibration.
      Mentions some interesting facts about DAOMATCH & DAOMASTER (not many details). Explains the process of photometric calibration (to a standard system) and aperture correction.
   2. DAOMASTER and DAOMATCH help file
      Describes in some detail how both these codes work.

3. Algorithms for CCD Stellar Photometry, Kenneth J. Mighell (1999).
   Basic techniques of aperture and PSF-fitting CCD stellar photometry are discussed in detail. Describes the process of finding stars as peaks and centering them, transforming from raw signal to instrumental magnitudes (and errors), aperture correction, PSF fitting (lots of math).

4. After Fitting the PSF
   Basic tutorial that gives a description of the DAOMATCH & DAOMASTER processes.

5. How to make Color-Magnitude Diagrams of Star Clusters, Chomiuk (2006)
   Small tutorial, not much information.

---

CCD photometry docs:

• https://www.aavso.org/ccd-photometry-guide-spanish
• http://www.britastro.org/vss/ccd_photometry.htm
• http://www.physics.csbsju.edu/370/photometry/manuals/OU.edu_CCD_photometry_wrccd06.pdf
• http://www.adass.org/adass/proceedings/adass98/mighellkj/