karpov-sv / stdpipe

Simple Transient Detection Pipeline

STDPipe - Simple Transient Detection Pipeline

AKA: random codes noone else will ever use

STDPipe is a set of Python routines for astrometry, photometry and transient detection related tasks, intended for quick and easy implementation of custom pipelines, as well as for interactive data analysis.

Design principles

• implemented as a library of routines covering most common tasks
• operates on standard Python objects: NumPy arrays for images, Astropy Tables for catalogs and object lists, etc
• does not try to re-implement the things already implemented in other Python packages
• conveniently wraps external codes that do not have their own Python interfaces (SExtractor, SCAMP, PSFEx, HOTPANTS, Astrometry.Net, ...)
  • wrapping is transparent: all data passed from Python, all options customizable from Python, all (or most of) outputs available back
  • everything operates on temporary files, nothing is kept after the run unless explicitly asked for

Features

• pre-processing - should be handled before in an instrument-specific way
  • bias/dark subtraction, flatfielding, masking
• object detection and photometry
  • SExtractor or SEP for detection, photutils for photometry
• astrometric calibration
  • Astrometry.Net for blind WCS solving
  • SCAMP or Astropy-based code for refinement
• photometric calibration
  • Vizier catalogues, passband conversion (PS1 to Johnson, Gaia to Johnson, ...)
  • spatial polynomial + color term + intrinsic scatter
• image subtraction
  • HiPS templates
  • PanSTARRS DR1 or Legacy Survey templates
  • HOTPANTS + custom noise model
  • ZOGY
• transient detection and photometry
  • noise-weighted detection, cutout adjustment, ...
• auxiliary functions
  • PSF estimation, simulated stars, FITS header utilities, plotting, ...
• light curve creation (soon)
  • spatial clustering, color regression, variability analysis, ...

Installation

STDpipe is available at https://github.com/karpov-sv/stdpipe and is mirrored at https://gitlab.in2p3.fr/icare/stdpipe

The package is in constant development, so to keep track of the changes the suggested way of installing it is by cloning the repository

git clone https://github.com/karpov-sv/stdpipe.git

and then installing from it in development (or "editable") mode by running the command

cd stdpipe
python3 -m pip install -e .

This way you may update the repository or apply local patches, and it will immediately be reflected in the installed package.

Apart of Python requirements that will be installed automatically, STDPipe also (optionally) makes use of the following external software:

• SExtractor
• SCAMP
• PSFEx
• SWarp
• HOTPANTS
• Astrometry.Net

Most of them may be installed from your package manager. E.g. on Debian or Ubuntu systems it may look like that:

sudo apt install sextractor scamp psfex swarp

or, on Miniconda/Anaconda, like that:

conda install -c conda-forge astromatic-source-extractor astromatic-scamp astromatic-psfex astromatic-swarp

You may also check more detailed installation instructions here.

Usage

Please consult the documentation for STDPipe for the basic usage patterns and description of its API. You may check the examples inside notebooks/ folder, especially the tutorial that demonstrates basic steps of a typical image processing.

stdpipe's Issues

Implementation on EP alerts with FRAM

Hello, is it possible to update FRAM code to have automatic triggering of FRAM for EP ?

cutouts

It would be nice if there was an option to overlay a circular aperture indicating the object in the center of the cutout.

calibrate_photometry

Perhaps you should have an early stopping criteria in the calibrate_photometry?

33 objects pass initial quality cuts
Iteration 0 : 33 / 35 - rms 0.45 0.45 - normed 13.73 2.62 - scale 4.02 4.02 - rms 0.02
Iteration 1 : 29 / 35 - rms 0.35 0.36 - normed 39.99 4.43 - scale 1.53 1.53 - rms 0.02
Iteration 2 : 19 / 35 - rms 0.30 0.17 - normed 211.11 3.47 - scale 1.00 1.00 - rms 0.02
Iteration 3 : 17 / 35 - rms 0.29 0.07 - normed 13.57 21.22 - scale 0.63 0.63 - rms 0.00
Iteration 4 : 14 / 35 - rms 0.30 0.08 - normed 0.86 1.55 - scale 0.55 0.55 - rms 0.00
Iteration 5 : 13 / 35 - rms 0.29 0.04 - normed 0.37 1.63 - scale 0.23 0.23 - rms 0.00
Iteration 6 : 12 / 35 - rms 0.29 0.02 - normed 0.20 347.96 - scale 0.00 0.00 - rms 0.00
/Users/mcoughlin/opt/anaconda3/envs/skyportal/lib/python3.9/site-packages/statsmodels/regression/linear_model.py:1671: RuntimeWarning: divide by zero encountered in double_scalars
/Users/mcoughlin/opt/anaconda3/envs/skyportal/lib/python3.9/site-packages/statsmodels/regression/_tools.py:121: RuntimeWarning: divide by zero encountered in double_scalars
Future exception was never retrieved
future: <Future finished exception=ZeroDivisionError('float division by zero')>
Traceback (most recent call last):
File "/Users/mcoughlin/opt/anaconda3/envs/skyportal/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/mcoughlin/Code/ZTF/skyportal-branches/kao/skyportal/skyportal/handlers/api/internal/image_analysis.py", line 710, in
lambda: reduce_image(
File "/Users/mcoughlin/Code/ZTF/skyportal-branches/kao/skyportal/skyportal/handlers/api/internal/image_analysis.py", line 348, in reduce_image
m = pipeline.calibrate_photometry(
File "/Users/mcoughlin/Code/ZTF/stdpipe-branches/astropy_columns/stdpipe/stdpipe/pipeline.py", line 324, in calibrate_photometry
m = photometry.match(obj['ra'], obj['dec'], obj[obj_col_mag],
File "/Users/mcoughlin/Code/ZTF/stdpipe-branches/astropy_columns/stdpipe/stdpipe/photometry.py", line 618, in match
C = sm.RLM(zero[idx]/total_err[idx], (X[idx].T/total_err[idx]).T).fit()
File "/Users/mcoughlin/opt/anaconda3/envs/skyportal/lib/python3.9/site-packages/statsmodels/robust/robust_linear_model.py", line 301, in fit
results = RLMResults(self, wls_results.params,
File "/Users/mcoughlin/opt/anaconda3/envs/skyportal/lib/python3.9/site-packages/statsmodels/robust/robust_linear_model.py", line 413, in init
self.cov_params_default = self.bcov_scaled
File "pandas/_libs/properties.pyx", line 36, in pandas._libs.properties.CachedProperty.get
File "/Users/mcoughlin/opt/anaconda3/envs/skyportal/lib/python3.9/site-packages/statsmodels/robust/robust_linear_model.py", line 450, in bcov_scaled
return k ** 2 * (1 / self.df_resid * ss_psi * self.scale ** 2) /
ZeroDivisionError: float division by zero

bg_mask.multiply(image1)

https://github.com/karpov-sv/stdpipe/blob/master/stdpipe/photometry.py#L806

I get:

bg_vals = bg_mask.multiply(image1)[bg_mask.data > 0]

TypeError: 'NoneType' object is not subscriptable

I know it works sometimes, but should do some handling, i.e. check that bg_mask.multiply(image1) is not None and then get the array.