Develop generic code to do statistics on the reconstructions about piff

[rmjarvis]

There already is code in Piff to do rho statistics on a single Piff run (image or set of images). It wouldn't be too hard to extend it to be able to read in multiple Piff solutions and do a single set of rho statistics on all of them at once.

[cpadavis]

Piff is supposed to already have a general framework for doing the statistics -- hence the statistics class. We have rho stats, 1d hists, and a PR for 2d histograms. What other statistics are you envisioning? I can think of a few:

1. Whisker plots (easy to implement once the 2d histogram PR gets accepted)
2. Correlations with other properties of stars (e.g. the true shears)

We do need to extend piff to be able to read in multiple piff solutions and do the stats on them all at once. I think the stars are not saved currently? If we added an option for saving the stars and fit properties, then reading in multiple piff solutions would be pretty easy to implement...

[RobertLuptonTheGood]

Another plea for modularity! Why do these belong in piff? Look at the LSST/astropy use case: people will use piff to estimate PSFs, but have their own analysis/stats code. I'd personally put the PSF estimation into a core package and other things (including star selection) into others; but even if we don't go that far at least can we make things totally separate at the code level (with, possibly, a common base package)

[esheldon]

I didn't realize this was already a submodule/package in PIFF. I had imagined a separate repo.

[esheldon]

New statistics: we have some mysteries in DES data that we can potentially clear up using the simulations. E.g the rho statistics look better for certain psfex parameters, but using the same parameters our measurements of flux are wrong. Maybe this can be clarified with the right statistic, given we know the truth.

[cpadavis]

Regardless of whether we do the stats within PIFF or migrate it elsewhere, I went and checked what is currently saved in the piff output. Stars are indeed saved (with the fitted params), except for cutouts. I have not tested whether piff can take these saved stars and the saved psf and use its stats code just from a .piff file.

[RobertLuptonTheGood]

That statement tells us that splitting the code will improve piff. I'd have thought it was essential that all such tests can be done off persisted piff products.

[cpadavis]

Agreed that we should be able to do all tests off persisted piff products!

As it stands I believe we couldn't compute rho stats off the .piff file because piff doesn't save the size/shape of the star.

[esheldon]

But note that saving size/shape locks one into that method of measuring the size and shape. What if we decide later that isn't a good way to measure size and shape?

[cpadavis]

well regardless you can't measure the size/shape in a manner that is independent of your model of the star and not locked into one method of measuring size/shape if you don't save the image :)

(currently piff computes the rho stats by going to the star image and computing the size/shape there via the gaussian model)

[rmjarvis]

Another plea for modularity! Why do these belong in piff? Look at the LSST/astropy use case: people will use piff to estimate PSFs, but have their own analysis/stats code. I'd personally put the PSF estimation into a core package and other things (including star selection) into others; but even if we don't go that far at least can we make things totally separate at the code level (with, possibly, a common base package)

The logic of putting it in Piff is that for real data, there is utility in having some way to blacklist particular exposures based on various measured statistics of the PSF solution. This is essentially part of the PSF solution (i.e. how much to we trust the answer).

So in practice, we would typically want to run some statistics on the solution and potentially mark it as unreliable based on the results. I think it's useful to keep this in Piff.

However, this doesn't preclude the option for people to also run statistics separately to see how good the solution is overall. I'd actually recommend Stile for a good package that does aggregate statistics. No need to reinvent that.

[esheldon]

My point is it should be straightforward for people to measure what they need given the piff output, rather than trying to measure and store all possible numbers that might be deemed useful later.

[rmjarvis]

I think this conversation ended without a specific plan of action. Closing this issue without prejudice. :) (I.e. Feel free to reopen this or a new one with more specific suggestions for a feature request along these lines.)