run on new PSF simulations about piff

[esheldon]

I'm willing to do the SExtractor run if anyone can give me an appropriate config file and point me at the recommended version of the code

[cpadavis]

I'm also willing to help out and run these at SLAC. For SExtractor and PSFEx I also just need the appropriate config files...

[esheldon]

I asked a question about config files and code version on the DES balrog slack channel.

[esheldon]

Eric Suchyta says:

For Y1 it’s 2.18.10

And config files can be found here

https://desar2.cosmology.illinois.edu/DESFiles/desarchive/OPS/config/20150806/finalcut/

In particular

20150806_sex.nnw
20150806_sex.conv
20150806_sex.config
20150806_sex.param.finalcut.20130702

The exact call used in Y1 by DESDM (note the date prefix has been stripped from the config file names)

/usr/apps/des/stacks/7.2-stacks/7.2.6/des_prereq/bin/sex /projects/des/Archive/OPS/red/20140715085529_20131130/red/DECam_00258910/DECam_00258910_01.fits[0] -c /usr/apps/des/software/7.2-level/7.2.18+1/des_home/etc/sex.config  -FILTER_NAME /usr/apps/des/software/7.2-level/7.2.18+1/des_home/etc/sex.conv -STARNNW_NAME /usr/apps/des/software/7.2-level/7.2.18+1/des_home/etc/sex.nnw -INTERP_TYPE VAR_ONLY  -WEIGHT_TYPE MAP_WEIGHT -WEIGHT_IMAGE /projects/des/Archive/OPS/red/20140715085529_20131130/red/DECam_00258910/DECam_00258910_01.fits[2],/projects/des/Archive/OPS/red/20140715085529_20131130/red/DECam_00258910/DECam_00258910_01.fits[2]  -SEEING_FWHM 1.034  -FLAG_IMAGE /projects/des/Archive/OPS/red/20140715085529_20131130/red/DECam_00258910/DECam_00258910_01.fits[1]  -CATALOG_NAME /projects/des/Archive/OPS/red/20140715085529_20131130/red/DECam_00258910/DECam_00258910_01_cat.fits -SATUR_LEVEL 38652.0000  -PARAMETERS_NAME /usr/apps/des/software/7.2-level/7.2.18+1/des_home/etc/sex.param -PSF_NAME /projects/des/Archive/OPS/red/20140715085529_20131130/red/DECam_00258910/DECam_00258910_01_psfcat.psf  -CHECKIMAGE_TYPE SEGMENTATION -CHECKIMAGE_NAME /projects/des/Archive/OPS/red/20140715085529_20131130/QA/DECam_00258910/DECam_00258910_01_seg.fits  -DETECT_THRESH 3.0 -PARAMETERS_NAME /usr/apps/des/software/7.2-level/7.2.18+1/des_home/etc/sex.param.finalcut.20130702

[esheldon]

The simulated images and truth files are here

http://www.cosmo.bnl.gov/Private/gpfs/workarea/esheldon/lensing/des-lensing/psfsim/v006/output/

Or at BNL

/gpfs/mnt/gpfs01/astro/workarea/esheldon/lensing/des-lensing/psfsim/v006/output

You can use the psfsim package to get filenames if you set the PSFSIM_DIR environment variable

https://github.com/esheldon/psfsim

[esheldon]

@cpadavis Can you do the SExtractor run?

@rmjarvis would you be willing to run PSFEx once we get catalogs? I was thinking it might not be too hard for you since you have been running PSFEx on real data

[rmjarvis]

Sure. I'm on vacation for the next 10 days in London with the family, but I'll try to carve out a little time to start that run at BNL.

[rmjarvis]

Should we start with the desdm psfex config file, or the v2 one? (Or maybe modified to have oversampling = 0.6 rather than 0.5.)

[esheldon]

I think we should test things we have run on the real data first. y1a-v02 is supposed to give the best rho stats of the ones you tried so far, so let's test that first.

[esheldon]

@cpadavis do you have a timescale over which you can run SExtractor on the simulations?

[cpadavis]

Just finished downloading.

I had to modify the config files you suggested I use because they expect *.psf, flag files, and weight maps. I also could not find SExtractor 2.18.10 on the sextractor website, so I'm just going with the most recent version of sextractor on there for now... (2.19.5)

The two things I am not sure on are:

• should I set the WEIGHT_TYPE to NONE or BACKGROUND?
• do I want to be using SEEING_FWHM of 1.034 (as in the command above) or something different?

I'm running sextractor now and should have a run with weight_type as none and seeing_fwhm=1.034 before the end of today

[cpadavis]

I forget -- for PSFEx, what column parameters do there need to be? I recall needing VIGNET, which I think isn't in the config file

[esheldon]

Regarding the SExtractor run: The noise is currently constant for all images and is set in the galsim config file based on a sky level and read noise

https://github.com/esheldon/psfsim_config/blob/master/psfsim-stars-v006.yaml

I don't have any experience running SExtractor so I can't help much more.

[esheldon]

As for PSFEx, we had planned to try running with the standard desdm config as well as mike's y1a1-v02 config.

I don't have a copy of that config, but there are notes here describing the changes needed

https://github.com/rmjarvis/DESWL/blob/master/psfex/notes.txt

[cpadavis]

OK catalogs with vignets done. I feel silly doing this because I don't have BNL access, so with the vignets you have to transfer ~100 gigs.

Config files:
http://www.slac.stanford.edu/~cpd/psfsim/20150806_sextractor_configuration_files/

Catalog entries without vignets:
http://www.slac.stanford.edu/~cpd/psfsim/catalogs/

catalog entries with vignets:
http://www.slac.stanford.edu/~cpd/psfsim/catalogs_psfex/

[esheldon]

I can copy them over.

sorry, I'm not that familiar with these codes. How do these pieces fit together? Is this the input Mike needs to run PSFEx?

[cpadavis]

This should be what Mike needs to run PSFex, unless I messed up.

Here is the sextractor pipeline: You take your image, run sextractor to get
a catalog. You can run psfex directly on the catalog and it will try to
determine the stars, but iirc Mike has a program that does a better job
finding the stars. So Mike will run findstars and then run psfex on that
output. Then you can run sextractor again on the base catalog + .psf
output to determine the values of columns which depend on the PSF (e.g.
MAG_PSF, SPREAD_MODEL)

On Mon, Aug 1, 2016 at 7:29 AM Erin Sheldon [email protected]
wrote:

I can copy them over.

sorry, I'm not that familiar with these codes. How do these pieces fit
together? Is this the input Mike needs to run PSFEx?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#33 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABsw5Ir1Y7Ywjt2HBmaJ99RfXZQDvoN4ks5qbgLXgaJpZM4JSwYn
.

[RobertLuptonTheGood]

We can run psfEx using LSST wrappers (use a command line driver called processFile or write a simple python wrapper) -- that means that you don't need to separately run SExtractor, and we've separated the star selection from the psf estimation

                        R

[esheldon]

At this stage I don't think we care about the sextractor measured quantities based on the PSF model, so a rerun is not necessary in any case.

It will be good to see the result using the LSST wrapper as well. We have never been quite sure if we are using PSFEx correctly, even after consulting Bertin.

[esheldon]

@cpadavis can you post the locations on the slac file system (rather than the web)? It will be more efficient to transfer via rsync.

[cpadavis]

config files at:

/nfs/slac/g/ki/ki19/des/cpd/20150806_sextractor_configuration_files

sextractor catalogs you can run psfex on at:

/nfs/slac/g/ki/ki19/des/cpd/PSFSIM/v006/output/catalogs_psfex

On Tue, Aug 2, 2016 at 10:52 AM Erin Sheldon [email protected]
wrote:

@cpadavis https://github.com/cpadavis can you post the locations on the
slac file system (rather than the web)? It will be more efficient to
transfer via rsync.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#33 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABsw5JGj8E6KDQE0oMpUYke1eT099Bprks5qb4PtgaJpZM4JSwYn
.

[esheldon]

@rmjarvis

Chris' catalogs are here:

/gpfs/mnt/gpfs01/astro/workarea/esheldon/lensing/des-lensing/psfsim/v006-sextractor/output

the files seem to have patterns like this psfsim-stars-v006-003767_cat.fits so the same as the simulation files but with _cat.fits

[rmjarvis]

OK. I'm back from London now (vacation after the Oxford meeting), but I need to go to California for a funeral this weekend, so it will probably be next week before I get a chance to work on this.

[rmjarvis]

I finished running PSFEx on the sims. The rho stats look much better than what we are getting on the data.

For comparison, here is the y1a1-v02 run on DES Y1 r-band data:

In fact, it's even slightly worse than depicted on the data, since the data run gets to take advantage of overlapping exposures beating down the power by a factor of nexp ~= 3, which I didn't do for the simulation run -- this is just the mean rho1 power averaged over all 20K simulation images.

So I think this means that whatever is causing us problems with PSFEx on the data is something that isn't yet included in the simulations.

[cpadavis]

Are we modeling inter-ccd correlations in the PSF? Could that cause it in
the actual data?

On Wed, Aug 24, 2016 at 6:30 PM Mike Jarvis [email protected]
wrote:

I finished running PSFEx on the sims. The rho stats look much better than
what we are getting on the data.
[image: rho1]
https://cloud.githubusercontent.com/assets/623887/17953442/523a146c-6a41-11e6-96ab-1d50ace6166a.png

For comparison, here is the y1a1-v02 run on DES Y1 r-band data:
[image: rho1_all_r]
https://cloud.githubusercontent.com/assets/623887/17953459/762e942e-6a41-11e6-82e8-14961ec75892.png

In fact, it's even slightly worse than depicted on the data, since the
data run gets to take advantage of overlapping exposures beating down the
power by a factor of nexp ~= 3, which I didn't do for the simulation run --
this is just the mean rho1 power averaged over all 20K simulation images.

So I think this means that whatever is causing us problems with PSFEx on
the data is something that isn't yet included in the simulations.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#33 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABsw5CVtXkulDj7fvRpbdn5OvYSemty1ks5qjPAggaJpZM4JSwYn
.

[esheldon]

aaron pointed out our variation across the ccd is too small by a factor of order 5 I recall. Looks like that would still not make up the difference with real data though.

[rmjarvis]

I think the main cause of the rho stats in the real data is from the WCS. PSFEx does things in CCD coordinates, remember, so it has to include the effect of the WCS on the PSF. This was one of the changes we made for Piff -- doing things in sky coordinates means we can remove the effect of the WCS from the PSF profiles before trying to interpolate them.

In these sims, the WCS are all simple Jacobians. We could make that more realistic by copying the WCS from data images, which are 3rd order. (And reality is probably closer to 5th order across the focal plane, but piecewise 3rd order might be an ok approximation to that.)

[esheldon]

In these sims I used an example wcs from a des exposure

See https://github.com/esheldon/psfsim_config/blob/master/psfsim-stars-v006.yaml

[esheldon]

So, is that actually being used in full? @rmjarvis your comment seems to imply it only takes the jacobian?

[rmjarvis]

No, they use the local WCS at a random location within a DES exposure. They don't use the full WCS solution. That particular wcs class was designed for use with a MEDS sim where we just wanted a local wcs for each galaxy. Here is what we probably want, which takes the full wcs from a DES image:

    wcs :
        type : Fits
        dir:
            type: FormattedStr
            format: "%s/OPS/red/20140715085838_20131201/red/DECam_00259397"
            items:
                - *DESDATA
        file_name:
            type : NumberedFile
            root : 'DECam_00154912_'
            digits : 2
            ext : '.fits'
            num :
                # Pick a random chip, but not either of our bad chips.
                type : ExcludedRandom  # This is a custom type defined in exclude_random.py
                min: 1
                max : 62
                exclude: [ 61, 31 ]

You can grab exclude_random.py from the galsim/examples/des directory.

[esheldon]

Would we also remove the des_wcs: entry under input: ?

[esheldon]

(or remove input: altogether?)

[rmjarvis]

Sure. It wouldn't be an error to leave it in, but it's not necessary anymore.

[esheldon]

OK, it sounds like we should run a new sim. Should we wait for Aaron's new variation numbers?

[rmjarvis]

Sure. Might as well try to get that right too.

We could also start making some of the other changes we talked about, like doing 62 (or 60) chip exposures and matching the positions/fluxes of objects from the real data catalogs to get more realistic distributions of those quantities. Probably still without galaxies I guess to keep that part simple.

[aaronroodman]

Here are some new numbers for variation of Wavefront terms. All values are in waves at 700nm.

I calculated the Mean and Spread == Max-Min for each Zernike term, spatially, inside each of the 62 CCDs .
Note that the variation across each sensor is not very Gaussian - a uniform distribution is closer, thats why
I calculated a spread. See plot below for these quantities.

Then to characterize the Means and Spreads, I calculated the width of the Means and range of the Spreads
over the 62 CCDs. The Means of the Means are zero. Here are those values:

Zern 4, Stdev of Mean = 0.085, Lo,Hi of Spread = 0.043,0.212
Zern 5, Stdev of Mean = 0.062, Lo,Hi of Spread = 0.018,0.259
Zern 6, Stdev of Mean = 0.054, Lo,Hi of Spread = 0.019,0.165
Zern 7, Stdev of Mean = 0.042, Lo,Hi of Spread = 0.011,0.152
Zern 8, Stdev of Mean = 0.044, Lo,Hi of Spread = 0.011,0.092
Zern 9, Stdev of Mean = 0.144, Lo,Hi of Spread = 0.011,0.236
Zern 10, Stdev of Mean = 0.136, Lo,Hi of Spread = 0.008,0.249
Zern 11, Stdev of Mean = 0.073, Lo,Hi of Spread = 0.015,0.103

So, use the “Stdev of Mean” values to generate a Mean for each term for the CCD, then pick uniformly inside
the Lo,Hi to pick a Spread. And then use that Spread to set the range of that term around the Mean.

You see that the Spread can be biggish - at 0.2 waves - which is the thing I was worried about.

Hope that is sorta clear,

Aaron

[data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAoAAAAyACAYAAABc8rJ+AAAABHNCSVQICAgIfAhkiAAAIABJREFUeJzs3XmcZFVh9//PF8iACzMjIKIyBASSR3FLNDEG1J+JpiMuMUYlEQkkMXn0h1Hc8pPxEZcYXILGkGCMURSBqElwgaiMEsU1ilFRfCCKyDKAoCwzPSzDGDi/P+5tprqmqru6u7qru87n/XrVq7ruPfeec5c6/a17b91KKQVJkiTVY6dRN0CSJElLywAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgAoSZJUGQOgJElSZQyAkiRJlTEASpIkVcYAKEmSVBkDoCRJUmUMgJIkSZUxAEqSJFXGAChJklQZA6AkSVJlDICSJEmVMQBKkiRVxgA4hpIcneSu9nFQj/GP7xj/G6No41JK8vPtsv7hqNvSKcn7O7ZDv8cJy6Cdr27b8sVRt0UzS/LMJF9Icn2S25JckeRjSSZG3bb5aNt/6ixlntDun49fqnbNxXJ5H3dK8roB+p4Z1/sStfP327ZcNeq2jKNdRt0ALapJ4CjgdV3Dj27H7b7kLVKnNwL/0GfcPwCHAOcuXXN2lORBwGuA60fZDs0uyUuAdwLvBd4G3AocCDwVeCKwYXStm7cyQJlvAr8GXLzIbRkn/wR8us+4E4CnAGcvXXN2lGQN8DfAj0fZjnFmABxvHwWeT0cATLIb8Gzg34BjRtMsAZRSLgcu7x6e5Djgl4DjSikXLLSeJKtKKdvmOfm7gDOA/wXsvNC2aFG9AvhoKeXPOoadD7xvWBUk2aWU8j/Dmt8wlFJuARb8PqlJKeVa4Nru4UmeCRwOvLOU8vGF1rPA/eWvgQuB64DfXGhbtCNPAY+vApwO7J/k0I7hzwICnNU+T9OeTjkvyWSSW5Kcm+SQrjJPTvLJJNcmuTXJRUlenmSnrnKXJzk9yRFJLm7n942u9uwgybPbw/4P7THuU0m+3fH62CRfTXJjkpuT/GeSw2dbOUnOT/K5HsN3OOWUZP8kZyb5SZKtSb7ddpSdZQ5uT7Vdn+T2JFcm+Uj3OhmgXY8F3gqcVUo5uWvcPZK8NcmPktzRPq9Pko4yU6fDfjfJe5L8hKYDJcnrpy4LSPLvSba0y/vaPm15Hk0QPX4uy6CR2YMBjtRm+yUij2v32S1Jbkjy9+0HxKlyU5dOvKjd764BtrZHZgZ9XxyY5IPtvnpbksuSvCvJ2h7temnbZ9ye5IIkhw2y0OlxCrh9f38pyW8m+WZHP/XMWeb16HZeT+sx7l3t+3vn9vURSf6jXf4tSb6VAS4zSfKBJL0++O3QJyXZK8m7k1zdruNLkvxpV5n7JTktyTVtmWuTnJ1kr9na0jWfA4H3A18D/qJr3M5Jjm/r39rWdVKSXTvK9N1fkhzTjntMkjOSbG7n8bdJVvVoy6HA84Bj57IMmhuPAI63K4Ev0pwG/ko77CjgYzSnh6ZJ8lTg48A5wJHt4FcDX0rysFLKNe2wBwGfB05p5/NomqOMewHru2b7OOAXaE4j3gG8CTgnyf6llMk+7T4H2Exz9PLVHe3bG3gy8KqOsvvTdFqX0Ryheno7/6eUUj7TZ/7Q/9TStOFJ9qU5unAd8FLgBuAI4Kwkv1NK+fe26KeAG4H/3T4/kOaT9E7AXTO0o7OuPYF/odluf9w1bmfgMzRH4t4IfI/mtNcJwH2Yvk4ATq]

On Aug 25, 2016, at 9:03 AM, Erin Sheldon <[email protected]mailto:[email protected]> wrote:

OK, it sounds like we should run a new sim. Should we wait for Aaron's new variation numbers?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/33#issuecomment-242443075, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEEa8zmWu0EECrydFO_WJ3AkM7iaQByKks5qjbztgaJpZM4JSwYn.

---

Prof. Aaron Roodman
Chair Dept. of Particle Physics & Astrophysics
SLAC National Accelerator Laboratory
Stanford University

SLAC National Accelerator Laboratory E-mail: [email protected]:[email protected]
2575 Sand Hill Rd. Phone: [phone]
MS 29
Menlo Park, CA 94025 URL: http://www.slac.stanford.edu/~roodman

---

[rmjarvis]

Thanks, Aaron!

The image you posted didn't go through. It it was important, maybe try editing the post on GitHub, and dragging the image onto the comment window.

[rmjarvis]

The Means of the Means are zero.

@aaronroodman: Is this really true? This surprises me. I would have thought for at least some of the aberrations, the mean would be non-zero.

[rmjarvis]

@esheldon I think this is an approximate translation of the prescription Aaron gave into the galsim config file. I'm not sure that the way we're doing the polynomials will give exactly the right spreads, but I think it should be close. There might be an extra factor of O(1) that needs to be applied.

eval_variables:
    fmean_4: { type: RandomGaussian, sigma: 0.085, index_key: file_num }
    fspread_4: { type: Random, min: 0.043, max: 0.212, index_key: file_num }
    fmean_5: { type: RandomGaussian, sigma: 0.062, index_key: file_num }
    fspread_5: { type: Random, min: 0.018, max: 0.259, index_key: file_num }
    fmean_6: { type: RandomGaussian, sigma: 0.054, index_key: file_num }
    fspread_6: { type: Random, min: 0.019, max: 0.165, index_key: file_num }
    fmean_7: { type: RandomGaussian, sigma: 0.042, index_key: file_num }
    fspread_7: { type: Random, min: 0.011, max: 0.152, index_key: file_num }
    fmean_8: { type: RandomGaussian, sigma: 0.044, index_key: file_num }
    fspread_8: { type: Random, min: 0.011, max: 0.092, index_key: file_num }
    fmean_9: { type: RandomGaussian, sigma: 0.144, index_key: file_num }
    fspread_9: { type: Random, min: 0.011, max: 0.236, index_key: file_num }
    fmean_10: { type: RandomGaussian, sigma: 0.136, index_key: file_num }
    fspread_10: { type: Random, min: 0.008, max: 0.249, index_key: file_num }
    fmean_11: { type: RandomGaussian, sigma: 0.073, index_key: file_num }
    fspread_11: { type: Random, min: 0.015, max: 0.103, index_key: file_num }

psf:
    # [ snip... ]
        defocus: 
            type: Eval
            str: "mean_4 + a * image_pos.x + b * image_pos.y + c * image_pos.x**2 + d * image_pos.y**2 + e * image_pos.x*image_pos.y"
            fa: { type: RandomGaussian, sigma: spread_4/2048., index_key: file_num }
            fb: { type: RandomGaussian, sigma: spread_4/4096., index_key: file_num }
            fc: { type: RandomGaussian, sigma: spread_4/2048**2, index_key: file_num }
            fd: { type: RandomGaussian, sigma: spread_4/2048/4096, index_key: file_num }
            fe: { type: RandomGaussian, sigma: spread_4/4096**2, index_key: file_num }

        astig1: 
            type: Eval
            str: "mean_5 + a * image_pos.x + b * image_pos.y + c * image_pos.x**2 + d * image_pos.y**2 + e * image_pos.x*image_pos.y"
            fa: { type: RandomGaussian, sigma: spread_5/2048., index_key: file_num }
            fb: { type: RandomGaussian, sigma: spread_5/4096., index_key: file_num }
            fc: { type: RandomGaussian, sigma: spread_5/2048**2, index_key: file_num }
            fd: { type: RandomGaussian, sigma: spread_5/2048/4096, index_key: file_num }
            fe: { type: RandomGaussian, sigma: spread_5/4096**2, index_key: file_num }

        astig2: 
            type: Eval
            str: "mean_6 + a * image_pos.x + b * image_pos.y + c * image_pos.x**2 + d * image_pos.y**2 + e * image_pos.x*image_pos.y"
            fa: { type: RandomGaussian, sigma: spread_6/2048., index_key: file_num }
            fb: { type: RandomGaussian, sigma: spread_6/4096., index_key: file_num }
            fc: { type: RandomGaussian, sigma: spread_6/2048**2, index_key: file_num }
            fd: { type: RandomGaussian, sigma: spread_6/2048/4096, index_key: file_num }
            fe: { type: RandomGaussian, sigma: spread_6/4096**2, index_key: file_num }

        coma1: 
            type: Eval
            str: "mean_7 + a * image_pos.x + b * image_pos.y + c * image_pos.x**2 + d * image_pos.y**2 + e * image_pos.x*image_pos.y"
            fa: { type: RandomGaussian, sigma: spread_7/2048., index_key: file_num }
            fb: { type: RandomGaussian, sigma: spread_7/4096., index_key: file_num }
            fc: { type: RandomGaussian, sigma: spread_7/2048**2, index_key: file_num }
            fd: { type: RandomGaussian, sigma: spread_7/2048/4096, index_key: file_num }
            fe: { type: RandomGaussian, sigma: spread_7/4096**2, index_key: file_num }

        coma2: 
            type: Eval
            str: "mean_8 + a * image_pos.x + b * image_pos.y + c * image_pos.x**2 + d * image_pos.y**2 + e * image_pos.x*image_pos.y"
            fa: { type: RandomGaussian, sigma: spread_8/2048., index_key: file_num }
            fb: { type: RandomGaussian, sigma: spread_8/4096., index_key: file_num }
            fc: { type: RandomGaussian, sigma: spread_8/2048**2, index_key: file_num }
            fd: { type: RandomGaussian, sigma: spread_8/2048/4096, index_key: file_num }
            fe: { type: RandomGaussian, sigma: spread_8/4096**2, index_key: file_num }

        trefoil1: 
            type: Eval
            str: "mean_9 + a * image_pos.x + b * image_pos.y + c * image_pos.x**2 + d * image_pos.y**2 + e * image_pos.x*image_pos.y"
            fa: { type: RandomGaussian, sigma: spread_9/2048., index_key: file_num }
            fb: { type: RandomGaussian, sigma: spread_9/4096., index_key: file_num }
            fc: { type: RandomGaussian, sigma: spread_9/2048**2, index_key: file_num }
            fd: { type: RandomGaussian, sigma: spread_9/2048/4096, index_key: file_num }
            fe: { type: RandomGaussian, sigma: spread_9/4096**2, index_key: file_num }

        trefoil2: 
            type: Eval
            str: "mean_10 + a * image_pos.x + b * image_pos.y + c * image_pos.x**2 + d * image_pos.y**2 + e * image_pos.x*image_pos.y"
            fa: { type: RandomGaussian, sigma: spread_10/2048., index_key: file_num }
            fb: { type: RandomGaussian, sigma: spread_10/4096., index_key: file_num }
            fc: { type: RandomGaussian, sigma: spread_10/2048**2, index_key: file_num }
            fd: { type: RandomGaussian, sigma: spread_10/2048/4096, index_key: file_num }
            fe: { type: RandomGaussian, sigma: spread_10/4096**2, index_key: file_num }

        spher: 
            type: Eval
            str: "mean_11 + a * image_pos.x + b * image_pos.y + c * image_pos.x**2 + d * image_pos.y**2 + e * image_pos.x*image_pos.y"
            fa: { type: RandomGaussian, sigma: spread_11/2048., index_key: file_num }
            fb: { type: RandomGaussian, sigma: spread_11/4096., index_key: file_num }
            fc: { type: RandomGaussian, sigma: spread_11/2048**2, index_key: file_num }
            fd: { type: RandomGaussian, sigma: spread_11/2048/4096, index_key: file_num }
            fe: { type: RandomGaussian, sigma: spread_11/4096**2, index_key: file_num }

[esheldon]

Thanks @aaronroodman and @rmjarvis

Yesterday I ran a version 007 with a full wcs but without the variations from Aaron. I'll run a version 008 today that also includes the above spread.

[esheldon]

I'm finding it hard to picture exactly why the values Aaron gave are also consistent with the plain std. dev. that @aaronroodman had given previously. I think the figure would help. Can you please repost the figure? The easiest way is to click under the message "Attach files by selecting them"

[esheldon]

@rmjarvis I'm getting an error with this new config.

ValueError: could not convert string to float: spread_5/2048.

[esheldon]

Here is the config: https://github.com/esheldon/psfsim_config/blob/master/psfsim-stars-v008.yaml

[rmjarvis]

Sorry. All those sigma definitions need to be wrapped in "$...". e.g. "$spread_5/2048." They need to be evaluated, and the $ prefix on the string is the thing that instructs GalSim to do so.

[rmjarvis]

Also, I just noticed I got the normalization for the d and e parameters reversed. d is the y^2 coefficient, so should be spread_*/4096**2. e is the xy coefficient, so it should be spread_*/2048./4096.

[rmjarvis]

Although it's possible that we don't want to do that. It might be better to have them all use 2048**2 normalization (and have a,b use 2048.) and just let the y direction run a bit more. I'm not sure which is actually more realistic.

[esheldon]

Thanks, after those fixes the config runs

[aaronroodman]

ah yes I left something out -

The numbers I sent characterize our “ideal” wavefront, where the means are defined to be zero.

On top of that I should add an additional Gaussian smearing to characterize the variation from image to image. And for that contribution the means have not been zero - although they should be going forward. Year 1 and part of 2 have non-zero mean Coma, and the mean of Astigmatism has been non-zero too.

As of two nights ago, the Blanco primary astigmatism is under active control by the donut system - hurrah!!

I will supply values for this additional smearing.

On Aug 25, 2016, at 9:15 PM, Mike Jarvis <[email protected]mailto:[email protected]> wrote:

The Means of the Means are zero.

@aaronroodmanhttps://github.com/aaronroodman: Is this really true? This surprises me. I would have thought for at least some of the aberrations, the mean would be non-zero.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/33#issuecomment-242626357, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEEa86hamhPJ_sRrnCMkVoc_YRggC5f8ks5qjmh3gaJpZM4JSwYn.

---

Prof. Aaron Roodman
Chair Dept. of Particle Physics & Astrophysics
SLAC National Accelerator Laboratory
Stanford University

SLAC National Accelerator Laboratory E-mail: [email protected]:[email protected]
2575 Sand Hill Rd. Phone: [phone]
MS 29
Menlo Park, CA 94025 URL: http://www.slac.stanford.edu/~roodman

---

[esheldon]

I'm running a v008 now based on the current config; we can incorporate more things later but I wanted to get something out before the weekend.

So there is already v007 and now v008 is coming, probably done in a few hours. So @rmjarvis can process these at his leisure.

[rmjarvis]

FYI, I started a Piff run on v009. It's substantially slower than PSFEx (not surprising, since we haven't tried to do any optimization yet), but I figured it was worth trying it to see how it compares to PSFEx.

Here are the rho stats for the best PSFEx result so far.

[rmjarvis]

I've finally managed to get a reasonable result from Piff on the same simulation. It took a while to track down some subtle errors in the I/O and draw routine when drawing from a file that's been written to disk. It's not quite as good as PSFEx yet, but at least it's in the same ball park now. (Early versions of this plot were off the top of the range for most of them.)

Here is the Piff config file I used for this. (Although all the 000000 values are changed on the command line when I run it to go through all the different images.) I'll try to few adjustments to see what parameters make things better or worse and report back.

[rmjarvis]

I iterated on this for a while and here is the best result I found:


This uses the following piff configuration:

input:
    image_dir: '/gpfs/mnt/gpfs01/astro/workarea/esheldon/lensing/des-lensing/psfsim/v009/output'
    images: 'psfsim-stars-v009-000000.fits'
    image_hdu: 0
    cat_dir: '/astro/u/mjarvis/work/sims/psfsim-stars-v009/v01'
    cats: 'psfsim-stars-v009-000000_psfcat.fits'
    cat_hdu: 2

    x_col: XWIN_IMAGE
    y_col: YWIN_IMAGE
    gain: 4.0
    noise: 469.  # Variance in ADU per pixel.
    sky: 231.    # Sky level in ADU per pixel.

    stamp_size: 31

output:
    dir: '/astro/u/mjarvis/work/sims/psfsim-stars-v009/v04'
    file_name: 'psfsim-stars-v009-000000_piff.fits'

psf:

    model:
        type: PixelGrid
        scale: 0.20
        size: 17
        start_sigma: 0.4

    interp:
        type: BasisPolynomial
        order: 3

    outliers:
        type: Chisq
        nsigma: 4
        max_remove: 3

verbose: 2

The file names and directories are updated on the command line, so those are just the first file in the list. This was a useful exercise, so thanks for making these sims, @esheldon.

I've now moved on to trying this on real data (DES Y1A1), where it seems to have some new problems for me to figure out, so I'm going to close this issue.