moustakas / desi-photometry Goto Github PK
View Code? Open in Web Editor NEWDESI Legacy Imaging Surveys Photometry for observed and potential (DESI) targets.
License: MIT License
desi-photometry's People
Contributors
Stargazers
Watchers
Forkers
karishseebaluckdesi-photometry's Issues
release 9010 vs 9012 bug
Sec 5.3 of the targeting paper documents a bug which affects targets observed in SV1--
https://arxiv.org/pdf/2208.08518.pdf
This bug impacts the Fuji/v2.0 catalogs but a fix was implemented in desihub/desispec#2057.
validate the completeness of the potential-targets vs observed-targets tables
Need to double-check that:
- If every target in targetphot-edr.fits is also in targetphot-potential-edr.fits AND
- Every row in targetphot-edr.fits has identical values to the corresponding row in targetphot-potential-edr.fits, THEREFORE
- One could create a 100% comprehensive target photometry table in a database by ONLY ingesting targetphot-potential-edr.fits.
consider using a KD tree to match faster
The tractorphot catalogs are matched by parsing targetid but it may be faster to use a KD tree.
update link to Meyers et al. 2022 paper
Currently, the link in the README points to the version of the targeting paper on docDB. Once the paper is on the arXiv (or published) then we'll need to update the link.
add "Known Issues" to the README
Here's some initial text:
For secondary targets in SV1, the targeting catalog filenames recorded in the
fiberassign header are inconsistent with the contents of the corresponding
fibermap catalog for a given TILEID.
include unique DESI identifier
@akremin can you please provide the code needed to build the DESI identifier proposed during the data systems calls?
work harder to extract Tractor photometry for objects in the DECaLS and BASS+MzLS overlap region
See the discussion here:
desihub/fastspecfit#121
Adam's name is spelled incorrectly in the README!
It's Myers not Meyers!
Inconsistent type for gaia_astrometric_params_solved
In the tractorphot files, the column gaia_astrometric_params_solved is a small integer (FITS type B), but in the targetphot files it is a boolean (FITS type L). Is this intentional?
out-of-memory when generating "potential targets" catalogs for Iron
@weaverba137 are the "potential targets" catalogs generated as part of this VAC loaded into a DESI database, or just the observed targets? The issue is that the main survey has become too large to put all the targets into a single catalog (see below), so we either need to drop them (if they're not used) or split them into healpixels (or something along those lines).
time /global/homes/i/ioannis/code/desihub/desi-photometry/lsdr9-photometry --reduxdir ${DESI_ROOT}/spectro/redux/iron -o ${SCRATCH}/lsdr9/iron --specprod iron --mp 128 --targetphot --potential --tractorphot
INFO:lsdr9-photometry:379:main: Reading /pscratch/sd/i/ioannis/lsdr9/iron/ancillary/targetphot-zcat-iron.fits
INFO:lsdr9-photometry:388:main: Read 24,896,930 objects from /pscratch/sd/i/ioannis/lsdr9/iron/ancillary/targetphot-zcat-iron.fits
INFO:lsdr9-photometry:421:main: Found 151,418,861 objects from 6,101 unique tiles
INFO:lsdr9-photometry:485:main: Working on survey cmx
INFO:lsdr9-photometry:506:main: Found 20,956 TARGETIDs and 1 unique tiles from survey: cmx
INFO:lsdr9-photometry:206:targetphot_one: Working on tile 80615
INFO:lsdr9-photometry:533:main: Total time for gather_targetphot = 0.434 min
INFO:lsdr9-photometry:551:main: Writing 20,956 objects to /pscratch/sd/i/ioannis/lsdr9/iron/potential-targets/targetphot-potential-cmx-iron.fits
INFO:lsdr9-photometry:570:main: Writing 20,956 objects to /pscratch/sd/i/ioannis/lsdr9/iron/ancillary/targetphot-potential-zcat-cmx-iron.fits
INFO:lsdr9-photometry:485:main: Working on survey main
INFO:lsdr9-photometry:506:main: Found 133,235,021 TARGETIDs and 5,346 unique tiles from survey: main
INFO:lsdr9-photometry:206:targetphot_one: Working on tile 1000
[snip]
INFO:lsdr9-photometry:206:targetphot_one: Working on tile 41378
INFO:lsdr9-photometry:533:main: Total time for gather_targetphot = 19.952 min
INFO:lsdr9-photometry:551:main: Writing 133,235,021 objects to /pscratch/sd/i/ioannis/lsdr9/iron/potential-targets/targetphot-potential-main-iron.fits
slurmstepd: error: Detected 1 oom-kill event(s) in StepId=5541863.interactive. Some of your processes may have been killed by the cgroup out-of-memory handler.
srun: error: nid004185: task 0: Out Of Memory
srun: launch/slurm: _step_signal: Terminating StepId=5541863.interactive
salloc: Relinquishing job allocation 5541863
salloc: Job allocation 5541863 has been revoked.
regenerate v2.0 catalogs
Now that the fiberassign files have been patched, regenerate the VACs.
add details regarding reproducibility of the VAC
Here's some initial text:
Reproducibility
---------------
Assuming all the relevant DESI environment variables have been set (write me),
one can regenerate the catalogs by cloning this repository and calling:
lsdr9-photometry --reduxdir /path/to/edr -o /path/to/output --outsuffix edr --mp 1 --targetphot
update the fujilupe-v1.0 example notebook
Once the files are in their final location, we need to update the notebook. See this comment for more details.
exclude cmx and sv targeting bits from main-survey catalogs
Right now the stacked targetphot catalogs for Iron and Guadalupe
/global/cfs/cdirs/desi/public/dr1/vac/lsdr9-photometry/guadalupe/v2.0/observed-targets/targetphot-guadalupe.fits
/global/cfs/cdirs/desi/public/dr1/vac/lsdr9-photometry/iron/v1.0/observed-targets/targetphot-iron.fits
include many extraneous cmx/sv targeting bit columns, e.g.,
fitsheader /global/cfs/cdirs/desi/public/dr1/vac/lsdr9-photometry/guadalupe/v2.0/observed-targets/targetphot-guadalupe.fits | grep "_TARGET"
TTYPE106= 'CMX_TARGET'
TTYPE107= 'DESI_TARGET'
TTYPE108= 'BGS_TARGET'
TTYPE109= 'MWS_TARGET'
TTYPE110= 'SV1_DESI_TARGET'
TTYPE111= 'SV1_BGS_TARGET'
TTYPE112= 'SV1_MWS_TARGET'
TTYPE113= 'SV2_DESI_TARGET'
TTYPE114= 'SV2_BGS_TARGET'
TTYPE115= 'SV2_MWS_TARGET'
TTYPE116= 'SV3_DESI_TARGET'
TTYPE117= 'SV3_BGS_TARGET'
TTYPE118= 'SV3_MWS_TARGET'
TTYPE119= 'SCND_TARGET'
TTYPE120= 'SV1_SCND_TARGET'
TTYPE121= 'SV2_SCND_TARGET'
TTYPE122= 'SV3_SCND_TARGET'
DEPNAM19= 'DESI_TARGET'In future versions of these catalogs, we should remove these. Specifically, for Iron and future productions we could make separate sv and main stacked catalogs.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.