Comments (12)
Also, I noticed while playing around with the notebook that a file called tiles-subset.fits appeared in my home directory (the notebook had been previously copied to my home directory). Is this the tiles file that should be used by Progress.get_exposures()
? If so, why is it not written to the same area as progress.fits?
from desisurvey.
This sounds related to changes @sbailey made to desisurvey.config
from desisurvey.
The tiles-subset.fits file is in fact written by the minitest notebook. I'm going to try a test on the basis that the tiles-subset.fits file is what should be the tileinfo
variable defined above.
from desisurvey.
If I explicitly load the desisurvey configuration file with the desisurvey-config.yaml
file in the two_percent_DESI minitest directory, then Progress.get_exposures()
runs successfully.
However, a subsequent sanity check in the minitest notebook then fails:
assert np.all(np.in1d(tiles_subset['TILEID'], explist['TILEID']))
Reassigning to @sbailey, because I don't know what the intended behavior is here.
from desisurvey.
So, I figured out why Progress.get_exposures()
has to read the tiles file. Because Progress
only reads the tiles file if it is initialized without progress='progress.fits'
. If it is initialized with progress='progress.fits'
, then it has to re-open the tiles file only to get EBMV
values. That's silly. The Progress
object should just include EBMV
so it always has it.
from desisurvey.
Looking at this now. I may need to pull in @dkirkby to understand how this is supposed to work within the design of the Progress object, Configuration singletons, etc. But in the meantime, I think this is the workaround that @weaverba137 already mentioned:
import desisurvey.config
from desisurvey.progress import Progress
desisurvey.config.Configuration(file_name=minitestdir+'/desisurvey-config.yaml')
p = Progress(restore='progress.fits')
explist = p.get_exposures()
explist.write(os.path.join(surveydir, 'exposures.fits'), overwrite=True)
desisurvey is using a Configuration object singleton as a form of global variable, including which tile file to use if overriding desimodel. You have to load the Configuration prior to creating the Progress object so that it knows to use that.
Note: minitest
is a variable defined in the notebook in the latest version merged into master. It may not have been in the version of the notebook you grabbed, but it basically points to the directory that contains the notebook, i.e. os.getcwd()
at the start of running.
I think the longer term solution is to add a constructor option to the Progress object with the config file to read if restore=None. Including EBMV in the Progress object to avoid re-reading the tiles file would be bonus, but it seems that the constructor would still need to read a configuration file to get config.min_snr2_fraction()
. i.e. the progress.fits file does not appear to be self contained for restoring a Progress object. I'll chat with @dkirkby.
from desisurvey.
A Progress FITS file is intended to be self-contained so I probably need to write min_snr2_fraction
into its header. The EBMV column is intentionally not part of the Progress table (which I tried to keep minimal) but get_exposures
can splice it in as a convenience (thus requiring access to the tiles file). I am ok with adding the EBMV column to Progress if that's the simplest solution, now that we support alternate tile files. In any case, I won't get a chance to work on this until next week.
from desisurvey.
In the latest test on two_percent_DESI, branch automini, and using desisurvey branch pass2program, I got a new type of error:
./test-tiles.fits
INFO:progress.py:133:__init__: Loaded progress from /global/cscratch1/sd/bweaver/desi/dev/end2end/survey/progress.fits.
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-12-2d2835b59e3d> in <module>()
8 config = Configuration(os.path.join(minitestdir, 'desisurvey-config.yaml'))
9 print(config.tiles_file())
---> 10 p = Progress(restore='progress.fits')
11 explist = p.get_exposures()
12
/global/common/cori/contrib/desi/desiconda/20170613-1.1.4-spectro/code/desisurvey/0.9.2/lib/python3.5/site-packages/desisurvey-0.9.2-py3.5.egg/desisurvey/progress.py in __init__(self, restore, max_exposures)
136 raise RuntimeError(
137 'Progress table has incompatible version {0}.'
--> 138 .format(table.meta['VERSION']))
139 # Check that the status column matches the current min_snr2.
140 snr2sum = table['snr2frac'].data.sum(axis=1)
RuntimeError: Progress table has incompatible version 4.
Here's the whole cell:
if os.path.exists(expfile):
print('Reading explist from {}'.format(expfile))
explist = Table.read(expfile)
else:
from desisurvey.config import Configuration
from desisurvey.progress import Progress
Configuration.reset()
config = Configuration(os.path.join(minitestdir, 'desisurvey-config.yaml'))
print(config.tiles_file())
p = Progress(restore='progress.fits')
explist = p.get_exposures()
# Sanity check that all tiles in the subset were observed in the exposures list
if not np.all(np.in1d(tiles['TILEID'], explist['tileid'])):
print("ERROR: some tiles weren't observed;\ncheck {} for failures".format(survey_logname) )
print("Missing TILEIDs:", set(tiles['TILEID']) - set(explist['tileid']))
else:
print("All tiles in the subset were observed at least once")
explist.write(expfile, overwrite=True)
print('Wrote {}'.format(expfile))
from desisurvey.
PS, note this line:
/global/common/cori/contrib/desi/desiconda/20170613-1.1.4-spectro/code/desisurvey/0.9.2/lib/python3.5/site-packages/desisurvey-0.9.2-py3.5.egg/desisurvey/progress.py in __init__(self, restore, max_exposures)
That should be my checkout git checkout of desisurvey, not the installed version! I checked very, very carefully to ensure that only my version of desisurvey was in sys.path
, but it's still turning up.
from desisurvey.
Ah, OK, the very first cell imports desisurvey, before I switch to the git checkout version. So I have to modify sys.path before anything else.
from desisurvey.
Once I made sure that I was starting from a completely clean end2end
directory, and made sure that the sys.path
was set correctly before any DESI packages were imported, then I'm able to get past the progress to exposures stage successfully. Continuing on with the rest of the notebook now.
from desisurvey.
I've run completely through the notebook with no problems. The solution is:
- Start from a completely clean output directory.
- Make sure
sys.path
is set correctly before importing any DESI packages.
from desisurvey.
Related Issues (20)
- RA, Dec mix up when calling desisurvey.ephem.get_object_interpolator HOT 1
- Found invalid plan_tiles in use_plan(). HOT 4
- Consider moving freeze_iers to desiutil HOT 3
- unit tests and surveyinit failing with new tile file HOT 4
- Make tile file more flexible to changes HOT 1
- Update default fiberassign cadence HOT 1
- HA optimizations should be made more flexible
- Consider alternative status file formats
- Change mechanism by which rules and tile file are specified in afternoon planning
- Remove simulate blocks from desisurvey
- should desisurvey depend on specsim? HOT 2
- Next tile to be chosen before previous exposure is saved to disc HOT 1
- unit tests failing on NERSC HOT 4
- revive tests with GitHub actions
- desisurvey broken with astropy 4.2 HOT 8
- reading horizons_2020_week1_moon.csv broken under astropy 4.2
- Unit tests failing with speclite error HOT 1
- desisurvey.ephem is broken in current DESI releases HOT 7
- survey simulations tutorial fails with AttributeError: 'Configuration' object has no attribute 'tiles_nogray' HOT 1
- Use exposures-daily and tiles-daily instead of tsnr-exposures & tiles
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from desisurvey.