astrocatalogs / astrocats Goto Github PK
View Code? Open in Web Editor NEWAstrocats package for constructing astronomical catalogs
Home Page: https://astrocats.space
License: MIT License
Astrocats package for constructing astronomical catalogs
Home Page: https://astrocats.space
License: MIT License
So currently when we do something like say add_photometry
, we pass a list of kwargs
to the function where each kwarg is really a string:
catalog.entries[name].add_photometry(time=mjd, band=band, magnitude=magnitude,
e_magnitude=e_magnitude, source=source)
But, since say magnitude
could be changed to a different string later, it might be better to pass the kwargs as a dictionary, with the dictionary members pulling from the KeyCollection
members, like so:
catalog.entries[name].add_photometry(**{PHOTOMETRY.TIME: mjd, PHOTOMETRY.BAND: band,
PHOTOMETRY.MAGNITUDE: magnitude, PHOTOMETRY.E_MAGNITUDE: e_magnitude,
PHOTOMETRY.SOURCE: source})
This makes the add_*
calls a bit longer but also more stable to future changes. Should we adopt this as a best practice?
Currently the Photometry
class contains only simple primitives, with different types of flux/brightness measures enumerated. This is something that could be both streamlined and expanded significantly. This issue is a place to record ideas and brainstorm.
Quantity
classes which include errors and units internallyCurrently, if the value for a key in an Entry
or CatDict
is itself a CatDict
(instead of a primitive; see for example), then a string literal is used as the key instead of a Key
instance. This is because the Key
class doesn't know how to handle a value which is not one of the (current) built-ins {STRING
, NUMERIC
, BOOLEAN
...}.
Add a way to use Key
instances to handle composite class values. Probably the best thing to do is to have the corresponding KeyCollection
do the type checking...
Currently the only source referenced for spectra that appear on the CfA's page is the URL to the page itself, however we do have the publication info available from that page. This should be added for completeness.
We have a Task
object defined in astrocats/catalog/task.py
which represents a to-do-task. These to-do-tasks include a function name (and the submodule they're located in) which should be executed to complete the given task. Instead, the Task
object should be expanded to include the function itself. Likely, this should be something like:
class Task:
def __init__(self, ...):
# setup all of the variables currently in the `Task` objects
def load(self, ...):
# The actual function to complete the task
A list of Task
objects will be created, and each will just have a few parameters (like they do now). If a Task
is active (i.e. Task.active == True
) then Task.load()
will be called in the import script.
Benefits:
tasks.json
input file. Instead all of the tasks can live in the tasks
directory, which will be searched. The default settings will be stored in the class definitions.Sams as the comment on 04dk, and I think this will be true for all SNe with both Drout11 and CCCP photometric references: there are 2 V and 2 R band photometries, that differ by 1 magnitude, but they are both CCCP photometry: the same data, different reduction. One of them must be superseded. I think the Drout11, which is published, should be final, and the CCCP one should be the quick and dirty reduction, but I am not sure, and there is no date on the reduction webpage.
This is a place to discuss / brainstorm how the setup script and installation of astrocats
and catalogs should work in the future.
One of the first priorities should be making a test-catalog more easily runnable, and clearly show important/helpful results: see e.g. #74 (comment)
I believe I'm coming to a consensus on the Superfit spectra. It's fine if and when they are shown, but:
--they appear to be rest frame spectra, so they need not be corrected for redshift in the import script. There may be exceptions, but for the most part this is true, and I'm tracking the exceptions.
--the epochs for some spectra appear to be the negative of what they should be, e.g., day -116 for 90aa I'm guessing should go to +116. This seems to be the case for others, and the underlying thing here is that there are a few naming conventions for the superfit files. Anything with a "m05" in the filename refers to day -5 in the usual sense, but for post-max, there are two different conventions, "u" and "p", both apparently meaning +number of days since maximum light.
Individual tasks should exit it they are either taking far longer than normal to complete, or are stalled somewhere. In the long-run, it might be good to even set these up as non-blocking subprocesses (or other threading) --- presumably url-connection cycles will be hogging lots of time where the processor/drive is still available.
Currently, a single redshift tag is used to mark a SN's redshift, this redshift might be the redshift of the supernova, or it might be the redshift of the host galaxy, depending on the source. A new tag should be created, "hostredshift" that is specifically for this information. When a host redshift is known for a source but the supernova redshift is not known, the supernova redshift should be set to the host redshift, but the supernova redshift should be marked as "derived."
No secondary source is shown for McCully 2014 for SN2008A, but it is likely to be UCB. Need to figure out why the secondary source is not being added.
The --depth arg was used to restrict the depth of the repositories checked out, but it appears that it can no longer be passed, unless the syntax has changed to something weird. I tried several combinations and none worked:
python -m astrocats --depth 1 supernovae import
python -m astrocats supernovae import --depth 1
The --depth arg is needed to get the Travis build working, otherwise we run into Travis' disk quota.
The spectra of 83G from SuSpect do not have the correct reference because SuSpect does not have the correct reference. Instead the pointer is to Cristiani et al. (1992), which is on SN 1986G, not 83G. Searching for the original work brings up nothing, and the last hint from SuSpect is that the data were donated by the Asiago SN Group. http://graspa.oapd.inaf.it/
Fortunately I was able to use SNID to determine that the wavelength solution is fine and in the observer frame for the files.
BTW there really is a SN 1983G, but the spectra can only be found through GELATO (another SNID-like tool): https://gelato.tng.iac.es/. Which means the spectra can only be seen after submitting a spectrum for comparison.
https://gelato.tng.iac.es/templates/
"The templates database is continuously growing with new additions, which are immediately available to all GELATO users. GELATO currently uses 2926 template spectra to classify your SNe."
"Available" in GELATO, not available for download.
sne-external-spectra/Suspect/Ibc_spec_ascii_20110715/1990I/1990I_19900608_2979_10596_00.dat
For some reason this file for SN 1990I is misaligned, and I don't think it should be. I came across this one going through my list and when I plot the same file it is not misaligned wrt to the other files; i.e. the file is in the [inappropriate] rest frame, but it is out of sync against the other spectra which are [given] in the rest frame as well. Because this issue is a mouthful of correct/rest/observer frame, we can revisit this together after I am done correcting the others, but just thought I'd mention it here.
GRB 080319B: M = -39 -- this is the brightness of the GRB. The associated SN peaked at around -19 (http://iopscience.iop.org/article/10.1088/0004-637X/691/1/723/meta, http://iopscience.iop.org/article/10.1088/0004-637X/725/1/625/meta)
SN 2013cq: M = -29 -- also a mix of SN and GRB magnitudes
SN 2013ez: M= -29 -- also a GRB. I see a pattern emerging!
SN 2013fu: M= -28 -- GRB
"Possible supernova": M= -28 -- Redshift highly uncertain, and may be lensed
SN 2008hw: M= -26 -- GRB
SN 2004cq: M= -26 -- two conflicting redshifts, the one being used is incorrect
SN 1998ev: M= -25 -- apparent mag wrong, should be 21.3 not 14
CSS140424:123226+154840: M= -25 -- the brightest point is unrelated to the SN; it is an artefact at the edge of the CSS detector. This point is about 2 years after the actual SN. Note also that the for the 'claimed type', there was an ATel on this that would be a much better reference than CRTS or Latest Supernova pages. I don't know if this is a general trend, but ATels give a lot more information than these other sources. If possible, I would strongly encourage checking ATels for every SN and linking to the catalogue page!
SN 2010ma: M= -24 -- GRB
LSQ12axx: M= -24 -- Redshift on Latest Supernova page (0.399) is wrong, should be 0.0399 from ATel
SN 2014W: M= -24 -- Redshift on Latest Supernova page is wrong (can't work out where the number they have came from). CBET gives recession velocity rather than redshift, which I guess complicates reading things in directly from there. Corresponds to z=0.036
2001iw: M= -24 -- stumped by this one. The redshift is correct. Multiple SN cosmology papers quote the observed peak as 22 mag (you've imported this point). This looks correct for a Ia. However, the R and I band light curves peak at around 17 mag. The catalogue quotes the sources as 1,2. Source 2 is the actual paper (Barris et al), and the light curves look like they peak at 22 mag. I can't see any tabulated data. However in source 1 (Sternberg catalogue) where it seems your photometry has come from, the light curves have been shifted up by about 5 mag! I hope this isn't a common problem with light curves from this source... Or that not many of your other light curves come from there!
2002ab: M= -24 -- oh dear, looks like exactly the same issue.
2001bu: M= -24 -- took a bit of work to figure this one out! Redshift is from 'Unified SN Catalogue' (Lennarz et al). The host designation has a Q in it, which it turns out means quasar. So I think this must be a quasar rather than a SN. Might be worth going through everything from that catalogue and looking out for untyped SNe with a Q in the host name!
2001ca: M= -24 -- also from Lennarz catalogue. Not listed as a quasar, but I also see that this catalogue flags things that were never confirmed as SNe. Again, may want to filter out objects like this: one detection, never confirmed as SN, redshift also flagged as uncertain...
PS1-12qh: M= -24 -- ATel says its a QSO, not SN or LBV. Latest SN page has also got the magnitude wrong; should be 19.2, not 18.6
SN 2001iy: M= -23.7 -- see 2001iw and 2002ab
SCP06F6: M= -23.6 -- nothing wrong with the data or redshift. Maybe this isn't too important, but the absolute mag here is exaggerated simply because we need a fairly hefty K-correction at this redshift. You might want to implement M = m - DM + 2.5log(1+z) to get more realistic absolute mags for high-z SNe.
2001jp -- see 2001iw etc
2011jb -- couple of bad points in CSS light curve throwing peak mag way off
2010hu -- also a weird CSS point unrelated to SN
2001fo -- another for the weird Sternberg subclass of error
That's all I have time for for now, more soon...
Thank you to Isaac Shivvers and Jeff Silverman for the clarification on this one. They found the same issue some time ago:
https://sne.space/sne/SN1990M/
The top-most spectrum is being posted with an epoch of -75, when it should instead be closer to +216 according to Silverman et al. (2012) -- 2012MNRAS.425.1789S
The affected file is:
sne-external-spectra/UCB/sn1990m-19900401-opt.flm
From the looks of it, it's definitely at day +200-300, and there's no such thing as -75 for a SN Ia. For 87A, yes.
Given that there is a legitimate misprint in the date for this spectrum, and that 1991 would make sense instead of 1990 (private comm, IS, JS), I think it would be fine to change the year to 1991 in the filename. But we can discuss later before I go and change anything. At any rate, hello future SN enthusiasts, watch your step.
This issue affects potentially up to 1,000 supernovae in the catalog, and its cause is known and will be fixed in an upcoming commit. For the moment, do not trust SN coordinates with ±00:xx:xx declinations!
@jparrent I'd like to figure out exactly why the SUSPECT and the data pulled from Sternberg seem to disagree in regards to the photometry, which is most prominently visible in the I-band (almost a half mag difference between the two sources), but is also apparent in the other bands as well. As far as I can tell, both Sternberg and SUSPECT pulled this data from the same batch of papers from Whitelock and Catchpole. Any idea why the two are different?
I've perused some small databases of extragalactic SNRs and have added several hundred to the OSC, but there are ~2,000 according to SIMBAD. Adding these will be tricky given the names of these SNRs vary wildly and do not conform to a standard naming scheme. We might need some SNR experts to aid us in adding these given the state of the available data.
I'm wondering if issue #32 is a bug.
sne-external-spectra/Suspect/II_spec_ascii_20110715/2003bg/2003bg_20031129_3800_9324_00.dat
Similarly for 03bg there is a lone spectrum that is misaligned from where it should be. In the plot on the 03bg page, it's the third from the bottom that's too far shifted toward the red. All the files are in the correct observer frame, yet one of them is off--the one that is off has the correct numbers in its file.
@guillochon So it turns out this one was being flagged as a Ia, when it's a Ic.
SUPERNOVA 2003aa IN NGC 3367
Further to IAUC 8063, T. Matheson, P. Challis, and R. Kirshner
report that a spectrum (range 370-750 nm) of SN 2003aa (cf. IAUC
8063), obtained by P. Berlind with the 1.5-m reflector on Feb. 2.31
UT, shows it to be a type-Ic supernova. The spectrum is similar to
that of SN 1994I, four days before maximum (Filippenko et al. 1995,
Ap.J. 450, L11).
I flagged the claimedtype of Ia as erroneous and wanted to update the value to Ic. I thought I could do this via the pencil icon but then got an error. What did I miss?
Perhaps an individual catalog doesn't need to use the output-repository structure that supernovae
does. For example, the catalog may not version-track the output files at all, they may all be contained in a single repository, or perhaps they want to keep the output in the same repository as the code itself.
Instead of requiring the repos.json
file and associated methods, we can just require the necessary API methods. i.e. instead of get_repo_output_file_list()
(which is used by the delete_old_entry_files()
method --- and associated task), require a more general get_output_file_list()
which can store/construct those files in any (reasonable) way they wish.
@jparrent Looking to digitize this paper, and they provide both S-corrected photometry and the raw photometry. Do you think it's best to import the corrected or uncorrected photometry into the catalog? (or both?)
@guillochon Its max B-band date can be updated from 2011 Sep 6 to 2011 Sep 10 (MJD = 55814.51).
Pereira et al. 2013
http://adsabs.harvard.edu/abs/2013A%26A...554A..27P
This is a running list of objects that have spectra in need of a shave. I will add "exclusion" regions to the json files after redshift corrections are complete.
This is a running list of SN that need to be reset to a proper observer frame. I will do these and submit a PR in a few weeks, but I'm listing them here until they are fully fixed:
Disclaimers:
--it's not all objects from SuSpect, much less all spectra for each object listed
--some of the singles to check may be false alarms, but there are no other sources by which to compare (e.g., CfA).
Below is a running list of objects with classifications and/or epochs to double check with SNID.
Currently there is no color table assigned to radio data, which is input as frequencies rather than named bands, so most radio data currently appears as black regardless of frequency. We will add a continuous color table to the radio data shortly.
there is VanDyk+ 2013 reference (3) but i do not see the NIR photometry, or any optical photometry associated to reference (3) a significant amount of photometry available for this object. And there is a missing reference to Morales-Garoffolo+ 2014 (some of their data is also in Brown 15, but not all). I will try and upload it.
https://github.com/astrofoley/astr596/tree/master/data/spectra
This is a pile of spectra that needs to be sifted through. I will do this and will supply what the catalog does not already have.
there are 2 V band photometries, that differ by 1 magnitude, but they are both CCCP photometry: the same data, different reduction. One of them must be superseded. I think the Drout11, which is published, should be final, and the CCCP one should be the quick and dirty reduction, but I am not sure, and there is no date on the reduction webpage.
The V and r photometry seem to be the same datapoints at an offset. In fact there seem to be 2 copies of the V band data, and of many other bands B, I... and the r data looks like it is a copy of the V data at an offset.
Currently the front-end shows non-standard supernova types when they are pulled from SNDB (e.g. Ia-norm, etc.), this has been fixed in the code but won't be reflected in the catalog for a few more hours until the import script finishes. I will close this issue once the script completes.
@guillochon I believe there's a bug that is making two spectra for 98S much narrower than they should be. It's the first two spectra at the top from UCB. In the display window, select the pan button, and drag the cursor to see the original greyed out spectra near 6560 Angstroms. The third spectrum down is what they should look like, although the third spectrum down is from Superfit where the presented frame is off, which is what originally caught the eye.
Isaac Shivvers has nicely written an API for us to pull photometry from SNDB, currently we still use the old scraping algorithm. We should switch to the new API ASAP.
This is the one file in excess of 100mb uncompressed, seems it is getting deleted by one of the auto-compression scripts. Hopefully will return in next daily update but keeping this issue open until it is resolved.
Need to find a way to get the subcommand parsers to show their own help, when -h
/--help
is passed after the subcommand name.
e.g.:
astrocats -h
should run the top-level help.astrocats -m catalog -h
should run catalog-level help.astrocats -m catalog import -h
should run the import
help.Just noticed this. When I order SN Ia ('Ia') in the table at sne.space by luminosity distance (nearest events at the top, so increasing distance downward), the events that do not have a distance are returned first. Is there a way to have the events with NULL or empty entries go somewhere else when sorted? I imagine it would have to work both ways too, either increasing or decreasing luminosity distance. For example, page 12 is when the first nearest distance measurement shows up (SN 1604A). The page flipping is fast enough that it's still readily usable, but just a heads up.
It dawned on me today that the photometry and spectra icons over at sne.space take the user to the same spot. I understand the displays are under construction, but the icons could be merged unless keeping them separate would be better in the long run.
Sorting by discovery date (most recent first), shows a few SN with incorrect discovery dates in the future:
SN 2010kb, SN 2010ka, SN 2010jz, SN 2010jy, SN 2010jx, SN 25747NSV
Received an e-mail from Victor Lebedev highlighting some naming errors, which we need to address:
I found a few bugs and typos in your supernova catalog:
two supernovae ( SMTJ21413915-5643445, MASTEROTJ134201.21-023856.2 )
present twice;The following supernovae in alias mentioned twice:
Gaia16agf
MASTEROTJ111920.95+724857.3
PSNJ18121900+2131150
SNhunt10
SNhunt138
SNhunt27
SNhunt28
SNHunt309
SNhunt313
SNhunt314
SNhunt33
SNhunt34
SNhunt36
SNhunt39
SNhunt41
SNhunt42
SNhunt45
SNhunt49
SNhunt51
SNhunt55
SNhunt58
SNhunt62
SNhunt63
SNhunt64
SNhunt66
SNhunt7
SNhunt70
SNhunt77
SNhunt79
SNhunt82
SNhunt84
SNhunt85supernovae SNhunt133, SNhunt281
are doubles PSNJ17124620+2313265, SN2015bpin a supernova AT2016aha erroneously alias name SNhunt307
supernova PESSTOESO154-G10 one hand you have a double SN2014eg , while in the NED is designated as SN2013fc
Finally, "Possible supernova" there can be named supernovae (and Name , and Alias).
The colored lines are helpful as is, particularly for catching misaligned spectra, but could there be a toggle switch in the spectrum figures for the line colors to all turn black? I suggest this now, for later, but for inspecting the evolution of certain features, it would really only be visually appealing in black after the data have been cleaned up a bit. Getting there.
This is also an issue on the about page. Have fixed in the code and will close the issue when the fix is live.
Spectra will need to be scaled to the photometry.
Original poster, guillochon: I'd like to add the metadata from the SUSPECT spectra to the catalog, and I think the easiest way to do this is to just add all the FITS header files to the SUSPECT folder. That way I can parse the accompanying header file when I read in the spectra.
*migrated issue
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.