tardis-sn / carsus Goto Github PK
View Code? Open in Web Editor NEWAtomic Database for Astronomy
Home Page: https://tardis-sn.github.io/carsus
Atomic Database for Astronomy
Home Page: https://tardis-sn.github.io/carsus
Make Chianti optional again
To be made into a conversion script.
from carsus.io.nist import NISTWeightsComp, NISTIonizationEnergies
from carsus.io.kurucz import GFALLReader
from carsus.io.zeta import KnoxLongZeta
from carsus.io.output import TARDISAtomData
output_file = "kurucz_As-U_ALL.h5"
atomic_weights = NISTWeightsComp('As-U') # weights from NIST
ionization_energies = NISTIonizationEnergies('As-U') # ionization energies from NIST
atomic_weights.to_hdf(output_file)
ionization_energies.to_hdf(output_file)
zeta = KnoxLongZeta('data_for_carsus/knox_long_recombination_zeta.dat') # zeta from Knox Long
zeta.to_hdf(output_file)
gfall_reader = GFALLReader('data_for_carsus/gfall.dat') ## Kurucz gfall data
atom_data = TARDISAtomData(gfall_reader, ionization_energies, ions='As-U')
atom_data.to_hdf(output_file)
Hello,
I am following the tutorial given here to create a TARDIS atomic data file. Everything works until the line atom_data = ...
, at which point, I get the following:
Downloading data from the NIST Atomic Weights and Isotopic Compositions database.
Downloading ionization energies from the NIST Atomic Spectra Database
[carsus.io.kurucz.gfall][WARNING] A specific combination to identify unique levels from the gfall data has not been given. Defaulting to ["energy", "j"]. (gfall.py:68)
Traceback (most recent call last):
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/series.py", line 987, in setitem
self._set_with_engine(key, value)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/series.py", line 1046, in _set_with_engine
self.index._engine.set_value(values, key, value)
File "pandas/_libs/index.pyx", line 95, in pandas._libs.index.IndexEngine.set_value
File "pandas/_libs/index.pyx", line 106, in pandas._libs.index.IndexEngine.set_value
ValueError: setting an array element with a sequence.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/nvieira/Documents/rprocess_2020/make_TARDISAtomData.py", line 112, in <module>
ions='As-U')
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/carsus-0.1.dev623-py3.6.egg/carsus/io/output/base.py", line 50, in __init__
self.ground_levels = ionization_energies.get_ground_levels()
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/carsus-0.1.dev623-py3.6.egg/carsus/io/nist/ionization.py", line 333, in get_ground_levels
levels = self.parser.prepare_ground_levels()
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/carsus-0.1.dev623-py3.6.egg/carsus/io/nist/ionization.py", line 174, in prepare_ground_levels
"L", "parity", "J"]] = ground_levels.apply(parse_ground_level, axis=1)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/frame.py", line 6487, in apply
return op.get_result()
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/apply.py", line 151, in get_result
return self.apply_standard()
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/apply.py", line 257, in apply_standard
self.apply_series_generator()
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/apply.py", line 286, in apply_series_generator
results[i] = self.f(v)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/carsus-0.1.dev623-py3.6.egg/carsus/io/nist/ionization.py", line 153, in parse_ground_level
lvl["J"] = lvl_tokens["J"]
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/series.py", line 1039, in __setitem__
setitem(key, value)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/series.py", line 1015, in setitem
self.loc[key] = value
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/indexing.py", line 190, in __setitem__
self._setitem_with_indexer(indexer, value)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/indexing.py", line 656, in _setitem_with_indexer
value=value)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 510, in setitem
return self.apply('setitem', **kwargs)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 395, in apply
applied = getattr(b, f)(**kwargs)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 827, in setitem
values, value = self._try_coerce_args(values, value)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 712, in _try_coerce_args
if np.any(notna(other)) and not self._can_hold_element(other):
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/dtypes/missing.py", line 333, in notna
res = isna(obj)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/dtypes/missing.py", line 99, in isna
return _isna(obj)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/dtypes/missing.py", line 114, in _isna_new
return _isna_ndarraylike(obj)
File "/home/nvieira/anaconda3/envs/tardis/lib/python3.6/site-packages/pandas/core/dtypes/missing.py", line 193, in _isna_ndarraylike
dtype = values.dtype
AttributeError: ("'function' object has no attribute 'dtype'", 'occurred at index 0')
I am working within an environment specifically for tardis, as suggested in the tardis installation guide. My install of tardis was cloned directly from the github and has successfully produced the spectra from the quickstart guide. I have verified that the version of pandas
in this environment is 0.24.2, as is requested in the tardis environment definition file. The version of carsus I am using has also been cloned directly from the github.
I have tried taking the other route of using the carsus quickstart guide, but have had no success there either. I'm at a bit of a loss.
Any help would be much appreciated! (Additionally, if anyone happens to have an usable .h5 file with all the elements from As-U in the meantime, that would also be very helpful.)
See: #191 (comment)
Additional context
Fiasco is an interface to the Chianti db. Maybe it may be easier for Carsus to depend on fiasco than chiantipy.
Maybe @wtbarnes and @wkerzendorf can have a chat 😉 to see which features are needed and whether fiasco makes it easier to maintain.
When trying to export a database to tardis the sql query seems to get too long. This happens when querying for all levels with a level id that is in the list of selected levels.
Solutions would be to batch these queries or to query for all and then perform the selection in python instead of on the database level.
@mishinma How did you create the database inside carsus-db
? How did you circumvent this problem or did it not appear back then?
Currently there are tons of development topics that are not properly documented.
Carsus README.md is almost non existent.
starkit/wsynphot
)The purpose of this issue is to gather all issues regarding the chianti database
Is your feature request related to a problem? Please describe.
TARDIS needs photoionization data to perform NLTE calculations. This data will come mainly from the CMFGEN atomic database. This issue describes the desired output format for this data.
Describe the solution you'd like
The goal is to export a table with frequency (in units Hz) and cross section values (in units cm^2) for each level. The frequencies start at the threshold for ionization and should follow a logarithmic grid. The image below shows an example of the desired ouput DataFrame:
The DataFrame has a MultiIndex with the atomic_number, ion_number, level_number of the level from which photoionization occurs. The destination levels is not specified since we assume all photoionizations go to the ground state of the next higher ionization state.
Describe alternatives you've considered
None. The format is perfect as is ;-)
Currently we can associate multiple Quantities with one object, for example a line. In practice this might be useful when there is a theoretical and a measured value. However the current model makes interacting with the database very difficult.
I propose to change all relationships with quantities to One-to-One relationships as part of the Quantity rework that should be coming. One could then add both, a measured and a theoretical value, to the same quantity.
Then one could include the logic to decide which value should be used by default at one place and would not have to duplicate it several times.
Missing:
2nd order Models like LineQuantity
and LevelEnergy
carsus
version with SQL code.Seems the problem is at carsus/data/basic_atomic_data.csv
.
Some elements have been renamed (for example Uuo
-> Og
) but this is not enough to explain this behavior.
For ds_id
and meas
/theor
and other categorical stuff.
See: https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html
Got this error when trying to ingest data:
sqlalchemy.exc.InterfaceError: (sqlite3.InterfaceError) Error binding parameter 0 - probably
unsupported type. [SQL: u'SELECT atom.atomic_number AS atom_atomic_number, atom.symbol AS
atom_symbol, atom.name AS atom_name, atom."group" AS atom_group, atom.period AS
atom_period \nFROM atom \nWHERE atom.atomic_number = ?'] [parameters: (1,)]
The full traceback http://pastebin.com/4VYMYuGS
A very similar question at stackoverflow http://stackoverflow.com/questions/22734671/weird-type-error-arising-when-i-add-to-a-database-using-sql-alchemy
I managed to add Si III and O I by creating a new hdf5 from the existing database 'kurucz_cd23_chianti_all' using the scripts in carsus-db. For some reason, using chianti_ions = 'H-He; Si II; Ca II; Mg II; S II; Si III; O I' didn't work and gave only H-He, Si II. But when I tried chianti_ions = 'H-He ; Si ; Ca ; Mg ; S ; O ' it created a file with all the collision data.
Status: unconfirmed
if we decide to go with nm - we should do that
lines['wavelength'] = lines['wavelength'] * u.AA.to(u.nm)
Originally posted by @wkerzendorf in #191 (comment)
pytest=5
pytest=4
pandas=1.0
astropy=3
pyparsing=2.2
SQLAlchemy=1.0
ChiantiPy
tardis
numexpr
default
This is not a bug, results are ok (but could be confusing).
Wavelength units are not consistent in GFALLReader
and ChiantiReader
.
In carsus/io/output/base.py
:
lines.loc[gfall_mask, 'wavelength'] = lines.loc[
gfall_mask, 'wavelength'].apply(lambda x: x*u.nm)
lines.loc[chianti_mask, 'wavelength'] = lines.loc[
chianti_mask, 'wavelength'].apply(lambda x: x*u.angstrom)
lines['wavelength'] = lines['wavelength'].apply(
lambda x: x.to('angstrom'))
lines['wavelength'] = lines['wavelength'].apply(lambda x: x.value)
... we can see GFALLReader
stores wavelength
in nm
and ChiantiReader
in AA
. Then every line is stored in AA
in TARDISAtomData
object.
Since ChiantiReader
(and later CMFGENReader
) are tailored to mimic GFALLReader
class, should store data in the same units for simplicity.
Currently carsus
does not sign the atomic data in a coherent way. For example, making two identical files leads to different MD5 sums.
Reproducibility of the atomic files should be one of the key features of Carsus, so I propose to re-think the hash/sign method.
to_msgpack
method seems to be the most efficient way to hash Pandas objects (we tried other methods).to_msgpack
is deprecated since Pandas 0.25 now we should use pyarrow
(see example below).key
and md5sum
columns per key available in the HDF5 file.pyarrow
serializationTested, work as expected. Consecutive builds of the same atomic file returns the same MD5.
import pyarrow as pa
def to_hdf(self, fname):
"""Dump the `base` attribute into an HDF5 file
Parameters
----------
fname : path
Path to the HDF5 output file
"""
self.atomic_weights.to_hdf(fname)
self.ionization_energies.to_hdf(fname)
self.zeta_data.to_hdf(fname)
with pd.HDFStore(fname, 'a') as f:
f.put('/levels', self.levels_prepared)
f.put('/lines', self.lines_prepared)
f.put('/macro_atom_data', self.macro_atom_prepared)
f.put('/macro_atom_references',
self.macro_atom_references_prepared)
md5_hash = hashlib.md5()
for key in f.keys():
context = pa.default_serialization_context()
serialized_df = context.serialize(f[key])
md5_hash.update(serialized_df.to_buffer())
uuid1 = uuid.uuid1().hex
print("Signing AtomData: \nMD5: {}\nUUID1: {}".format(
md5_hash.hexdigest(), uuid1))
f.root._v_attrs['md5'] = md5_hash.hexdigest().encode('ascii')
f.root._v_attrs['uuid1'] = uuid1.encode('ascii')
There are two stored versions:
carsus/tests/data
carsus/data
is that a sensible temperature range? This should be somewhere better documented.
Originally posted by @wkerzendorf in #191 (comment)
Seems the only piece of TARDIS code required by Carsus is:
from tardis.util.colored_logger import ColoredFormatter, formatter_message
This bloats Carsus environment with unneeded dependencies such as Cython
and tqdm
.
Hi all
I'm keen on trying to run TARDIS with heavier elements than the ones included in the default distribution (Up to Z = 30).
I have installed TARDIS with "pip install git+https://github.com/tardis-sn/tardis“ and have set up the tardis conda environments, as suggested as using the depencecy.yml from "https://raw.githubusercontent.com/tardis-sn/tardis/master/tardis_env27.yml”. However, scipy and pyne had dependency clashes and only after updating all the required packages with (conda update —all) can TARDIS successfully do the spectral synthesis.
Similarly, I have installed carsus with "pip install git+https://github.com/tardis-sn/carsus“ and all required packages. I have install ChiantiPy and downloaded the Chianti database and carsus-db.
Running the notebook, https://github.com/tardis-sn/carsus/blob/master/docs/notebooks/kurucz_chianti_h_he.ipynb, throws this error when running the Ionization Energies box:
Downloading ionization energies from the NIST Atomic Spectra Database
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/nist/ionization.py:89: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support skipfooter; you can avoid this warning by specifying engine='python'.
usecols=range(5), names=column_names, skiprows=3, skipfooter=1)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-3-b697c495f097> in <module>()
5 ioniz_energies_ingester.ingest(
6 ionization_energies=True,
----> 7 ground_levels=True
8 )
9 session.commit()
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/nist/ionization.pyc in ingest(self, ionization_energies, ground_levels)
287 # Download data if needed
288 if self.parser.base is None:
--> 289 self.download()
290
291 if ionization_energies:
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/nist/ionization.pyc in download(self)
221 def download(self):
222 data = self.downloader(spectra=self.spectra)
--> 223 self.parser(data)
224
225 def ingest_ionization_energies(self, ioniz_energies=None):
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/base.pyc in __call__(self, input_data)
42
43 def __call__(self, input_data):
---> 44 self.load(input_data)
45
46
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/nist/ionization.pyc in load(self, input_data)
89 usecols=range(5), names=column_names, skiprows=3, skipfooter=1)
90 for column in ['ground_shells', 'ground_level', 'ionization_energy_str']:
---> 91 base[column] = base[column].map(lambda x: x.strip())
92 self.base = base
93 # column_names = ['atomic_number', 'ion_charge', 'ground_shells', 'ground_level', 'ionization_energy_str', 'ionization_energy_uncertainty_str']
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/pandas/core/series.pyc in map(self, arg, na_action)
2352 else:
2353 # arg is a function
-> 2354 new_values = map_f(values, arg)
2355
2356 return self._constructor(new_values,
pandas/_libs/src/inference.pyx in pandas._libs.lib.map_infer()
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/nist/ionization.pyc in <lambda>(x)
89 usecols=range(5), names=column_names, skiprows=3, skipfooter=1)
90 for column in ['ground_shells', 'ground_level', 'ionization_energy_str']:
---> 91 base[column] = base[column].map(lambda x: x.strip())
92 self.base = base
93 # column_names = ['atomic_number', 'ion_charge', 'ground_shells', 'ground_level', 'ionization_energy_str', 'ionization_energy_uncertainty_str']
AttributeError: 'NoneType' object has no attribute 'strip'
I think this is due to the variable number of footer lines that are returned from the NIST query.
If I change the pandas.read_csv call in:
https://github.com/tardis-sn/carsus/blob/master/carsus/io/nist/ionization.py
from
base = pd.read_csv(StringIO(text_data), sep='|', header=None, usecols=range(5), names=column_names, skiprows=3, skipfooter=1)
to:
base = pd.read_csv(StringIO(text_data), sep='|', header=None, usecols=range(5), names=column_names, skiprows=3).dropna(how = "any")
correctly chops away the rows containing NaN values and the NIST part can be concluded. However, when ingesting the Kurucz database (GFALLIngester), the line-list are not correctly connected and it throws a SQL-related error my SQL-fu is not strong enough to clear up.
Ingesting levels from ku_latest
Ingesting levels for He 0
Ingesting levels for He 1
Ingesting levels for Li 0
Ingesting levels for Li 1
Ingesting levels for Be 0
Ingesting levels for Be 1
Ingesting levels for Be 2
Ingesting lines from ku_latest
Ingesting lines for He 0
Ingesting lines for He 1
---------------------------------------------------------------------------
InterfaceError Traceback (most recent call last)
<ipython-input-4-3d924feb90ac> in <module>()
1 gfall_ingester = GFALLIngester(session, gfall_fname, ions='H-Be')
----> 2 gfall_ingester.ingest(levels=True, lines=True)
3 session.commit()
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/kurucz/gfall.py in ingest(self, levels, lines)
435
436 if lines:
--> 437 self.ingest_lines()
438 self.session.flush()
439
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/kurucz/gfall.py in ingest_lines(self, lines)
394 print("Ingesting lines for {} {}".format(convert_atomic_number2symbol(atomic_number), ion_charge))
395 # print(ion)
--> 396 lvl_index2id = self.get_lvl_index2id(ion)
397
398 for index, row in ion_lines.iterrows():
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/kurucz/gfall.py in get_lvl_index2id(self, ion)
328
329 lvl_index2id = list()
--> 330 for id, index in q_ion_lvls:
331 lvl_index2id.append((index, id))
332
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/query.pyc in __iter__(self)
2875 context.statement.use_labels = True
2876 if self._autoflush and not self._populate_existing:
-> 2877 self.session._autoflush()
2878 return self._execute_and_instances(context)
2879
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in _autoflush(self)
1442 "consider using a session.no_autoflush block if this "
1443 "flush is occurring prematurely")
-> 1444 util.raise_from_cause(e)
1445
1446 def refresh(
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/util/compat.pyc in raise_from_cause(exception, exc_info)
201 exc_type, exc_value, exc_tb = exc_info
202 cause = exc_value if exc_value is not exception else None
--> 203 reraise(type(exception), exception, tb=exc_tb, cause=cause)
204
205 if py3k:
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in _autoflush(self)
1432 if self.autoflush and not self._flushing:
1433 try:
-> 1434 self.flush()
1435 except sa_exc.StatementError as e:
1436 # note we are reraising StatementError as opposed to
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in flush(self, objects)
2241 try:
2242 self._flushing = True
-> 2243 self._flush(objects)
2244 finally:
2245 self._flushing = False
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in _flush(self, objects)
2367 except:
2368 with util.safe_reraise():
-> 2369 transaction.rollback(_capture_exception=True)
2370
2371 def bulk_save_objects(
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.pyc in __exit__(self, type_, value, traceback)
64 self._exc_info = None # remove potential circular references
65 if not self.warn_only:
---> 66 compat.reraise(exc_type, exc_value, exc_tb)
67 else:
68 if not compat.py3k and self._exc_info and self._exc_info[1]:
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in _flush(self, objects)
2331 self._warn_on_events = True
2332 try:
-> 2333 flush_context.execute()
2334 finally:
2335 self._warn_on_events = False
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.pyc in execute(self)
389 self.dependencies,
390 postsort_actions):
--> 391 rec.execute(self)
392
393 def finalize_flush_changes(self):
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.pyc in execute(self, uow)
554 uow.states_for_mapper_hierarchy(
555 self.mapper, False, False),
--> 556 uow
557 )
558
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/persistence.pyc in save_obj(base_mapper, states, uowtransaction, single)
179 _emit_insert_statements(base_mapper, uowtransaction,
180 cached_connections,
--> 181 mapper, table, insert)
182
183 _finalize_insert_update_commands(
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/persistence.pyc in _emit_insert_statements(base_mapper, uowtransaction, cached_connections, mapper, table, insert, bookkeeping)
864 else:
865 result = cached_connections[connection].\
--> 866 execute(statement, params)
867
868 primary_key = result.context.inserted_primary_key
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/base.pyc in execute(self, object, *multiparams, **params)
946 raise exc.ObjectNotExecutableError(object)
947 else:
--> 948 return meth(self, multiparams, params)
949
950 def _execute_function(self, func, multiparams, params):
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/sql/elements.pyc in _execute_on_connection(self, connection, multiparams, params)
267 def _execute_on_connection(self, connection, multiparams, params):
268 if self.supports_execution:
--> 269 return connection._execute_clauseelement(self, multiparams, params)
270 else:
271 raise exc.ObjectNotExecutableError(self)
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/base.pyc in _execute_clauseelement(self, elem, multiparams, params)
1058 compiled_sql,
1059 distilled_params,
-> 1060 compiled_sql, distilled_params
1061 )
1062 if self._has_events or self.engine._has_events:
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/base.pyc in _execute_context(self, dialect, constructor, statement, parameters, *args)
1198 parameters,
1199 cursor,
-> 1200 context)
1201
1202 if self._has_events or self.engine._has_events:
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/base.pyc in _handle_dbapi_exception(self, e, statement, parameters, cursor, context)
1411 util.raise_from_cause(
1412 sqlalchemy_exception,
-> 1413 exc_info
1414 )
1415 else:
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/util/compat.pyc in raise_from_cause(exception, exc_info)
201 exc_type, exc_value, exc_tb = exc_info
202 cause = exc_value if exc_value is not exception else None
--> 203 reraise(type(exception), exception, tb=exc_tb, cause=cause)
204
205 if py3k:
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/base.pyc in _execute_context(self, dialect, constructor, statement, parameters, *args)
1191 statement,
1192 parameters,
-> 1193 context)
1194 except BaseException as e:
1195 self._handle_dbapi_exception(
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/default.pyc in do_execute(self, cursor, statement, parameters, context)
505
506 def do_execute(self, cursor, statement, parameters, context=None):
--> 507 cursor.execute(statement, parameters)
508
509 def do_execute_no_params(self, cursor, statement, context=None):
InterfaceError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (sqlite3.InterfaceError) Error binding parameter 1 - probably unsupported type. [SQL: u'INSERT INTO transition (type, lower_level_id, upper_level_id, data_source_id) VALUES (?, ?, ?, ?)'] [parameters: ('line', id
index
0 12253
0 53998, id
index
764 13017
764 54762, 3)] (Background on this error at: http://sqlalche.me/e/rvf5)
It looks like it correctly gets the lines for the neutral species, but complains when trying the singly ionised. Maybe it is not correctly separating the ions by charge?
Note that this is copied from:
...it is likely relevant to both repositories so I posted it on both for now!
I have generated an atomic dataset following the procedure outlined at:
https://github.com/tardis-sn/carsus/blob/master/docs/notebooks/quickstart.ipynb
It worked cleanly and smoothly. However, there seem to be some suspicious levels and some duplicate levels appearing in the database, the origins of which I don't know - in some places this might be a matching issue?
Specifically, here's the listing I get of carbon 1 energy levels from a TARDIS run with the dataset made following the recipe above:
In [31]: simulation.plasma.atomic_data.levels.loc[(6,0)]
Out[31]:
energy g metastable
level_number
0 0.000000e+00 1.0 True
1 3.257771e-15 3.0 True
2 8.621174e-15 5.0 True
3 2.024711e-12 5.0 True
4 2.024711e-12 5.0 True
5 4.300260e-12 1.0 True
6 4.300260e-12 1.0 True
7 4.300260e-12 1.0 True
8 6.701314e-12 5.0 True
9 1.198491e-11 1.0 False
10 1.198872e-11 3.0 False
11 1.199677e-11 5.0 False
12 1.231235e-11 3.0 False
13 1.273052e-11 7.0 False
14 1.273110e-11 3.0 False
15 1.273132e-11 5.0 False
16 1.367794e-11 3.0 False
17 1.384344e-11 3.0 False
18 1.384764e-11 5.0 False
19 1.385427e-11 7.0 False
20 1.405290e-11 3.0 False
21 1.417379e-11 1.0 False
22 1.417625e-11 3.0 False
23 1.418032e-11 5.0 False
24 1.442373e-11 5.0 False
25 1.469491e-11 1.0 False
26 1.494879e-11 3.0 False
27 1.494905e-11 5.0 False
28 1.494922e-11 1.0 False
29 1.543067e-11 5.0 False
... ... ... ...
803 1.803847e-11 3.0 True
804 1.803848e-11 1.0 True
805 1.803849e-11 3.0 True
806 1.803918e-11 5.0 True
807 1.803920e-11 5.0 True
808 1.803920e-11 7.0 True
809 1.803921e-11 3.0 True
810 1.803922e-11 7.0 True
811 1.803923e-11 5.0 True
812 1.803924e-11 3.0 True
813 1.803925e-11 1.0 True
814 1.803925e-11 3.0 True
815 1.803989e-11 5.0 True
816 1.803991e-11 5.0 True
817 1.803991e-11 7.0 True
818 1.803992e-11 3.0 True
819 1.803993e-11 7.0 True
820 1.803994e-11 5.0 True
821 1.803995e-11 3.0 True
822 1.803996e-11 1.0 True
823 1.803996e-11 3.0 True
824 1.804056e-11 5.0 True
825 1.804057e-11 5.0 True
826 1.804057e-11 7.0 True
827 1.804058e-11 3.0 True
828 1.804059e-11 7.0 True
829 1.804060e-11 5.0 True
830 1.804060e-11 3.0 True
831 1.804061e-11 1.0 True
832 1.804061e-11 3.0 True
[833 rows x 3 columns]
There are duplicate levels, evident from the start: e.g. levels 3 and 4 I think are both the same, as are 5, 6 and 7. (I believe that 3 and 4 are both supposed to be 2s^2 2p^2 ^1D_2, while 5, 5 and 7 are all supposed to be 2s^2 2p^2 ^1S_0).
I don't know whether this issue existed pre-CARSUS in the atomic data or is new.
There also seem to be a few oddities in the CII energy level list, but I've not quite understood that yet.
I propose to iterate over all Ion and gather the data individually. That makes it possible to select the datasource on a per_ion basis.
More ideas to come...
to_hdf
method in CMFGEN parsers does not accept fname
parameter. It just "converts" the plain text files to HDF5 saving them in the same folder. I think this is good as default behavior, but would be nice to let users save the file to an arbitrary location.
def to_hdf(key='/energy_levels', fname=None):
if fname is None
fname = self.fname
...
Implement black
formatting plus the check pipeline.
Version is shown only when docs are built locally.
After merging PR #191 we can start to discuss how to add cross sections to Carsus. First step would be creating a new method for the CMFGENReader
class in carsus/io/cmfgen/base.py
. This class should set an attribute called cross_sections
or similar. This attribute is a table, not necessarily the "final" table we want to store on the HDF5 file.
For example, for levels
and lines
the three classes GFALLReader
, ChiantiReader
and CMFGENReader
return tables with identical columns. Then, in the TARDISAtomData
class we perform several other calculations until we get the desired results to store.
It's important to say that the CMFGENReader
does not have a nice interface like the Chianti or GFALL counterparts. The current way to select levels and lines is: a) parsing the files, and b) creating a dict of dicts:
si0_lvl = CMFGENEnergyLevelsParser('/home/epassaro/Desktop/tardis-sn/atomic_data/CMFGEN/atomic/SIL/I/23nov11/SiI_OSC')
si0_osc = CMFGENOscillatorStrengthsParser('/home/epassaro/Desktop/tardis-sn/atomic_data/CMFGEN/atomic/SIL/I/23nov11/SiI_OSC')
si1_lvl = CMFGENEnergyLevelsParser('/home/epassaro/Desktop/tardis-sn/atomic_data/CMFGEN/atomic/SIL/II/16sep15/si2_osc_kurucz')
si1_osc = CMFGENOscillatorStrengthsParser('/home/epassaro/Desktop/tardis-sn/atomic_data/CMFGEN/atomic/SIL/II/16sep15/si2_osc_kurucz')
cmfgen_data = {'Si 0': {'levels': si_0_lvl, 'lines': si_0_osc},
'Si 1': {'levels': si_1_lvl, 'lines': si_1_osc}}
cmfgen_reader = CMFGENReader(cmfgen_data)
Then the cmfgen_reader
object is passed to the TARDISAtomData
class:
atom_data = TARDISAtomData(nist_weights, nist_energies, ... , cmfgen_reader)
.
pho
filesAccording to the GSoC 2019 work, we found three types of cross section files:
si1_cross_sections = CMFGENPhotoionizationCrossSectionParser(path_to_file))
if df.shape[1] == 2:
df.columns = ['Energy', 'Sigma']
elif df.shape[1] == 1:
df.columns = ['Fit coefficients']
elif df.shape[1] == 8: # Verner ground state fits. TODO: add units
df.columns = ['n', 'l', 'E', 'E_0',
'sigma_0', 'y(a)', 'P', 'y(w)']
So the new method for CMFGENReader
class should set a consistent cross_sections
attribute (a DataFrame) for these three kind of files.
Also the base
attribute of this parser is a bit confusing, returns a list of "small" DataFrames.
lines
and collisions
are connected to levels
through the get_lvl_index2id
method on TARDISAtomData
class.
To be discussed later...
levels
, lines
attribute of GFALL, Chianti and CMFGEN readers and see how consistent they are.CMFGENReader
to produce a table for cross sections.Probably before this we need to dig more into how the TARDISAtomData
class works.
TARDISAtomData
called create_cross_sections
, cross_sections_prepared
.See #191 (comment)
questionable - we need to do this differently. Raise issue.
Originally posted by @wkerzendorf in #191 (comment)
Proceed carefully.
From the upcoming docs:
In the recent years, much effort has been made to make Carsus easier to use and develop. A large piece of code was rewritten to avoid using an intermediate SQL database between the readers and the final atomic data.
This resulted in an simpler, faster and more maintainable Carsus codebase. But we lost some features in the process, being the most important one the tracking of physical quantities. Currently, we couldn’t solve the management of units in a sensible way.
We need to design and implement a new unit tracking system as soon as possible. In the meantime, should be enough to document the units used by the GFALL/CHIANTI/CMFGEN readers and the output module.
We agreed to document units until we design a new unit tracking system.
The regex expression used by CMFGENOscillatorStrengthsParser
is not working perfectly. Since lower and upper levels are tabulated as i-j
sometimes upper level is saved as -j
in the DataFrame.
I was wrong, the energy levels file contains negative ID's (for example -86
):
https://github.com/tardis-sn/carsus-refdata/blob/master/cmfgen/energy_levels/si2_osc_kurucz
Solution 1: update the regular expression to skip the dash.
Solution 2: just take .abs()
for j-column.
Mark these levels as theoretical.
The last version of ChiantiPy
that works with carsus
is 0.8.4
.
Since ChiantiPy have only a few dependencies maybe it will work for years... or not.
The following dependencies are required to run ChiantiPy,
Python3 (Python 3 is now required as of version 0.8.0)
Numpy
Scipy
Matplotlib
ipyparallel
This was done during GSoC, it's important to have this documented because those calculations are not super trivial. Then put a link in _create_einstein_coeff
docstring.
EDIT: also include description in docstring.
Thanks for the detailed explanation. We should make sure that this is consistent throughout all of the code. @wkerzendorf, @afloers and myself like ion_charge better than ion_number (since it is easier to interpret). Can you make an issue that we want to change to "ion_charge" also in the rest of the code?
Originally posted by @chvogl in #191 (comment)
The SQL ingesters had a nice logging, would be nice to have the same feature in the new code:
Add a References section like in TARDIS documentation. and put a link to 2019 poster there.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.