dgbowl / dgpost Goto Github PK
View Code? Open in Web Editor NEWdgpost: datagram post-processing toolkit
Home Page: https://dgbowl.github.io/tomato
License: GNU General Public License v3.0
dgpost: datagram post-processing toolkit
Home Page: https://dgbowl.github.io/tomato
License: GNU General Public License v3.0
There is a bug in the helper function load_data
, where the near
parameter does not get picked up properly when a yaml
-file input is used.
https://github.com/dgbowl/dgpost/blob/master/src/dgpost/utils/helpers.py#L286-L296
In dgpost~2.0
, we are using the unit registry included in yadg~4.2
. This is however restricting and should be relaxed, so that compatibility with the base pint unit registry is restored.
This is likely a breaking change and therefore would need thorough testing.
Create a front page for the documentation, linking to the following items:
recipe
schema descriptionrecipes
: load
, extract
, transform
, save
transform
moduletransform
moduleThink about conversion of co-feeds in inlet mixture.
When running the following command:
dgpost ./Recipe/dgpost/dgpost.recipe.01_electro.yaml --patch ./Output/20230602-15p-Cu-PsA1-I-01/datagram_20230602-15p-Cu-PsA1-I-01 -v
I received the following error
Traceback (most recent call last):
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\Scripts\dgpost.exe\__main__.py", line 4, in <module>
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\dgpost\__init__.py", line 5, in <module>
from .main import run_with_arguments, run
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\dgpost\main.py", line 16, in <module>
from dgpost.utils import parse, load, extract, transform, save, plot, pivot
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\dgpost\utils\__init__.py", line 1, in <module>
from .extract import extract
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\dgpost\utils\extract.py", line 74, in <module>
from dgpost.utils.helpers import (
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\dgpost\utils\helpers.py", line 27, in <module>
ureg.define("refractive_index_units = [] = RIU")
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\pint\facets\plain\registry.py", line 469, in define
self._helper_dispatch_adder(definition)
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\pint\facets\plain\registry.py", line 492, in _helper_dispatch_adder
adder_func(definition)
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\pint\facets\group\registry.py", line 85, in _add_unit
super()._add_unit(definition)
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\pint\facets\nonmultiplicative\registry.py", line 72, in _add_unit
super()._add_unit(definition)
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\pint\facets\plain\registry.py", line 571, in _add_unit
self._helper_adder(definition, self._units, self._units_casei)
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\pint\facets\plain\registry.py", line 509, in _helper_adder
self._helper_single_adder(
File "C:\Users\plnu\AppData\Local\miniconda3\envs\test_yadg\lib\site-packages\pint\facets\plain\registry.py", line 532, in _helper_single_adder
raise RedefinitionError(key, type(value))
pint.errors.RedefinitionError: Cannot redefine 'RIU' (<class 'pint.delegates.txt_defparser.plain.UnitDefinition'>)
You can find the .nc file I used and the recipe here. This file was created using yadg5.1. When using yadg5.04 to process this experiment, the script ran just fine.
I dug a bit deeper into the source of the error. The last pasrt where the traceback stay in dgpost is at line 27 in `dgpost\utils\helpers.py'
Here the line that potentiall causing issue is
ureg = pint.get_application_registry()
ureg.define("refractive_index_units = [] = RIU")
I isolated this code snippet into a separate jupyter notebook file (putting correct import for Pint of couse). I found that if I run this using Pint==0.23, then this snippet ran just fine. But if I use Pint==0.24.1 (which is required by yadg5.1) then I recevie the same error as I got above. Please let me know if you need any additional information on this. Thank you.
In addition, in order to test yadg5.1, I created a new conda environment for this. I found that if I just pip install yadg and dgpost directly, this will not work as the latest version of numpy 2 and pandas 2 has many incompatibility with many underlying library. With some experiment, I found that pandas==2.0.0 and numpy==1.23.5 works for the time being. You may would like to consider updating the requirement for yadg and dgpost with this.
Currently a big missing feature in dgpost
is to be able to "pivot" tables using an arbitrary data column.
For instance, in electrochemistry data, we often have a running index denoting the current charge/discharge cycle, and the cell capacity (charge) should be integrated between changes in that index. This is currently not possible in dgpost
, as all functions from the transform
library are applied on the whole table.
Instead of restricting the plotting to use a single table for the whole figure, we should allow selecting the source table on a per-subfigure basis. This would allow plotting data from multiple tables in multiple subfigures in one figure, using a shared x-axis, but at their native frequency.
It should be possible to use NetCDF to roundtrip data in addition to pickles.
[mol/s]
without internal standardBugs:
Pivot
does not actually process timedelta
Documentation:
dgbowl_schemas.Recipe
properlydgbowl_schemas
should be reworked to always point to the latest version of schema: dgbowl_schemas.Recipe
works, but the components (load, extract, etc.) don'tTo allow for better interpolation of electrochemical data as a function of timesteps, the following two functions should be implemented:
q = โซ[0->t](I dt)
<I> = dq / dt
where dt
is the difference of two consecutive timestamps.
The validation of recipe
schemas should be handed off to the dgbowl-schemas
package:
strictyaml
to pydantic
v1.0
of recipe
schemaThe rates
library in the transform
module should be extended with functions that calculate formation rates from a batch experiment. Given a set of timestamps and corresponding concentrations and volumes, the rates can be calculated in a fairly straightforward way as ndot = dc * V / dt
.
unpivot
, the accompanying inverse operation to pivot
, should be included, so that e.g. timeseries data can be re-expanded back onto the whole table.
When saving tables, it should be possible to optionally select columns which are to be exported.
Paramteter uncertainty is still missing in impedance.fit_circuit
.
For further information look:
Given a set of freq
, Re(Z)
and -Im(Z)
data, it would be great to find all values of Re(Z)
(and freq
) where -Im(Z) = 0
.
Explore multiindex pandas functionality to "nest" column headers instead of separating them using "->"
.
Extracting variables:
key
(source) entry, keep the parent_key->first_key->*
and parent_key->first_key->second_key
syntaxas
(target) entry, ->
syntax is translated into a multiindex columnxout->species
)traces->detectors->axes
)Units:
df.attrs["units"]
using ->
, use a dictionary structure.Transforming columns:
The documentation of dgpost needs tidying up for v1.0:
I am now working on implementing the yadg 5 upgrade to the pipeline of CO2 reduction e.chem data analysis. I folked yadg 5.0 and use this version to create the datagram.nc file. While running yadg, I also encountered some errors. So, I had to make some minor adjustments to the script, but I eventually could get datagram.nc file. Now that I use this datagram.nc to run dgpost, I encounterd an error. I am not sure whether this is the issue with dgpost itself or it is something with the .nc file itself.
Please find attached the ymal and datagram (.nc) file used in the run as follows: https://drive.switch.ch/index.php/s/fb1oqUuXDz9HRXW
I ran dgpost using the following command:
dgpost "C:\Users\plnu\Dropbox\PythonStuff\Empa\Notebooks\Empa_30 Testing yadg5\Test auto DB\AutoDB\Recipe\dgpost\dgpost.recipe.01_electro_v5.yaml" --patch "C:\Users\plnu\Dropbox\PythonStuff\Empa\Notebooks\Empa_30 Testing yadg5\Test auto DB\AutoDB\Output\test_fran_file\datagram_test_fran_file" -v
But got the following error:
`INFO:dgpost.main:Processing 'load'.
Traceback (most recent call last):
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\xarray\backends\file_manager.py", line 211, in _acquire_with_cache_info
file = self._cache[self._key]
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\xarray\backends\lru_cache.py", line 56, in getitem
value = self._cache[key]
KeyError: [<class 'h5netcdf.core.File'>, ('C:\Users\plnu\Dropbox\PythonStuff\Empa\Notebooks\Empa_30 Testing yadg5\Test auto DB\AutoDB\Output\test_fran_file\datagram_test_fran_file.nc',), 'r', (('decode_vlen_strings', True), ('invalid_netcdf', None)), '4f541ab0-0650-487e-9c15-0232df7eec9c']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\runpy.py", line 196, in _run_module_as_main
return run_code(code, main_globals, None,
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\Scripts\dgpost.exe_main.py", line 7, in
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\dgpost\main.py", line 229, in run_with_arguments
run(args.infile, patch=args.patch)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\dgpost\main.py", line 95, in run
tables[el["as"]] = load(fp, None, el["type"])
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\dgpost\utils\load.py", line 51, in load
return datatree.open_datatree(path, engine="h5netcdf")
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\datatree\io.py", line 58, in open_datatree
return open_datatree_netcdf(filename_or_obj, engine=engine, **kwargs)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\datatree\io.py", line 66, in open_datatree_netcdf
ds = open_dataset(filename, **kwargs)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\xarray\backends\api.py", line 573, in open_dataset
backend_ds = backend.open_dataset(
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\xarray\backends\h5netcdf.py", line 402, in open_dataset
store = H5NetCDFStore.open(
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\xarray\backends\h5netcdf.py", line 175, in open
return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\xarray\backends\h5netcdf.py", line 126, in init
self.filename = find_root_and_group(self.ds)[0].filename
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\xarray\backends\h5netcdf.py", line 186, in ds
return self.acquire()
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\xarray\backends\h5netcdf.py", line 178, in _acquire
with self._manager.acquire_context(needs_lock) as root:
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\contextlib.py", line 135, in enter
return next(self.gen)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\xarray\backends\file_manager.py", line 199, in acquire_context
file, cached = self._acquire_with_cache_info(needs_lock)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\xarray\backends\file_manager.py", line 217, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\h5netcdf\core.py", line 1051, in init
self._h5file = self._h5py.File(
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\h5py_hl\files.py", line 562, in init
fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\h5py_hl\files.py", line 235, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py\h5f.pyx", line 102, in h5py.h5f.open
OSError: Unable to synchronously open file (file signature not found)`
dgpost
should be able to read NeXus files, in particular ones containing NXlog
entries (i.e. timeseries data). Support for other NeXus files (e.g. NXdata
entries) is perhaps less important at this stage.
The save function should document the version of dgpost
used to create the saved table. The load function should be version-aware. This is especially important for round-tripping saved pkl/json
files, as a move to multiindex tables is planned for v2.0
(#2).
dgpost
should be able to load other files than datagrams
. Especially pkl
files which include a pandas.DataFrame
for chaining and round-tripping data, but also non-datagram jsons
and perhaps csv
and xlsx
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.