Comments (11)
Another option is to add an HDF5 backend with pytables. @ToddSmall has a demo script somewhere that shows how you can pass around in-memory HDF5 objects between processes.
from xarray.
Another similar option would be to use in-memory HDF5 objects for which Todd Small found an option:
Writing to a string:
h5_file = tables.open_file("in-memory", title=my_title, mode="w", 12
driver="H5FD_CORE", driver_core_backing_store=0)
... [add variables] ...
image = h5_file.get_file_image()
Reading from a string
h5_file = tables.open_file("in-memory", mode="r", driver="H5FD_CORE",
driver_core_image=image,
driver_core_backing_store=0)
from xarray.
Looks like this may be the only option. Based on my tests, netCDF4 is strongly antithetical to any kind of streams/piped buffers. If we go the hdf5 route, we'd have to reimplement the CDM/netcdf4 on top of hdf5, no?
from xarray.
HDF5 supports homogeneous n-dimensional arrays and metadata, which in
principle should be all we need. Actually, under the covers netCDF4 is
HDF5. But yes, we would have to do some work to reinvent this.
On Wed, Feb 26, 2014 at 2:32 PM, ebrevdo [email protected] wrote:
Looks like this may be the only option. Based on my tests, netCDF4 is
strongly antithetical to any kind of streams/piped buffers. If we go the
hdf5 route, we'd have to reimplement the CDM/netcdf4 on top of hdf5, no?Reply to this email directly or view it on GitHubhttps://github.com/akleeman/xray/issues/23#issuecomment-36186205
.
from xarray.
Just wrote a little library to do netCDF4 via h5py: https://github.com/shoyer/h5netcdf
Unfortunately h5py still can't do in-memory file images (h5py/h5py#552). But it does give an alternative way to read/write netCDF4 without going via the Unidata libraries. There is experimental support for engine='h5netcdf'
in my dask PR: #381
pytables was not a viable option because it can't read or write HDF5 dimension scales, which are necessary for dimensions in netCDF4 files.
from xarray.
In an effort to reduce the issue backlog, I'll close this, but please reopen if you disagree
from xarray.
This is actually finally possible to support now with h5py, which as of the latest release supports reading/writing to file-like objects in Python.
from xarray.
In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity
If this issue remains relevant, please comment here or remove the stale
label; otherwise it will be marked as closed automatically
from xarray.
Still an issue, as far as I can tell. Possibly duplicated in #3372.
from xarray.
This must be the one of the oldest active issues in the project!
Curious if folks have thoughts on the current state here? Netcdf4-python has a some support for in-memory datasets but we are not exposing them with xarray (as far as I can tell). How about h5netcdf?
from xarray.
There is effort in #6956 to enable this for netcdf4.
This is possible for h5py/h5netcdf, too. See https://docs.h5py.org/en/stable/high/file.html#file-drivers. xarray has just to enable this via kwarg driver="core"
. Not sure how much effort this will take.
from xarray.
Related Issues (20)
- Assigning `pd.Series` as a coord drops the index name, causing a confusing error
- DataTree.update can cause multiple root groups. HOT 6
- Segmentation fault similar to issue 8410 HOT 1
- Losing top level name attribute when saving and then reopening using h5netcdf HOT 15
- Segmentation fault similar to issue 8410 HOT 3
- Unloaded DataArray opened with scipy missing attributes when sliced with a sequence HOT 3
- Weighted quantile calculation is incorrect HOT 2
- Xarray now forces Pandas == 2.2.2 install HOT 4
- Calling `.isel()` on a timezone-aware dimension/index causes it to lose timezone information HOT 2
- Broken Benchmarked by ASV Badge URL HOT 1
- storing results of interpolate_na into numpy.array HOT 3
- Pandas-stubs forces numpy <2.0.0 HOT 2
- Missing coordinate attributes after weighted mean HOT 1
- ds.weighted should skip non-numeric data_vars
- Integrate Treescope into Xarray's interactive HTML repr HOT 3
- Inconsistent `run_spec` during zarr dataset opening by multiple bound methods HOT 11
- Failure on `main` testing xarray.cross HOT 2
- pickle error in Dataset.sortby() when used with file-like
- Enable multi-coord grouping from xarray HOT 3
- Document `keepdims` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from xarray.