Comments (1)
My guess is, that you have multiple threads on the worker (or multiple workers that see the same filesystem). Since simplecache really is simple, it assumes that if a file is present, it is the whole cached file. So if one thread starts to download and another tries to open the file before that finishes, it will read the partial file on disk. That would account for what you see. I haven't yet thought of how you can solve this...
It may be worth opening an issue on fsspec, whereby the cacher downloads to a different filename and moves to the final destination when done (which may result in some files downloading multiple times, but that's not too bad).
from kerchunk.
Related Issues (20)
- Invalid try/except block in `scan_grib`? HOT 8
- Create a MultiZarr json file from netcdf files of unequal time length. HOT 2
- Support for small files in `_split_file` HOT 5
- Support for failed chunk requests HOT 13
- kerchunking zarr from OSN, bucket not found HOT 2
- Release? HOT 4
- Kerchunk doesn't translate HDF5 hard links HOT 3
- add to xarray backends docs HOT 22
- grib_tree's unexpected behaviour HOT 5
- Support the Open Meteo custom data format HOT 3
- NetCDF file has one time step, kerchunk-generated reference has nine time steps? HOT 3
- How to use subchunking HOT 10
- inline_threshold not encoding time value? HOT 9
- Got PermissionError: Access Denied when trying the example from the tutorial HOT 8
- KeyError when using Parquet-converted JSON reference for NetCDF4 access over HTTP HOT 12
- Chunk size mismatch on concatenation HOT 3
- MultiZarrToZarr append method - coo_map not working as expected HOT 5
- No module named 'eccodes' when trying to read kerchunked grib dataset with remote Dask cluster HOT 2
- Error with combining kerchunk mappings with MultiZarrToZarr HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kerchunk.