Comments (12)
This change is related to file locking: ...
Yes! The error is gone after having this change reverted.
from silx.
Thank you for agreeing to investigate!
- Please download the latest GitHub version, I've just updated it. You don't need to install it, simply unpack somewhere.
- In parseq/gui/fileTreeModelView.py comment line 22 out (os.environ["HDF5_USE_FILE_LOCKING"]); this is a workaround for it to work with silx 2.0.0.
- run parseq/tests/test_fileTreeModelView.py that in a second will navigate to the hdf5 file: parseq/tests/data/hdf5/20240213s.h5
- Go into
instrument
and you will see this when the links are alive (except eiger_xes that was not copied):
and with albaem_2d_01, _02 and pcap links broken otherwise.
from silx.
@kklmn , PR #4074 that has been merged to
main
should fix this issue. Could you check that your issue is fixed with the current tip of the project?
Yes, checked, it's fixed.
Thank you!
from silx.
Since this was working with silx 1.1.2, it sounds to be a regression in silx v2.
Did you change any other package when switching from silx v1 to v2?
To be complete, what is the version of libhdf5 you are using (h5py.version.hdf5_version
)?
Are the HDF5 files you access being otherwise opened for writing or reading?
What is the silx developers’ view on hdf5 file locking
We try to support HDF5's file locking disabled. We even provide some helpers for accessing HDF5 files being written. We are relying on this for online data access (though my personal opinion is that it is not a good idea since we keep facing issues).
In silx view, we do the same "workaround" as you do: set HDF5_USE_FILE_LOCKING before anything imports h5py
silx/src/silx/app/view/main.py
Line 123 in 0abd43e
It's been working fine even with changes in libhdf5 related to file locking. The issue is that you should not write files without locking because they can get corrupted if another application is opening them.
from silx.
Hi,
Did you change any other package when switching from silx v1 to v2?
No. I've now checked it again on Python 3.11.6: after uninstalling silx and pip install silx==1.1.2
:
> h5py.version.hdf5_version
'1.14.2'
And here after commenting out
# os.environ["HDF5_USE_FILE_LOCKING"] = "FALSE" # to work with external links
all external links are still alive.
After upgrading silx to 2.0.0, h5py.version.hdf5_version is still '1.14.2'. And all external links are broken.
Doing
os.environ["HDF5_USE_FILE_LOCKING"] = "FALSE" # to work with external links
restores the links.
Are the HDF5 files you access being otherwise opened for writing or reading?
No, these are offline data files, accessed only by silx.gui.hdf5.Hdf5TreeModel.
from silx.
I tried to find out the cause, and nothing changed in silx.gui.hdf5
that could explain this...
Could you try to browse the same files with silx view --hdf5-file-locking <main_file.h5>
?
The --hdf5-file-locking
option enables file locking, so it would be a similar test since silx view
also uses Hdf5TreeModel
, and would be good for comparison.
If this reproduce the error, could you provide us the file? I couldn't reproduce with dummy files with external links
from silx.
Certainly, trying my files with silx view
was my first move. They behave normally there.
The difference is that in silx view
all hdf5 files are at the topmost level, whereas in my model they are nodes of the file tree, like here:
After a few days of searching I still can't see where is the change in silx that breaks the external links.
from silx.
This change is related to file locking: https://github.com/silx-kit/silx/pull/3939/files#diff-f7305bec0300313b03a14fece7a1e78cdcb4994f086ee18eb467567d212cfc38
But looking at parseq code I don't see how it can cause this issue, but maybe worth reverting to check.
Could you provide a way to reproduce the issue with parseq so I can investigate?
from silx.
Thanks for the easy way to reproduce!
This is indeed related to #3939 and to an issue we recently found in libhdf5/h5py regarding file locking and external links: https://forum.hdfgroup.org/t/external-link-and-file-locking-disabled-issue/12012: External links do not inherit file locking from the file linking them (and file locking is enabled by default) and you cannot open a file twice with different file locking in the same process.
I'm still not sure what's the best way out of this at silx level until this is fixed in libhdf5/h5py.
I feel it's always going to cause trouble one way or another unless either os.environ["HDF5_USE_FILE_LOCKING"] = "FALSE"
is set throughout the whole application or file locking is always enabled....
attn @woutdenolf
from silx.
Simple way to reproduce with silx view
:
- Create 2 files with an external link:
import h5py with h5py.File("main.h5", "w") as h5f: h5f["link"] = h5py.ExternalLink("linked.h5", "/data") with h5py.File("linked.h5", "w") as h5f: h5f["data"] = 1
- Comment:
silx/src/silx/app/view/main.py
Line 123 in 0abd43e
- Open the two files before accessing the external link:
silx view main.h5 linked.h5
from silx.
@kklmn , PR #4074 that has been merged to main
should fix this issue. Could you check that your issue is fixed with the current tip of the project?
We plan on making a 2.1.0 release in a short time.
from silx.
Thanks for the testing!
from silx.
Related Issues (20)
- Customization of roi handles HOT 3
- Deprecated: fontMetrics is deprecated in PyQt5 HOT 1
- Scipy 1.13.0 removes scipy.signal.gaussian HOT 2
- silx.opencl: Avoid executing OpenCL kernels at import time HOT 4
- Drop support for Python 3.7 HOT 1
- Dead Links in README.rst HOT 1
- numpy 2 deprecation warnings HOT 1
- overriding the match ratio in opencl/sift/match.py HOT 1
- h5py_utils fork deprecation warning
- Building `silx` against development version of numpy HOT 2
- `noexcept clause is ignored` warning in Cython code HOT 3
- Add `Tiled` data sources to browse Bluesky runs HOT 8
- Warning in Cython compilation HOT 3
- [2.1.0] Debian build and how to skip ascent downloading ? HOT 3
- How can I copy the data in Raw view? HOT 2
- silx view - HDF5view - improve scroll area HOT 1
- silx.gui.dialog GroupDialog and DatasetDialog are missing an API to select an hdf5 entity
- Add a 'auto' imagej like option HOT 2
- silx view: Large 2D datasets are open HOT 2
- Documentation: Describe conda packaging
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from silx.