Comments (7)
This can definitely happen in any kind of environment with high writes. I'm assuming you're properly awaiting/calling .close()
on your Writers.
It might also have to do with mmap, so you could try building without the mmap feature.
As far as what to do: I don't really expect to be able to fix all cases of cache corruption. What cacache is designed for isn't completely preventing corruption (since I don't consider that possible), but preventing you from reading bad data. My recommendation, and what I do in my own applications, is that if I read data and get an IntegrityError, the correct action is to delete the bad content and redownload the cached data. This should only happen occasionally, of course.
from cacache-rs.
The big surprise was the corruption of existing data. I did not anticipate that, I expected that writing identical content to a different key would not re-write and corrupt the content for the old key.
from cacache-rs.
I would not expect that either, that's very strange--considering there's separate files and filenames for everything. The only way I can see this happening is if you're trying to write a bunch of empty files (which would explain the NULs), and they all end up writing to the same empty content file, which will share the filename
from cacache-rs.
I wrote a wrapper for my usage of cacache::write that calls cacache::exists to ensure it doesn't exist before calling cacache::write. I assumed that it would significantly cut down on the amount of file corruption we're seeing in the wild. If your code already does something like that my wrapper will be ineffective and I need to do something more drastic.
from cacache-rs.
@bryanlarsen I wouldn't do that: cacache takes care of doing things as atomically as possible, which is how it can operate completely lockless. Having this kind of two-step operation might inject a race condition which will almost definitely be hit under high-concurrency environments where you're hitting the same data.
That said!
I have run into some weird cache corruption stuff when I was using cacache in orogene (which is VERY high throughput), but it only happened occasionally and I couldn't really track down why.
from cacache-rs.
Why? If I run two threads of if !exists() { write() }
simultaneously, if they don't race it does what I want and if it does race it degenerates into my old code, which would just be two threads of write()
simultaneously. And if it's not possible to call write()
simultaneously cacache is broken.
from cacache-rs.
because if something happens after the exists()
that corrupts or removes the package, you then fail to rewrite the package
from cacache-rs.
Related Issues (20)
- feature request: tokio::io::AsyncSeek support for Reader HOT 5
- about High frequency writing and deletion HOT 1
- Validation of documented system properties
- Fully deletion does not delete the value HOT 2
- Compilation error HOT 2
- feature request: cacache::Writer without a key HOT 2
- Multiple file hashes HOT 1
- Feature "tokio" does not compile with rustc 1.77.1 HOT 3
- Remove Key and clean the value
- The way I'm using cacache is very slow HOT 6
- "raw_metadata" is not raw after serialization HOT 3
- Deadlock when accessing across two threads. HOT 2
- Is cacache checking for hash collisions? HOT 1
- reading from a cacache::WriteOpts Writer, before its completed HOT 3
- Cannot delete fully index entry if content file doesn't exist.
- Some variants for copy/reflink/hard-link are missed
- [Question] Why the size of buf is not consistent?
- Unwrap on Error for Tokio Join Handle
- Lockless? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cacache-rs.