Comments (15)
Any solution for this
from walrus.
That's really odd... can you set up a script I can try to reproduce this?
from walrus.
Just speculating, but are there any save operations going on in a separate thread/process?
The model save operation does a delete as part of the update process. This could potentially introduce very small timing windows where other processes would experience exactly these failures.
from walrus.
Another thought. Would a lua script that sets all the key/value pairs of a model in an atomic operation potentially avoid having to do that delete operation during a save?
from walrus.
from walrus.
I have the same issue when using a Model to share status between two or more workers.
So, can I say that now Walrus models are not thread-safe?
from walrus.
So, can I say that now Walrus models are not thread-safe?
Has nothing to do with thread-safety. Redis is single-threaded, after all.
from walrus.
I also run into KeyErrors when there are two workers interacting with the same db: one is iterating through a Model.query() result while the other one is modifying/deleting model objects.
from walrus.
Yea there's no isolation. I strongly advise to use a relational db.
from walrus.
I am experiencing this as well. Isn't this something solved by doing the save using a pipeline. I think the problem is just finding the data in one index before it is in another. Pipelines atomically write.
from walrus.
I got around this by
-
Managing my own "namespaces" with a field on the Model to enforce isolation by restricting each client to a certain namespace.
-
Bypassing the Model API and interacting with Redis directly (only appropriate for certain cases). Huge performance boost since instead of doing
model.load(...) for model in models...
you can do something like:
with db.pipeline() as pipe:
for id_key in id_keys:
pipe.hgetall(id_key)
model_data_dicts = pipe.execute()
from walrus.
Yeah, definitely a nice performance boost, but sort of sep from the main issue. As is, without using a basic pipeline, I think save
is basically broken. Iterating the collection from any other thread/process can fail since it isn't fully written to redis yet.
from walrus.
I strongly advise to use a relational db.
No one should be using the walrus models code. I need to put a warning in the docs to that effect. It was an experiment and yeah it kinda works, but it's definitely nothing I would ever use myself and that is a big red flag to me.
from walrus.
That statement confuses me honestly. There are many use cases for Redis in which a modeled approach to transient data makes sense, to hide the complexity of the various data structures. That is how I landed here. I was doing the sorts of things you are doing, on my own, but then found Walrus. I have a case in which I most definitely do not want a relational database (very high throughput of transient objects).
I get though that it is your project and you want to deprecate that feature.
from walrus.
https://walrus.readthedocs.io/en/latest/models.html
deprecate that feature.
I do not intend to "deprecate" anything, and I did not intend to give that impression. What I do want to convey is that the implementation has a lot of rough edges and is too flaky for me to ever want to use it. I feel it is necessary to put a warning up that the code is "experimental"-quality, so that people can re-evaluate walrus if they are expecting some kind of stability guarantee.
from walrus.
Related Issues (20)
- model HOT 1
- Walrus models throwing error in fetching a column value HOT 4
- Pass list of redis clients
- Trimming streams based on MINID
- add CachedProperty HOT 1
- cache.cached doesn't work if the function returns None HOT 1
- "lock.event:" pollution HOT 4
- [feature]Any plan for supporting model relationship mapping? HOT 1
- Consumer group pending exception HOT 1
- VSCode type checking can't identify RedisPy's Redis as parent HOT 2
- Database() is falsy when database is empty HOT 2
- Empty queries in graph HOT 1
- A problem with query in graph.py HOT 1
- Lock event keys never expire HOT 1
- There is a problem. HOT 3
- Walrus Support for Redis Python Client 4.x+ HOT 2
- Is Expiring Hashes supported in Walrus? HOT 3
- Missing argument in method calling
- Support from_url database connection HOT 2
- Extending Containers with custom behavior HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from walrus.