Comments (5)
@amclain - I have similar concerns. I also hesitate to work on a complicated implementation for a feature that seems to really only be useful in a test environment. Making this kind of operation coherent in a distributed way with respect to CRDTs is nontrivial, and adding that kind of complexity is going to increase defect probabilities.
So, based on both the data security and implementation complexity concerns, I think I want to approach this differently. Rather than deleting data as an operation that affects the disk persistence layer and other nodes in the cluster, I think I want to view this as basically a "restart" command, that would remove all in-memory data structures and actors - basically, act as if you were doing a clean boot of the application, but without having to actually restart the OS process / Docker container.
Basically, this would give a cleaner, faster way to acheive the same effect as the restart-the-docker-container workaround that @amclain and @solnic are already doing.
It wouldn't add any complexity to the data management layer (which resolves the implementation complexity concern) and it seems to resolve the security concerns cited by @amclain because data isn't truly lost if it still exists on disk and/or in other nodes in the cluster - a robust usage of jylis in production that cares about data persistence shouldn't be relying only on the in-memory process anyway.
I'll work on this a bit today.
from jylis.
If I may add my opinion:
- too keep things easy, if persistence is just replaying the commands, just append the "FLUSHALL". this might be inefficient while replaying, but this can be tackled in a later optimization.
FLUSHALL
should delete all data on all nodes.
from jylis.
In my usage a FLUSHALL
command would be helpful for integration tests. However, it's more of a convenience because I am able to achieve the same outcome by running the database without disk persistence and restart it between tests to clear out the memory (using Docker). I'm curious what other use cases could benefit from FLUSHALL
.
My concern with the command is the case of a malicious user gaining access to a production database cluster and executing the FLUSHALL
command. Maybe that wouldn't be so bad if Jylis is being used as a cache, but definitely not good if it's the primary data store. If FLUSHALL
is implemented, maybe it should be opt-in at startup and ignored by cluster nodes that haven't opted in? Basically what I picture is opting a single node in for tests and opting out in production.
from jylis.
I think I want to view this as basically a "restart" command
Using SIGHUP
or SIGUSR2
may be an easy and secure way to do this. If you implement the command in the protocol just be careful with it being a potential DDoS vector (maybe still opt-in). I'm not sure how much network overhead is created if one node is continuously wiped clean and has to keep resyncing with the cluster -- something to think about.
from jylis.
I've implemented it as SYSTEM FORGETALL
: c4181b0
Opened #14 to track the remaining security concern.
from jylis.
Related Issues (9)
- Roadmap HOT 5
- Replicated data is not synced to disk HOT 4
- Read persistent data in reverse order HOT 2
- Security: Option to disable (or opt in?) to some SYSTEM commands.
- Document the "stable" executable in the Makefile. HOT 2
- Investigate portability issues with the docker image and prebuilt binary. HOT 2
- Feature: TLOG cursor HOT 2
- TLOG experiences performance issues with large recordset
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jylis.