GithubHelp home page GithubHelp logo

Add equivalent of FLUSHALL command about jylis HOT 5 CLOSED

jemc avatar jemc commented on May 31, 2024
Add equivalent of FLUSHALL command

from jylis.

Comments (5)

jemc avatar jemc commented on May 31, 2024 1

@amclain - I have similar concerns. I also hesitate to work on a complicated implementation for a feature that seems to really only be useful in a test environment. Making this kind of operation coherent in a distributed way with respect to CRDTs is nontrivial, and adding that kind of complexity is going to increase defect probabilities.

So, based on both the data security and implementation complexity concerns, I think I want to approach this differently. Rather than deleting data as an operation that affects the disk persistence layer and other nodes in the cluster, I think I want to view this as basically a "restart" command, that would remove all in-memory data structures and actors - basically, act as if you were doing a clean boot of the application, but without having to actually restart the OS process / Docker container.

Basically, this would give a cleaner, faster way to acheive the same effect as the restart-the-docker-container workaround that @amclain and @solnic are already doing.

It wouldn't add any complexity to the data management layer (which resolves the implementation complexity concern) and it seems to resolve the security concerns cited by @amclain because data isn't truly lost if it still exists on disk and/or in other nodes in the cluster - a robust usage of jylis in production that cares about data persistence shouldn't be relying only on the in-memory process anyway.

I'll work on this a bit today.

from jylis.

mfelsche avatar mfelsche commented on May 31, 2024

If I may add my opinion:

  • too keep things easy, if persistence is just replaying the commands, just append the "FLUSHALL". this might be inefficient while replaying, but this can be tackled in a later optimization.
  • FLUSHALL should delete all data on all nodes.

from jylis.

amclain avatar amclain commented on May 31, 2024

In my usage a FLUSHALL command would be helpful for integration tests. However, it's more of a convenience because I am able to achieve the same outcome by running the database without disk persistence and restart it between tests to clear out the memory (using Docker). I'm curious what other use cases could benefit from FLUSHALL.

My concern with the command is the case of a malicious user gaining access to a production database cluster and executing the FLUSHALL command. Maybe that wouldn't be so bad if Jylis is being used as a cache, but definitely not good if it's the primary data store. If FLUSHALL is implemented, maybe it should be opt-in at startup and ignored by cluster nodes that haven't opted in? Basically what I picture is opting a single node in for tests and opting out in production.

from jylis.

amclain avatar amclain commented on May 31, 2024

I think I want to view this as basically a "restart" command

Using SIGHUP or SIGUSR2 may be an easy and secure way to do this. If you implement the command in the protocol just be careful with it being a potential DDoS vector (maybe still opt-in). I'm not sure how much network overhead is created if one node is continuously wiped clean and has to keep resyncing with the cluster -- something to think about.

from jylis.

jemc avatar jemc commented on May 31, 2024

I've implemented it as SYSTEM FORGETALL: c4181b0

Opened #14 to track the remaining security concern.

from jylis.

Related Issues (9)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.