GithubHelp home page GithubHelp logo

jonike / rstore Goto Github PK

View Code? Open in Web Editor NEW

This project forked from lycying/rstore

0.0 3.0 0.0 415 KB

rstore是一个使用golang实现的快速、轻量级的数据库中间件，使用redis协议交互，根据路由规则，某类key可同时存放在mysql，postgresql，redis中。自带web管理工具

Go 99.93% Makefile 0.07%

rstore's Introduction

alpha 版

rstore是一个使用golang实现的快速、轻量级的数据库中间件，可使用redis协议与其交互，根据路由规则，某类key可同时存放在mysql，postgresql，redis中（更多的后端如hbase也可以很容易支持）。为了方便配置管理，restore以插件形式提供了名为eyes的后台管理工具，访问任意节点的相应端口，即可进行web交互。

rstore主要解决使用redis作为存储但容量有限的场景，通过将数据转移到传统db中，节省成本，你可以一半key存放在postgresql中，另一半存放在mysql中。rstore尽量让此过程平滑过渡，业务方无须关心数据的具体组织方式。

特性

  • 使用redis协议访问交互，可使用redis-cli，redis-benchmark等现有工具和客户端
  • 解决用redis做存储的公司的成本，从reids切换到db只需要几分钟(数据迁移工具开发中)
  • 保持长连接，通过代理的方式减少缓存服务器的连接数
  • 所有配置可在线生效，不需要重启代理服务器
  • 使用key正则匹配的方式解决redis key的随意存储，没有备案的key将无法入库
  • 正则的group可作为hashkey，这和Twemproxy的hashtag类似但更强大
  • 可使用多层路由，比如1000-2000范围使用hash方式存放在redis，0-1000范围的使用mod存放在mysql，路由支持range、mod、hash、ketama_hash等
  • 如db有效率问题，db可前置redis做缓存
  • 为提高数据一致性，提供延迟读从的功能 (todo)
  • 支持单写、双写、多写，通过权重进行读请求分配
  • 配置管理可选etcd，zookeeper (todo)，这是抽象层
  • 有支持工具来进行问题排查，比如一个key从路由到后端的path (todo)
  • 管理接口通过提供rest api实现，你可以很容易集成到你自己的管理后台中
  • 监控通过提供rest api，你可以很容集成到grafana、 zabbix、nagios中
  • 插件方式，很容易做其他后端存储的功能 (调整中)

缺点

  • 不支持针对多个值的操作，比如取sets的子交并补等
  • 不支持Redis的事务操作
  • redis的不少特性不支持，不过以笔者的经验，现有的支持足够了
  • 增删节点数据还没有Reblance，你还是需要自己去写迁移工具 (暂时的)
  • 自动failover需要自己实现

性能

测试中。

与你的db配置有关。

扩容

对于扩容最简单的办法是：

  • 创建新的集群；
  • 双写两个集群；
  • 把数据从老集群迁移到新集群（不存在才设置值，防止覆盖新的值）；
  • 复制速度要根据实际情况调整，不能影响老集群的性能；
  • 切换到新集群即可，如果使用rstore代理层的话，可以做到迁移对读的应用透明。

设计

### 后台管理工具概览 数据库节点列表，可设置连接池等

组合数据库节点，组成数据库集群，如箭头所指为一主一丛节点

设置分片规则，规则条目可选择数据库集群，也可选择子分片规则

正则匹配key列表，hashslot指明哪个group作为分区键，注意0表示整个的key

schema

var redisScm = {
    "type": "object",
    "title": "Redis",
    "properties": {
        "Type":{
            "type":"string",
            "enum": ["redis"],
            "required":true
        },
        "Name":{
            "type":"string",
            "title": "Supply a unique name",
            "default": ""
        },
        "Enable":{
            "type": "boolean",
            "title": "If inused",
            "format": "checkbox"
        },
        "Server": {
            "type": "string",
            "title": "Server",
            "minLength": 2,
            "default": "localhost"
        },
        "Port": {
            "type": "integer",
            "title": "Port",
            "default": 6379,
        },
        "Mark": {
            "type": "string",
            "title": "Mark",
            "format": "textarea",
            "default": "",
        },
    },
};
var postgresScm = {
    "type": "object",
    "title": "Postgres",
    "properties": {
        "Type":{
            "type":"string",
            "enum": ["postgres"],
            "required":true
        },
        "Name":{
            "type":"string",
            "title": "Supply a unique name",
            "default": ""
        },
        "Enable":{
            "type": "boolean",
            "title": "If inused",
            "format": "checkbox"
        },
        "Url": {
            "type": "string",
            "title": "URL",
            "minLength": 12,
            "default": "postgres://postgres:postgres@localhost/postgres?sslmode=disable"
        },
        "MaxIdle": {
            "type": "integer",
            "title": "MaxIdle",
            "default": 10,
        },
        "MaxOpen": {
            "type": "integer",
            "title": "MaxOpen",
            "default": 10,
        },
        "MaxLifetime": {
            "type": "integer",
            "title": "MaxLifetime (Second)",
            "default": 60,
        },
        "Mark": {
            "type": "string",
            "title": "Mark",
            "format": "textarea",
            "default": "",
        },
    },
};
var mysqlScm = {
    "type": "object",
    "title": "Mysql",
    "properties": {
        "Type":{
            "type":"string",
            "enum": ["mysql"],
            "required":true
        },
        "Name":{
            "type":"string",
            "title": "Supply a unique name",
            "default": ""
        },
        "Enable":{
            "type": "boolean",
            "title": "If inused",
            "format": "checkbox"
        },
        "Url": {
            "type": "string",
            "title": "URL",
            "minLength": 12,
            "default": "user:password@tcp(localhost:5555)/dbname?tls=skip-verify&autocommit=true"
        },
        "MaxIdle": {
            "type": "integer",
            "title": "MaxIdle",
            "default": 10,
        },
        "MaxOpen": {
            "type": "integer",
            "title": "MaxOpen",
            "default": 10,
        },
        "MaxLifetime": {
            "type": "integer",
            "title": "MaxLifetime (Second)",
            "default": 60,
        },
        "Mark": {
            "type": "string",
            "title": "Mark",
            "format": "textarea",
            "default": "",
        },
    },
};
////////////////////////////////////////////
//dbgroup
var scm = {
    "type": "object",
    "title": "DB Group",
    "properties": {
        "Type":{
            "type":"string",
            "required":true,
            "enum": ["redis","postgres","mysql"]
        },
        "Name":{
            "type":"string",
            "title": "Supply a unique name",
            "default": ""
        },
        "Enable":{
            "type": "boolean",
            "title": "If inused",
            "format": "checkbox"
        },
        "ReplicateMode":{
            "type": "string",
            "title": "ReplicateMode",
            "required":true,
            "enum": ["writeall","writeone","discard"]
        },
        "Mark": {
            "type": "string",
            "title": "Mark",
            "format": "textarea",
            "default": "",
        },
        "Items":{
            "type": "array",
            "title":"DB Items",
            "items":{
                "type":"object",
                "title":"DBGroup Item",
                "properties": {
                    "Name":{
                        "type": "string",
                        "title": "DBRef",
                        "format": "select",
                        "required":true,
                        "enum": ["redis"]
                    },
                    "IsMaster":{
                        "type": "boolean",
                        "title": "Master",
                        "format": "checkbox",
                        "default":true,
                    },
                   "ReadWeight":{
                        "type": "integer",
                        "title": "ReadWeight",
                        "default":1
                    },

                },
            },
        }
    },
};

////////////////////////////////////////////
//shard
var scm = {
    "type": "object",
    "title": "Shard Defined",
    "properties": {
        "ShardType":{
            "type":"string",
            "ShardType":"string",
            "required":true,
            "enum": ["hash","mod","range","ketama_hash"],
            "default":"hash"
        },
        "Name":{
            "type":"string",
            "title": "Supply a unique name",
            "default": ""
        },
        "ShardMap":{
            "type": "array",
            "uniqueItems": true,
            "title":"Shard Map",
            "items":{
                "title":"Shard Item",
                "oneOf": [
                {
                    "title":"dbgroup",
                    "properties": {
                        "RefName":{
                            "type": "string",
                            "title": "Reference Name",
                            "format": "select",
                            "required":true,
                            "enum": []
                        },
                        "ShardStr":{
                            "type": "string",
                            "title": "ShardStr",
                        },
                        "RefType":{
                            "type": "string",
                            "title": "Reference Type",
                            "format": "select",
                            "required":true,
                            "enum": ["dbgroup"],
                            "minLength": 7,
                        },

                    },
                },
                {
                    "title":"shard",
                    "properties": {
                        "RefName":{
                            "type": "string",
                            "title": "Reference Name",
                            "format": "select",
                            "required":true,
                            "enum": []
                        },
                        "ShardStr":{
                            "type": "string",
                            "title": "ShardStr",
                        },
                        "RefType":{
                            "type": "string",
                            "title": "Reference Type)",
                            "format": "select",
                            "required":true,
                            "enum": ["shard"],
                            "minLength": 5,
                        },

                    },
                },

                ]
            },
        }
    },
};
////////////////////////////////////////////
//rule
var scm = {
    "type": "object",
    "title": "Rule Defined",
    "properties": {
        "Name":{
            "type":"string",
            "title": "Supply a unique name",
            "default": ""
        },
        "Order":{
            "type":"integer",
            "title": "Order",
            "default": 0
        },
        "Regexp":{
            "type":"string",
            "title": "Regexp",
            "minLength": 2,
            "default": "app:(\\d+):age"
        },
        "HashSlot":{
            "type":"integer",
            "title": "HashSlot",
            "default": 0
        },
        "ShardName":{
            "type":"string",
            "title": "ShardName",
            "format": "select",
            "required":true,
            "enum": []
        },
        "Example":{
            "type":"string",
            "title": "Example",
            "default": ""
        },
        "Mark":{
            "type":"string",
            "title": "Mark",
            "format": "textarea",
            "default": ""
        },
    },
};

rstore's People

Contributors

lycying avatar

Watchers

avatar avatar avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.