frankfangtl / rump

Hot sync two Redis servers using dumps.

Hot sync two Redis databases using dumps.

Why

There's no easy way to get/sync data from an AWS ElastiCache Redis cluster.

@bdq: Hey, let's keep our staging Redis containers in sync with our AWS ElastiCache. BGSAVE and copy the .rdb?

@nixtrace: Yeah, awesome, let me try... Nope, not supported.

@bdq: Ah, that's bad. We'll have to set the containers as SLAVEOF?

@nixtrace: That makes sense, doing it... Nope, not supported.

@bdq: WAT. Let's use an open source tool to do the sync?

@nixtrace: Most of them use KEYS to get the keys, we'd DoS our own server.

@bdq: Let's write a script?

@nixtrace: Tried. Bash doesn't like key dumps, Ruby/Python + deps take more space than Redis inside the container.

@bdq and @nixtrace: Let's write it in Go?

Rump is able to transfer keys from an ElastiCache cluster or any Redis server to another Redis server, by only using SCAN, DUMP and RESTORE.

Features

• Uses SCAN instead of KEYS to avoid DoS your own server.
• Can sync any key type.
• Drops the TTL on purpose, since it wouldn't be in sync.
• Doesn't use any temp file.
• Uses buffered channels to optimize slow source servers.
• Uses pipelines to minimize network roundtrips.

Examples

# Sync local Redis DB 1 to DB 2.
$ rump -from redis://127.0.0.1:6379/1 -to redis://127.0.0.1:6379/2

# Sync ElastiCache cluster to local.
$ rump -from redis://production.cache.amazonaws.com:6379/1 -to redis://127.0.0.1:6379/1

# Sync protected ElastiCache via EC2 port forwarding.
$ ssh -L 6969:production.cache.amazonaws.com:6379 -N [email protected] &
$ rump -from redis://127.0.0.1:6969/1 -to redis://127.0.0.1:6379/1

Demo

Contribute

• We use GitHub issues to discuss everything: features, bugs, docs.
• Before sending a pull request always open an issue.

Install

You can find pre-compiled binaries on the releases page. If you don't see your OS/Arch there, just ask :)

Mentions

• Sticker Mule Blog
• The Changelog
• Go Weekly
• DB Weekly
• Redis Weekly
• Cron Weekly

Maintainers

nixtrace

License

Rump is licensed under the MIT License