iaslanidis / pylsyncd Goto Github PK
View Code? Open in Web Editor NEWPython Live Syncing Daemon
Home Page: http://code.google.com/p/pylsyncd/
License: GNU General Public License v3.0
Python Live Syncing Daemon
Home Page: http://code.google.com/p/pylsyncd/
License: GNU General Public License v3.0
PYLSYNCD -- Python daemon for asynchronous data replication using rsync ======================================================================= Project URL: http://code.google.com/p/pylsyncd/ Development: http://github.com/iaslanidis/pylsyncd/ Pylsyncd is a python implementation similar to lsyncd that uses rsync to synchronize local directories with several remote machines. Pylsyncd recursively monitors a set of directories using pyinotify, a pure Python module used for monitoring filesystems changes that relies on inotify. The main advantage of pylsyncd against lsyncd is that it aggregates changes within a configurable timeframe and is then able to optimize the remaining changeset, often resulting in a lot less rsync invocations. It uses message queues in order to synchronize several destination servers in parallel, saving up time when it is required to have more than one destination. It has been tested in heavy loaded environments. If you are interested in contributing or improving pylsyncd, do not hesitate to contact us. Quickstart ---------- If you're just starting out with pylsyncd, this might help you: $ mkdir -p /tmp/{src,dst} $ pylsyncd -d -i /tmp/src/./ /tmp/dst/ # ... watch the debug log and try to understand it ... While pylsyncd is running and monitoring /tmp/src, open another shell: $ mkdir /tmp/src/foo $ touch /tmp/src/foo/testfile # ... play around and watch the debug log ... The default aggregation window is 60 seconds, so your changes above are queued, optimized and finally synchronized. Performance ----------- You might want to lower the IO priority of pylsyncd, so the forked rsync processes won't interfere with your workload. This can be done using ionice(1) on Linux 2.6.13 or later (see man ionice(1)): $ ionice -n7 pylsyncd ... In addition you can fine-tune some (currently hardcoded) limits in pylsyncd.py. Some notes on paths ------------------- The pylsyncd source path must be local (we need to watch it with pyinotify) while the destination path may be any valid rsync(1) destination. If the source path is "." or contains the virtual root marker "/./" then all destination paths are rewritten to be relative to the specified local virtual root. For example: pylsyncd /tmp/foo/./bar/baz /tmp/destination will result in: /tmp/destination/bar /tmp/destination/bar/baz More examples: pylsyncd /tmp/source/./ /tmp/destination source = /tmp/source target = /tmp/destination (local machine), destination paths are rewritten to be relative to the virtual root at /tmp/source pylsyncd . srv2.example.com: srv3.example.net: srv4.example.org: source = "." (current working directory) target = homedir of current user at remote servers srv2.example.com, srv3.example.net and srv4.example.org, remote paths are relative to the current dir method = rsync on top of remote shell protocoll, eg. rsh/ssh. pylsyncd /var/www [email protected]: source = /var/www target = /var/www at remote server srv2.example.com, user www-data method = rsync on top of remote shell protocoll, eg. rsh/ssh. pylsyncd /var/www/./ [email protected]:/var/cluster/www source = /var/www target = /var/cluster/www at remote server srv2.example.com, user www-data, remote paths start at /var/cluster/www and are relative to local /var/www method = rsync on top of remote shell protocoll, eg. rsh/ssh. pylsyncd /var/www/./ srv2.example.com::webdata source = /var/www target = module webdata at remote server srv2.example.com, remote paths are relative to local /var/www method = rsync protocoll For more information on destination notation see man rsync(1).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.