jbd / msrsync Goto Github PK

View Code? Open in Web Editor NEW

464.0 464.0 73.0 135 KB

Multi-stream rsync wrapper

License: GNU General Public License v3.0

Makefile 1.04% Python 98.96%

multi-stream-rsync parallel python rsync

msrsync's People

Contributors

Stargazers

Watchers

msrsync's Issues

Thanks for help us , this software is very very useful

This is not issue, but only thanks.

file deletion

Hi,
I've tried to add to rsync params the --delete option for enable a syncronization scenario and it doesn't works.

It's a desidered behaviour?

Performance transfer to remote host

Hi,
I'm running 2 hosts with identical HW.
The NIC supports a bandwith of >20Gbits/sec.
Please check the output of iperf below.

ld4465:~ # iperf2 -c 192.168.100.11
------------------------------------------------------------
Client connecting to 192.168.100.11, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  3] local 192.168.100.13 port 60636 connected with 192.168.100.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  25.1 GBytes  21.5 Gbits/sec

I used NFS mount to connect the target server to the source server.
Then I started msrsync with these parameters:
ld4464:/backup # msrsync -P -p 16 /backup/ML1/ /mnt/ML1/

When I check the number of rsync-processes on the source host, I get only 9!

ld4464:~ # ps -ef | grep "rsync -a"
root     73333 73230 87 14:46 pts/1    00:07:56 /usr/bin/rsync -a --numeric-ids --from0 --files-from=/tmp/msrsync-tq0w9Z/tmpesZxD1 /backup/ML1 /mnt/ML1/
root     73334 73232 89 14:46 pts/1    00:08:10 /usr/bin/rsync -a --numeric-ids --from0 --files-from=/tmp/msrsync-tq0w9Z/tmpfbXcjJ /backup/ML1 /mnt/ML1/
root     73335 73231 85 14:46 pts/1    00:07:48 /usr/bin/rsync -a --numeric-ids --from0 --files-from=/tmp/msrsync-tq0w9Z/tmpvQTsDf /backup/ML1 /mnt/ML1/
root     73339 73334  0 14:46 pts/1    00:00:00 /usr/bin/rsync -a --numeric-ids --from0 --files-from=/tmp/msrsync-tq0w9Z/tmpfbXcjJ /backup/ML1 /mnt/ML1/
root     73340 73333  0 14:46 pts/1    00:00:00 /usr/bin/rsync -a --numeric-ids --from0 --files-from=/tmp/msrsync-tq0w9Z/tmpesZxD1 /backup/ML1 /mnt/ML1/
root     73341 73339 95 14:46 pts/1    00:08:42 /usr/bin/rsync -a --numeric-ids --from0 --files-from=/tmp/msrsync-tq0w9Z/tmpfbXcjJ /backup/ML1 /mnt/ML1/
root     73342 73340 92 14:46 pts/1    00:08:26 /usr/bin/rsync -a --numeric-ids --from0 --files-from=/tmp/msrsync-tq0w9Z/tmpesZxD1 /backup/ML1 /mnt/ML1/
root     73343 73335  0 14:46 pts/1    00:00:00 /usr/bin/rsync -a --numeric-ids --from0 --files-from=/tmp/msrsync-tq0w9Z/tmpvQTsDf /backup/ML1 /mnt/ML1/
root     73344 73343 92 14:46 pts/1    00:08:26 /usr/bin/rsync -a --numeric-ids --from0 --files-from=/tmp/msrsync-tq0w9Z/tmpvQTsDf /backup/ML1 /mnt/ML1/

Why are only 9 processes running?

--rsync --log-file being ignored?

Hi,
I think the always present --log-file option is overriding or ignoring the fact that log-file is set in the -r option.
Even with log-file set, the location seems to get put into /tmp.

I somewhat verified this by this hack:

if '--log-file' in options:
            rsync_cmd = "%s %s %s %s" % (RSYNC_EXE, options + ' --quiet --stats --verbose --from0', src + os.sep, dst)
        else:
            rsync_cmd = "%s %s %s %s" % (RSYNC_EXE, options + ' --quiet --stats --verbose --from0 --log-file %s' % rsync_log, src + os.sep, dst)

Which seems to place it to the desired file. Is this by design?

-d flag causes error

repro: run with -d flag. There is no else for the line that checks if it is present, so it attempts to continue with the loop without running rsync, so it throws the error below:

Process Process-2:
Traceback (most recent call last):
File "/misc/local/python-2.7.11/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/misc/local/python-2.7.11/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/bin/msrsync", line 824, in rsync_worker
rsync_mon_result = {"type": TYPE_RSYNC, "rsync_result": rsync_result, "size": bucket_size, "files_nr": bucket_files_nr, "jq_size": jobs_queue.qsize()}
UnboundLocalError: local variable 'rsync_result' referenced before assignment

installation is impossible on Debian 12

I need a version that does not require Python 2.X but can automatically work with Python 3.X, as Debian 12 makes it impossible to install Python 2.7.18, the latest. Is this possible? If not, is can you recommend a similar tool but a more updated one?

On centos 8 stream it fails

This fails
msrsync -p32 --rsync "-avzh --numeric-ids" /var/lib/mysql/* .
Source '/var/lib/mysql/aria_log.00000001' is not a directory

but this works
rsync -avzh --numeric-ids /var/lib/mysql/* .

Exclude Directories with msrsync not working

Hello,

Trying to add "--exclude-from=" with -r option while syncing directories with excluding few, which seems to be not working.

How to copy files to remote server by msrsync

I try to use your tools to copy files to remote computers, but I am failed. Is there any way to help?
I don't want to use NAS ，or NFC or Samba

Randomize the buckets?

Hi, I am syncing two file servers. the source fileserver is unraid which is an jbod consumer type fileserver. The destination where i'm running this script is freenas. I've NFS mounted the unraid share locally. Ideally would like to spread transfer across all jbod drives, but the file positions kind of sequential (that's the way unraid populates the drives).. so if this script could randomize or perhaps have threads starting in offset positions, it could spread the IO transfer? just a thought. thanks.

please add the option for sing file or multi files

When I tried to rsync one file from remote server, it said:

Source 'user@IP:/dir/222.7z' is not a directory

issue with rsync option --delete

it seems that passing the --delete option to rsync produces inconsisten backups when using p >= 2

I used to backup a large nfs directory and the copied directory was 39GB instead of 42.

I think that the --delete option deletes files handled by other rsync processes. Is this the case?

python3 support

Hello!

Does your tool support python3?

-r option not parsed?

Unless I'm greatly mistaken, the -r flag is not being processed into --rsync. For example, I had -r '-W --delete --inplace' in my command, but it did not throw an exception on the --delete, and looking at the destination, it was definitely using temp files rather than using --inplace.

--delete flag

Can you explain why a --delete flag would be catastrophic with msrsync? Ideally I'd like to be able to use msrsync to create a backup from one enterprise NAS to another, but without the delete flag, that's not really feasible.

How to use this to rsync over the network?

I could not figure out or find an example on how to transfer files other the network with msrsync

Displaying progress

Hello!

I have just recently found your program and it helped me alot to shift x TB of data from A to B.

However, I would like to ask you if you can implement another feature: progress display.
I mean not a forecast, but just to know how many bytes have been transferred, and how many are still pending.

THX

only 5 processes

I am testing capabilities of your app.
I made folder inside subfolder with 10 1GB files.

it works well with option -p 2,3,4,5 but after 5 it just keep using 5 threads. I can see that from refreshing target folder.
Do u know why?

What is "monq" and "jq" in the output

Hi,

Nice utility! I like the ease of it, and I think prefer it over fpsync.

I may be missing it, but what are the monq and jq in the progress line? It's not clear in the documentation (for me).

delete option

I ran the msrync to try to replace an existing rsync command and got the error message

error: Cannot use --delete option type with msrsync. It would lead to disaster :)

While I find the error message to be acceptable (in fact, this is the issue I'm trying to Google for solving in parallel rsyncs), it would be nice if the README of this repo discussed this limitation up front. It would have saved me from the time spent downloading and testing it.

how to resume from the last termination of msrsync

Hi, Thanks for the tool. This is really helpful. I have a requirement to copy 2 TB of files from source to target nfs file system.
After copying about 1TB, ./msrsync failed due to inodes utilisation on the target file system, which is fixed now.

But when I restarted the ./msrsync, it did not start actual rsync it is just processing existing files for almost 1 day now. Is there a way to quickly resume the copy without actually verifying the existing files? Please let me if my understanding is correct and how to overcome this?

./msrsync -p 32 --progress --stats
[12433616/12433616 entries] [1.2 T/1.2 T transferred] [226 entries/s] [23.6 M/s bw] [monq 0] [jq 0]

SSH

Is there a way to pass an ssh source to the script?

msrsync --rsync "-avzp --remove-source-files -e ssh" [email protected]:/mnt/Data/Media/Movies /mnt/pools/A/A0/Data/Media

Ability to process remote directories

Any update on being able to process remote directories?

Allowing remote hosts in destination

I'd like to be able to specify user@host:/dir as a remote destination. The code appears to only allow local destinations. Is this just a matter of modifying _check_srcs_dest ?

Keep Option

Can we get a little more clarification of the keep option? Does msrsync do a mirrored sync without that option specified? I'm a little unclear about exactly what keep does. By default rsync doesn't delete anything so i'm wondering if that's the same approach msrsync is taking

using rsync exclude

It is possible to use excludes to avoid syncing certain sub folders as a part of the -r arguments?

UnboundLocalError: local variable 'rsync_result' referenced before assignment

msrsync --stats -d -p 4 --rsync "-ravutg --exclude=lost+found" /source/ /destination/
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "./msrsync", line 824, in rsync_worker
rsync_mon_result = {"type": TYPE_RSYNC, "rsync_result": rsync_result, "size": bucket_size, "files_nr": bucket_files_nr, "jq_size": jobs_queue.qsize()}
UnboundLocalError: local variable 'rsync_result' referenced before assignment

Does not show progress

Hello, I'm running with the following command and it doesn't show the progress.

msrsync -P -p 8 --rsync "-azP" $DIR1/ $DIR

Python 3 support

Please make it compatible with Python3

sudo msrsync -p 8 -xa --progress --exclude /nfs / /nfs/k3s-wrkr02
/usr/bin/env: ‘python2’: No such file or directory

msrync with rsyncd

Hello,

Hope you are well and safe. I 'd like to ask whether it's possible to use msrsync with the rsync daemon in order to record the changes in the source and speed up the decision times for what needs to be synced or if this will confuse things for msrysnc. Thanks.

Best regards,
G

[root@media ~]# pwd
/root
[root@media ~]# ls -l ./test/|wc -l
100
[root@media ~]# du -hs test/
400K    test/
[root@media ~]# msrsync -p2 test/ /root/test2/
[root@media ~]# echo $?
0
[root@media ~]# ls -l ./test2/|wc -l
100
[root@media ~]# du -hs test2/
404K    test2/

Fails with space in path, even when quoted.

[root@media ~]# msrsync -p2 test/ "/root/test 2/"
rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.

Usage: rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
  or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.

Options
 -v, --verbose               increase verbosity
     --info=FLAGS            fine-grained informational verbosity
     --debug=FLAGS           fine-grained debug verbosity
[...]
 -4, --ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (-h is --help only if used alone)

Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers
rsync error: syntax or usage error (code 1) at options.c(2301) [client=3.1.2]
errors during rsync command (see '/tmp/msrsync-nhl2uc/0000/0000/tmp6jCVFu.log' rsync log file):
/usr/bin/rsync -aS --numeric-ids --quiet --verbose --stats --from0 --files-from=/tmp/msrsync-nhl2uc/0000/0000/tmp6jCVFu --log-file /tmp/msrsync-nhl2uc/0000/0000/tmp6jCVFu.log test /root/test 2/

msrsync error: somes files/attr were not transferred (see previous errors)

[root@media ~]# cat /tmp/msrsync-nhl2uc/0000/0000/tmp6jCVFu.log
cat: /tmp/msrsync-nhl2uc/0000/0000/tmp6jCVFu.log: No such file or directory

And with space in src path:

[root@media ~]# msrsync -p2 "/root/test 2/" "/root/test/"
rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.

Usage: rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
  or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.

Options
 -v, --verbose               increase verbosity
     --info=FLAGS            fine-grained informational verbosity
     --debug=FLAGS           fine-grained debug verbosity
[...]
 -4, --ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (-h is --help only if used alone)

Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers
rsync error: syntax or usage error (code 1) at options.c(2301) [client=3.1.2]
errors during rsync command (see '/tmp/msrsync-4aGcsq/0000/0000/tmpePSZwr.log' rsync log file):
/usr/bin/rsync -aS --numeric-ids --quiet --verbose --stats --from0 --files-from=/tmp/msrsync-4aGcsq/0000/0000/tmpePSZwr --log-file /tmp/msrsync-4aGcsq/0000/0000/tmpePSZwr.log /root/test 2 /root/test/

msrsync error: somes files/attr were not transferred (see previous errors)
[root@media ~]#

Exit code is 0 despite failure

Despite a failure in one of the worker thread, msrsync exited with code 0

Missing SSH support

The main reason for me in using this tools was to speed ssh transfers, but this doesn't seem supported?

$ time msrsync  -p8 --progress  --rsync ' -a -e "ssh -o Compression=no"'  [email protected]:/mnt/data/Bilder/ Bilder_syncdir/syncdir/
Source '[email protected]:/mnt/data/Bilder/' is not a directory

jbd / msrsync Goto Github PK

msrsync's People

Contributors

Stargazers

Watchers

Forkers

msrsync's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs