Comments (12)
Here's the log on connecting:
2017-03-13 11:10:13,135 - ERROR - Error while parsing ssh_config file: [Errno 2] No such file or directory: '/home/backup/.ssh/config'. Trying to continue anyway...
2017-03-13 11:10:13,375 - DEBUG - starting thread (client mode): 0xed6c7290L
2017-03-13 11:10:13,376 - DEBUG - Local version/idstring: SSH-2.0-paramiko_2.1.2
2017-03-13 11:10:13,376 - DEBUG - Remote version/idstring: SSH-2.0-mod_sftp/0.9.9
2017-03-13 11:10:13,376 - INFO - Connected (version 2.0, client mod_sftp/0.9.9)
2017-03-13 11:10:13,376 - DEBUG - kex algos:[u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1', u'rsa1024-sha1'] server key:[u'ssh-rsa', u'ssh-dss'] client encrypt:[u'aes256-ctr', u'aes192-ctr', u'aes128-ctr', u'aes256-cbc', u'aes192-cbc', u'aes128-cbc', u'blowfish-ctr', u'blowfish-cbc', u'cast128-cbc', u'arcfour256', u'arcfour128', u'3des-ctr', u'3des-cbc'] server encrypt:[u'aes256-ctr', u'aes192-ctr', u'aes128-ctr', u'aes256-cbc', u'aes192-cbc', u'aes128-cbc', u'blowfish-ctr', u'blowfish-cbc', u'cast128-cbc', u'arcfour256', u'arcfour128', u'3des-ctr', u'3des-cbc'] client mac:[u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96', u'hmac-ripemd160', u'[email protected]'] server mac:[u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96', u'hmac-ripemd160', u'[email protected]'] client compress:[u'none'] server compress:[u'none'] client lang:[u''] server lang:[u''] kex follows?False
2017-03-13 11:10:13,376 - DEBUG - Kex agreed: diffie-hellman-group1-sha1
2017-03-13 11:10:13,376 - DEBUG - Cipher agreed: aes128-ctr
2017-03-13 11:10:13,377 - DEBUG - MAC agreed: hmac-sha2-256
2017-03-13 11:10:13,377 - DEBUG - Compression agreed: none
2017-03-13 11:10:13,484 - DEBUG - kex engine KexGroup1 specified hash_algo <built-in function openssl_sha1>
2017-03-13 11:10:13,485 - DEBUG - Switch to new keys ...
2017-03-13 11:10:13,585 - DEBUG - userauth is OK
2017-03-13 11:10:13,591 - INFO - Authentication (publickey) successful!
2017-03-13 11:10:13,593 - DEBUG - [chan 0] Max packet in: 32768 bytes
2017-03-13 11:10:13,597 - DEBUG - [chan 0] Max packet out: 32768 bytes
2017-03-13 11:10:13,597 - DEBUG - Secsh channel 0 opened.
2017-03-13 11:10:13,601 - DEBUG - [chan 0] Sesch channel 0 request ok
2017-03-13 11:10:13,602 - INFO - [chan 0] Opened sftp connection (server version 3)
2017-03-13 11:10:13,603 - DEBUG - [chan 0] listdir('MYSQL_BACKUPS_126/')
from sftpclone.
Thanks for your appreciation. :)
About the issue, I have a few questions:
- Can you try backing up different files (e.g. a bunch of text files)?
- Can you try backing up the same files to a different host (another Hetzner host would be perfect, but any SSH host could be useful too)?
from sftpclone.
Hey @aldur , thanks for the quick reply.
This is what we have tried so far:
- Created 10.000 txt files in a folder 'experiment' and tried to clone that.
=> EOF in transport thread - Fired up a fresh AWS EC2 instance and cloned the original MYSQL_BAKUPS folder to there
=> EOF in transport thread - Cloned the fresh 10.000 txt to EC2
=> EOF in transport thread
Is there any way to get more information out of sftpclone? In the past we used Rsync and even in case of problems with reading/cloning a certain file it would always complete. Is there any way to put sftpclone into 'skip problematic files' mode?
from sftpclone.
To be clear: we are using sftpclone 1.2
from sftpclone.
I'm having troubles reproducing the issue.
Here's what I did:
$ for i in {1..10000}; do base64 /dev/urandom | head -c 100 > file_$i.txt; done
$ cd ..
$ sftpclone -a test test-server:~/test -l DEBUG
On macOS 10.12.3 and Python 3.6 this completes without issues.
Let's try narrowing the environmental factor.
Any chance you can try using Python 3? Also: what was the OS of the EC2 instance?
from sftpclone.
Let's 😄 .
Any chance you can try using Python 3?
Unfortunately the server is running CentOS 6.8. So Python 2.7.8 is the best we can do right now.
Also: what was the OS of the EC2 instance?
Ubuntu 14.04
Additionally I will try to clone those 10.000 (dummy) txt files from the Ubuntu instance to our CentOS instance. Will report back here.
from sftpclone.
Thanks. Just to be clear, you made all the previous tests from the CentOS host, right?
I'm downloading a CentOS VM to make some tests too.
from sftpclone.
Yup.
from sftpclone.
I just cloned 10.000 dummy .txt files from the Ubuntu (Trusty, 14.04) server with Python 2.7.6 and sftpclone 1.2 to the CentOS server, without any problems. So the problem seems to reside within the/our CentOS environment..
from sftpclone.
Is your CentOS server 32 or 64 bits? Did you compile Python by yourself?
I tried the experiment by using a 64 bit CentOS 6.8 virtual machine, using Python 2.7.13 and it completed the transfer.
To be clear, I get a EOF in transport thread
at the bottom of my logs, but it means that the connection has been closed, and the number of files cloned is correct.
from sftpclone.
Hey @aldur, after counting the files I now see that everything is in fact in order. It just seemed like sftpclone errored out early in the process because of the big differences in transfer speed.
So it is working just fine! Apologies for the false negative.
Keep up the great work!
from sftpclone.
Great, glad to hear this. :)
from sftpclone.
Related Issues (20)
- Vignette for programmatic usage HOT 2
- Switch for never delete? HOT 3
- Upload create breaks HOT 27
- Crash if the ssh_config entry contains "IdentityFile" HOT 4
- a problem with unicode in file names HOT 15
- Generate new key pairs HOT 1
- Syntax error, sftpclone.py", line 718 HOT 3
- "@" not allowed to be in the username, password HOT 1
- Agent support doesn't work if there is more than one SSH-key loaded into the agent HOT 3
- Error caused by slash directions on Windows HOT 4
- Support of proxycommand HOT 5
- Feature request: reverse sftpclone support, where we can sync local directory with the contents from a remote dir HOT 1
- sftpclone breaks with Python 3.7
- Feature request: Allow for creating a remote folder HOT 3
- Feature request: Don't touch files matching excluded patterns on remote server
- Folder stucture Problem
- Exclude files HOT 1
- crash if .ssh/config is not available
- how to disable debug logging ?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sftpclone.