'EOF in transport thread' error about sftpclone

[fastf0rward]

Here's the log on connecting:

2017-03-13 11:10:13,135 - ERROR - Error while parsing ssh_config file: [Errno 2] No such file or directory: '/home/backup/.ssh/config'. Trying to continue anyway...
2017-03-13 11:10:13,375 - DEBUG - starting thread (client mode): 0xed6c7290L
2017-03-13 11:10:13,376 - DEBUG - Local version/idstring: SSH-2.0-paramiko_2.1.2
2017-03-13 11:10:13,376 - DEBUG - Remote version/idstring: SSH-2.0-mod_sftp/0.9.9
2017-03-13 11:10:13,376 - INFO - Connected (version 2.0, client mod_sftp/0.9.9)
2017-03-13 11:10:13,376 - DEBUG - kex algos:[u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1', u'rsa1024-sha1'] server key:[u'ssh-rsa', u'ssh-dss'] client encrypt:[u'aes256-ctr', u'aes192-ctr', u'aes128-ctr', u'aes256-cbc', u'aes192-cbc', u'aes128-cbc', u'blowfish-ctr', u'blowfish-cbc', u'cast128-cbc', u'arcfour256', u'arcfour128', u'3des-ctr', u'3des-cbc'] server encrypt:[u'aes256-ctr', u'aes192-ctr', u'aes128-ctr', u'aes256-cbc', u'aes192-cbc', u'aes128-cbc', u'blowfish-ctr', u'blowfish-cbc', u'cast128-cbc', u'arcfour256', u'arcfour128', u'3des-ctr', u'3des-cbc'] client mac:[u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96', u'hmac-ripemd160', u'[email protected]'] server mac:[u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96', u'hmac-ripemd160', u'[email protected]'] client compress:[u'none'] server compress:[u'none'] client lang:[u''] server lang:[u''] kex follows?False
2017-03-13 11:10:13,376 - DEBUG - Kex agreed: diffie-hellman-group1-sha1
2017-03-13 11:10:13,376 - DEBUG - Cipher agreed: aes128-ctr
2017-03-13 11:10:13,377 - DEBUG - MAC agreed: hmac-sha2-256
2017-03-13 11:10:13,377 - DEBUG - Compression agreed: none
2017-03-13 11:10:13,484 - DEBUG - kex engine KexGroup1 specified hash_algo <built-in function openssl_sha1>
2017-03-13 11:10:13,485 - DEBUG - Switch to new keys ...
2017-03-13 11:10:13,585 - DEBUG - userauth is OK
2017-03-13 11:10:13,591 - INFO - Authentication (publickey) successful!
2017-03-13 11:10:13,593 - DEBUG - [chan 0] Max packet in: 32768 bytes
2017-03-13 11:10:13,597 - DEBUG - [chan 0] Max packet out: 32768 bytes
2017-03-13 11:10:13,597 - DEBUG - Secsh channel 0 opened.
2017-03-13 11:10:13,601 - DEBUG - [chan 0] Sesch channel 0 request ok
2017-03-13 11:10:13,602 - INFO - [chan 0] Opened sftp connection (server version 3)
2017-03-13 11:10:13,603 - DEBUG - [chan 0] listdir('MYSQL_BACKUPS_126/')

[aldur]

Thanks for your appreciation. :)

About the issue, I have a few questions:

1. Can you try backing up different files (e.g. a bunch of text files)?
2. Can you try backing up the same files to a different host (another Hetzner host would be perfect, but any SSH host could be useful too)?

[fastf0rward]

Hey @aldur , thanks for the quick reply.

This is what we have tried so far:

1. Created 10.000 txt files in a folder 'experiment' and tried to clone that.
   => EOF in transport thread
2. Fired up a fresh AWS EC2 instance and cloned the original MYSQL_BAKUPS folder to there
   => EOF in transport thread
3. Cloned the fresh 10.000 txt to EC2
   => EOF in transport thread

Is there any way to get more information out of sftpclone? In the past we used Rsync and even in case of problems with reading/cloning a certain file it would always complete. Is there any way to put sftpclone into 'skip problematic files' mode?

[fastf0rward]

To be clear: we are using sftpclone 1.2

[aldur]

I'm having troubles reproducing the issue.

Here's what I did:

$ for i in {1..10000}; do base64 /dev/urandom | head -c 100 > file_$i.txt; done
$ cd ..
$ sftpclone -a test test-server:~/test -l DEBUG

On macOS 10.12.3 and Python 3.6 this completes without issues.

Let's try narrowing the environmental factor.
Any chance you can try using Python 3? Also: what was the OS of the EC2 instance?

[fastf0rward]

Let's 😄 .

Any chance you can try using Python 3?

Unfortunately the server is running CentOS 6.8. So Python 2.7.8 is the best we can do right now.

Also: what was the OS of the EC2 instance?

Ubuntu 14.04

Additionally I will try to clone those 10.000 (dummy) txt files from the Ubuntu instance to our CentOS instance. Will report back here.

[aldur]

Thanks. Just to be clear, you made all the previous tests from the CentOS host, right?
I'm downloading a CentOS VM to make some tests too.

[fastf0rward]

Yup.

[fastf0rward]

I just cloned 10.000 dummy .txt files from the Ubuntu (Trusty, 14.04) server with Python 2.7.6 and sftpclone 1.2 to the CentOS server, without any problems. So the problem seems to reside within the/our CentOS environment..

[aldur]

Is your CentOS server 32 or 64 bits? Did you compile Python by yourself?
I tried the experiment by using a 64 bit CentOS 6.8 virtual machine, using Python 2.7.13 and it completed the transfer.
To be clear, I get a EOF in transport thread at the bottom of my logs, but it means that the connection has been closed, and the number of files cloned is correct.

[fastf0rward]

Hey @aldur, after counting the files I now see that everything is in fact in order. It just seemed like sftpclone errored out early in the process because of the big differences in transfer speed.

So it is working just fine! Apologies for the false negative.

Keep up the great work!

[aldur]

Great, glad to hear this. :)