mar10 / pyftpsync

Synchronize directories using FTP(S), SFTP, or file system access.

pyftpsync

Synchronize directories using FTP(S), SFTP, or file system access.

Summary

Synchronize directories using FTP(S), SFTP, or file system access.

• This is a command line tool...
• ... and a library for use in your Python projects.
• Upload, download, and bi-directional synch mode.
• Allows FTP-to-FTP and Filesystem-to-Filesystem synchronization as well.
• Architecture is open to add other target types.

Note: Version 4.0 drops support for Python 2.

Quickstart

Python 3.8+ is required, pip recommended:

$ pip install pyftpsync --upgrade
$ pyftpsync --help

Note:
MS Windows users that only need the command line interface may prefer the MSI Installer or install using the Windows Package Manager:

> winget install pyftpsync

See Command Line Interface for details.

In addition to the direct invocation of upload, download, or sync commands, version 3.x allows to define a pyftpsync_yaml file in your project's root folder which then can be executed like so::

$ pyftpsync run

See Run from pyftpsync.yaml for details.

Documentation

Read the Docs for details.

pyftpsync's Issues

Add support for FTP active mode

Hi :

• Due to security concerns, do pyftpsync support active mode for FTP ?
• The FTP server denied passive mode and do not allow pyftpsync to write any lock file ?

Thanks !

resp* '230 Login successful.'
*cmd* 'CWD /tmp/b'
*resp* '250 Directory successfully changed.'
*cmd* 'PWD'
*resp* '257 "/tmp/b" is the current directory'
*cmd* 'TYPE I'
*resp* '200 Switching to Binary mode.'
*cmd* 'PASV'
*resp* '550 Permission denied.'
Could not write lock file: 550 Permission denied.
Upload /tmp/a
                  to ftp://localhost/tmp/b
*cmd* 'TYPE A'
*resp* '200 Switching to ASCII mode.'
*cmd* 'PASV'
*resp* '550 Permission denied.'
Skip remove lock file (was not written)
*cmd* 'QUIT'

Error importing SFTPTarget with paramiko > 3.0

When I import SFTPTarget (Python 3.10.11), I get the following error:

  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/xxxx, line 15, in <module>
    from ftpsync.sftp_target import SFTPTarget
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/ftpsync/sftp_target.py", line 50, in <module>
    class SFTPTarget(_Target):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/ftpsync/sftp_target.py", line 385, in SFTPTarget
    _paramiko_py3compat_u = paramiko.py3compat.u
AttributeError: module 'paramiko' has no attribute 'py3compat'

Versions:

paramiko                    3.1.0
pyftpsync                   4.0.0

This works with paramiko==2.12.0, so this is is apparently a breaking change in that library.

Request for multiples languages support.

Hi, could you add support for multiple languages?

So that messages are displayed in the predefined language, allowing us to make translations and send them as a pull request.
It would be great for the project to support multiple languages.

I can't upload files in ftp

I tried run in console

`pyftpsync upload ~/Documents/SHARE/LA_VOZ/ ftp://my_ftp.com/home/user/target``

Error is:

Could not remove lock file: path_to_ftp/target

File "C:\Users\khali\Documents\SHARE\venv-ftp\Scripts\pyftpsync-script.py", line 9, in <module>
    load_entry_point('pyftpsync==1.0.3', 'console_scripts', 'pyftpsync')()
  File "c:\users\khali\documents\share\venv-ftp\lib\site-packages\ftpsync\pyftpsync.py", line 198, in run
    s.run()
  File "c:\users\khali\documents\share\venv-ftp\lib\site-packages\ftpsync\synchronizers.py", line 128, in run
    res = self._sync_dir()
  File "c:\users\khali\documents\share\venv-ftp\lib\site-packages\ftpsync\synchronizers.py", line 375, in _sync_dir
    remote_entries = self.remote.get_dir()
  File "c:\users\khali\documents\share\venv-ftp\lib\site-packages\ftpsync\ftp_target.py", line 270, in get_dir
    self.ftp.retrlines("MLSD", _addline)
  File "C:\Users\khali\AppData\Local\Programs\Python\Python35\lib\ftplib.py", line 467, in retrlines
    with self.transfercmd(cmd) as conn, \
  File "C:\Users\khali\AppData\Local\Programs\Python\Python35\lib\ftplib.py", line 398, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "C:\Users\khali\AppData\Local\Programs\Python\Python35\lib\ftplib.py", line 364, in ntransfercmd
    resp = self.sendcmd(cmd)
  File "C:\Users\khali\AppData\Local\Programs\Python\Python35\lib\ftplib.py", line 272, in sendcmd
    return self.getresp()
  File "C:\Users\khali\AppData\Local\Programs\Python\Python35\lib\ftplib.py", line 245, in getresp
    raise error_perm(resp)
ftplib.error_perm: 500 Unknown command.

Download Synchronizer tries to upload when local file is older than remote

I think there is bug , the Download synchronizer tries to upload to remote an older local file, unless i don't get and it should be like that , but the remote should be not writable at all when using the Download Syncronizer

    def sync_older_local_file(self, local_file, remote_file):
        if self._check_del_unmatched(local_file):
            return False
        self._log_action("copy", "modified", "<", local_file)
        self._copy_file(self.local, self.remote, remote_file)

should be the opposite i guess

    def sync_older_local_file(self, local_file, remote_file):
        if self._check_del_unmatched(local_file):
            return False
        self._log_action("copy", "modified", "<", local_file)
        self._copy_file(self.remote, self.local, remote_file)

error install in python 3.5 on windows

I using python 3.5 on windows.

C:\Users\name>pip install  https://github.com/mar10/pyftpsync/archive/master.zip
Collecting https://github.com/mar10/pyftpsync/archive/master.zip
  Downloading https://github.com/mar10/pyftpsync/archive/master.zip (144kB)
    100% |################################| 153kB 13kB/s
Requirement already satisfied (use --upgrade to upgrade): colorama in c:\winpython-64bit-35\python-3.5.1.amd64\lib\site-packages (from pyftpsync==1.0.4.dev20160912)
Requirement already satisfied (use --upgrade to upgrade): keyring in c:\winpython-64bit-35\python-3.5.1.amd64\lib\site-packages (from pyftpsync==1.0.4.dev20160912)
Requirement already satisfied (use --upgrade to upgrade): pywin32-ctypes; sys_platform == "win32" in c:\winpython-64bit-35\python-3.5.1.amd64\lib\site-packages (from keyring->pyftpsync==1.0.4.dev20160912)
Installing collected packages: pyftpsync
  Running setup.py install for pyftpsync ... error
    Complete output from command C:\WinPython-64bit-35\python-3.5.1.amd64\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\B15D~1\\AppData\\Local\\Temp\\pip-76zjhw99-build\\setup.py';exec(compile(getattr(to
kenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:\Users\B15D~1\AppData\Local\Temp\pip-1no0bm_5-record\install-record.txt --single-version-externally-managed --compile:
    Initializing HOME environment variable to '\Users\วรรณพงษ์'
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: option --single-version-externally-managed not recognized

    ----------------------------------------
Command "C:\WinPython-64bit-35\python-3.5.1.amd64\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\B15D~1\\AppData\\Local\\Temp\\pip-76zjhw99-build\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__
file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:\Users\B15D~1\AppData\Local\Temp\pip-1no0bm_5-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in
C:\Users\B15D~1\AppData\Local\Temp\pip-76zjhw99-build\

Messages should not be printed to stdout when used as a library

It'd be nice if the library printed messages/errors via the standard Python logging module. Right now, the library prints to stdout in all circumstances, which bypasses logging functionality (and goes against the common practice of libraries not printing to stdout).

As you already use the _log_action method for all print messages, it would be straightforward to change the print(...) statement within it to use instead the logging system, and enable a stream handler pointed at stdout for the command line interface.

Add streaming support to handle large files

There is problem syncronizing large files , since its done all in memory , eventually you ll run out of it
Download to memory --> flush to disk

I implemented some kind of a streaming solution:

class StreamingFileTarget(FsTarget):
   
    from functools import partial

    def write_file(self, name, source, blocksize=DEFAULT_BLOCKSIZE, callback=None):

        def write_stream_data(fp_dst, callback, data):
            fp_dst.write(data)
            if callback:
                callback(data)

        self.check_write(name)

        with open(os.path.join(self.cur_dir, name), "wb") as fp_dst:
            write_stream_partial = partial(write_stream_data, fp_dst, callback)
            source.read_stream(name, write_stream_partial)

class StreamingFtpTarget(FtpTarget):

    def read_stream(self, name, callback):
        self.ftp.retrbinary("RETR %s" % name, callback)

But def _copy_file needs to be alerted

        dest.write_file(file_entry.name, src, callback=__block_written)

Add py35 testing to travis

See travis-ci/travis-ci#4794

Support for SSH key pairs

Thanks for this excellent package for syncing folder trees, it is exactly what we need for our project.

We want to use the private_key argument on pysftp.Connection for SSH key pairs previously configured using paramiko. I am planning to fork the repo and implement this, if it will be useful I am happy to do it as a PR, or maybe there is aready a way to do this that I have missed (we are planning to provide cross-platform support so would rather avoid using system keyrings).

Client won't delete local files on "pyftpsync download --delete [...]"

Tried with both command line and passing {'delete': True} to DownloadSynchronizer. Using "--resolve=remote" doesn't make any difference with the problem.
Python 3.7, server using pyftpdlib.

Cannot share local and remote targets for both upload and download synchronizers

If I create:

local_target = FsTarget("/path/to/local/dir")
remote_target = FtpTarget(path="/path/to/remote/dir", host="ftp.example.com")

then use these targets for both an upload and a download synchronizer:

ftp_downloader = DownloadSynchronizer(local_target, remote_target)
ftp_uploader = UploadSynchronizer(local_target, remote_target)
ftp_downloader.run()
ftp_uploader.run()

this leads to the following error on the ftp_uploader.run() line:

RuntimeError: target is read-only: <ftp:ftp.example.com/path/to/remote/dir + .>

This is because the constructors for DownloadSynchronizer and UploadSynchronizer set the remote/local targets as readonly, respectively, without making a copy of the provided object. A simple hack to fix this is to explicitly set the local/remote targets in this case to non-readonly in between:

ftp_downloader = DownloadSynchronizer(local_target, remote_target)
ftp_uploader = UploadSynchronizer(local_target, remote_target)
ftp_downloader.run()
# HACK: set remote to non-readonly
remote_target.readonly = False
ftp_uploader.run()

It would probably be better if the constructor explicitly sets the read/write status of both the remote and local targets. For example, the DownloadSynchronizer would have:

def __init__(self, local, remote, options):
    super(DownloadSynchronizer, self).__init__(local, remote, options)
    local.readonly = False
    remote.readonly = True

DownloadSynchronizer delete local files, when the 'delete' option is False

I used DownloadSynchronizer to download files from one remote ftp server. And when the files in the remote server was deleted, I found the DownloadSynchronizer would also delete the same local files. My opts is : opts = {"resolve": "skip", "verbose": 3,"delete" : False }. In order to avoid this issue, I just deleted 'pair.override_operation("delete_local", "restore")' under DownloadSynchronizer.re_classify_pair. Is this a bug or my opts has problem? Please kindly advice this. Thanks.

cannot sync on Windows server

Hi, I come to your project and tried to sync a directory on win10 with a win10 IIS FTP server. But unfortunately I got errors:
here is my demo:

from ftpsync.targets import FsTarget
from ftpsync.ftp_target import FtpTarget
from ftpsync.synchronizers import BiDirSynchronizer

local = FsTarget("c:\\win10disableimgpath\\")
user ="administrator"
passwd = "123456"
remote = FtpTarget("/controller", "10.1.101.195", username=user, password=passwd)
opts = {"resolve": "skip", "verbose": 1, "dry_run" : False}
s = BiDirSynchronizer(local, remote, opts)
s.run()

on the ftp://10.1.101.195/controller/ directory, I have a simple txt file in it.
like this. and the lock file is created yet.

.pyftpsync-lock.json 50 B 16/9/26 4:11:00
pysync.txt 32 B 16/9/26 4:02:00

and here is the error:

C:\Python27\python.exe C:/center/test/sk2.py
Synchronize c:\win10disableimgpath
Traceback (most recent call last):
with ftp:10.1.101.195/controller
File "C:/center/test/sk2.py", line 25, in
s.run()
File "C:\Python27\lib\site-packages\ftpsync\synchronizers.py", line 128, in run
res = self._sync_dir()
File "C:\Python27\lib\site-packages\ftpsync\synchronizers.py", line 375, in _sync_dir
remote_entries = self.remote.get_dir()
File "C:\Python27\lib\site-packages\ftpsync\ftp_target.py", line 270, in get_dir
self.ftp.retrlines("MLSD", _addline)
File "C:\Python27\lib\ftplib.py", line 437, in retrlines
conn = self.transfercmd(cmd)
File "C:\Python27\lib\ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "C:\Python27\lib\ftplib.py", line 339, in ntransfercmd
resp = self.sendcmd(cmd)
File "C:\Python27\lib\ftplib.py", line 249, in sendcmd
return self.getresp()
File "C:\Python27\lib\ftplib.py", line 224, in getresp
raise error_perm, resp
ftplib.error_perm: 500 Command not understood.

can you please help with this?

problem with old server ftp

I have an old server where it does not cover MLSD, but I need to synchronize in the normal way

python 3.5 types

I use python 3.5, the error types

Traceback (most recent call last):
File "/home/r1ka/backup.py/PostgreSQL_bacup2.py", line 58, in
s = BiDirSynchronizer(local, remote, opts)
File "/home/r1ka/backup.py/ftpsync/synchronizers.py", line 535, in init
super(BiDirSynchronizer, self).init(local, remote, options)
File "/home/r1ka/backup.py/ftpsync/synchronizers.py", line 66, in init
remote.open()
File "/home/r1ka/backup.py/ftpsync/ftp_target.py", line 88, in open
self.ftp.connect(self.host, self.port)
File "/usr/local/lib/python3.5/ftplib.py", line 146, in connect
if port > 0:
TypeError: unorderable types: str() > int()

Bidirectional sync over FTPS doesn't delete folders

Platform: Win10 / Python 3.7
Pyftpsync v: 3.1.0

Hi! Sorry if this is not a bug, but running the bidirectional sync on cmd does not remove directories (local or remote) that have been deleted since last sync. It actually tries to copy again the deleted folder. I found this very odd and decided to report. Am I doing something wrong?

CMD example: pyftpsync sync "{local}" "ftps://{USR}:{PSW}@{SERVER}:{PORT}/{remote}" --resolve=new --progress

Again, I'm sorry if this is the normal behaviour! At least that's not what I expected, and couldn't find anything about it in the docs. I can supply more details as needed. Thanks!

BidirSynchronizer

for bi-directional sync

Error to sync a ftp2 server

the return error is the command MLDS nos is valid, the 4.0 resolve this?

INFO:keyring.backend:Loading KWallet
INFO:keyring.backend:Loading SecretService
INFO:keyring.backend:Loading Windows
INFO:keyring.backend:Loading chainer
INFO:keyring.backend:Loading macOS
INFO:pyftpsync:Download to /home/guilherme.zulian/teste
                from ftp://200.201.27.44/
INFO:pyftpsync:Encoding local: utf-8, remote: utf-8
INFO:pyftpsync:Setting FTP encoding to utf-8 (was latin-1).
ERROR:pyftpsync:Could not write lock file: 553 Could not create file.
Traceback (most recent call last):
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/ftp_target.py", line 514, in get_dir
    self._ftp_retrlines_native("MLSD", _addline, encoding)
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/ftp_target.py", line 772, in _ftp_retrlines_native
    self.ftp.retrbinary(command, _on_read_chunk)
  File "/usr/lib/python3.8/ftplib.py", line 425, in retrbinary
    with self.transfercmd(cmd, rest) as conn:
  File "/usr/lib/python3.8/ftplib.py", line 382, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/usr/lib/python3.8/ftplib.py", line 348, in ntransfercmd
    resp = self.sendcmd(cmd)
  File "/usr/lib/python3.8/ftplib.py", line 275, in sendcmd
    return self.getresp()
  File "/usr/lib/python3.8/ftplib.py", line 248, in getresp
    raise error_perm(resp)
ftplib.error_perm: 500 Unknown command.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "t.py", line 11, in <module>
    s.run()
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 1268, in run
    res = super(DownloadSynchronizer, self).run()
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 213, in run
    res = self._sync_dir()
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 485, in _sync_dir
    remote_entries = self.remote.get_dir()
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/ftp_target.py", line 520, in get_dir
    raise RuntimeError(
RuntimeError: The FTP server does not support the 'MLSD' command.

Error downloading or syncing some specific folders sftp

Hi, Im using your pyftpsync tool to sync a remote sftp to an other machine.
This seemed to be working fine until it stopped at a certain folder.. No clue why this folder specifically.
I have tried syncing individual folders instead of syncing the entire tree.. this worked for some folders but not for all. And now i noticed it would also stop after x amount of files.
Any ideas?

Error when client sends "OPTS UTF8 ON"

Hey everyone! Sorry for the inconvenience, but I'm currently having this issue, trying to connect a pyftpsync client to a pyftpdlib server.
The client response is:
"Could not send 'OPTS UTF8 ON': '501 Invalid argument.'"
The pyftpdlib server logs no output.

From my searches, looks like pyftpsync shouldn't be sending this command at all, as it's deprecated. Please correct me if i'm wrong.
I appreciate any help!

Python 3.7
pyftpdlib 1.5.6
pyftpsync 3.1.0

download copies older local file in wrong direction

When i run pyftpsync --verbose --progress download -x (where the remote is a readonly ftp target) then an error is hit saying the dest is readonly (duhhh). I believe the error is in synchronizers.py line 981. The implementation of sync_older_local_file for DownloadSynchronizer should be:

def sync_older_local_file(self, local_file, remote_file):
if self._check_del_unmatched(local_file):
return False
self._log_action("copy", "modified", "<", local_file)
self._copy_file(self.remote, self.local, remote_file)

and not:

def sync_older_local_file(self, local_file, remote_file):
if self._check_del_unmatched(local_file):
return False
self._log_action("copy", "modified", "<", local_file)
self._copy_file(self.local, self.remote, remote_file)

Improve feedback while running

[v2.0] exclude files

Hi, I tried to use your very good script in mine to update my web site.
But I can't use exclude option.
When I try :

local = FsTarget( path_TmpGIT )
remote = FtpTarget("/", ftpHOST, username=ftpUSER, password=ftpPASS)
opts = {"force": True, "delete_unmatched": True, "verbose": 5, "no_color": True}
s = UploadSynchronizer(local, remote, opts)
s.run()

My .git folder is upload despite the default option. And when i tried, it's not better :

local = FsTarget( path_TmpGIT )
remote = FtpTarget("/", ftpHOST, username=ftpUSER, password=ftpPASS)
opts = {"force": True, "delete_unmatched": True, "verbose": 5, "no_color": True, "exclude": ".git"}
s = UploadSynchronizer(local, remote, opts)
s.run()

exclude directories does when pyftpsync is used within a script for upload

exclude file does work as expected
exclude directory does work when pyftpsync is called from a shell

application error with msi installer

Hi,

i installed the msi version "pyftpsync-2.0.0-win32.msi". That worked, but calling "pyftpsync --help" in command promt, a application error window pops up: Error Code 0xc000007b: application couldn't be started correctly.

Any advice on that?

Thanks in advance

Upload synchronize except where local file has been deleted?

I'm using v2.0.0 to sync files from one computer to an FTP server. Only a history of the last 7 files are kept on the local computer: I have a script that periodically deletes anything older than 7 days. I've found that UploadSynchronizer deletes the file on the remote FTP server after the local copy is deleted, which is not the behaviour I want. I'd like to sync local files to the FTP server, but not have to keep the full archive of files on the local computer.

I'm currently using options:

        upload_opts = {"force": True,
                       "resolve": "local",
                       "verbose": 3}

I think this is because pyftpsync keeps track of files it's seen before, and when it no longer sees the local file I delete, it also deletes it on the remote.

Is there some way to turn off this behaviour? I looked around the code and couldn't see a way - it seems that it's hard-coded in the operations map in resources.py?

Add option to run without meta files

.pyftpsync-meta.json file is being added within each directory on remote. Please add a flag to remove this file on completion if it is needed for ftp sync.

Import error

I install pypftsync with following detail on my machine:
pyftpsync/3.1.0 Python/3.9.0 Windows-10-10.0.19041-SP0

When I try to run script

`from ftpsync.ftp_target import FTPTarget
from ftpsync.synchronizers import UploadSynchronizer

local = FsTarget("~/temp")
user ="joe"
passwd = "secret"
remote = FTPTarget("/temp", "example.com", username=user, password=passwd)
opts = {"force": False, "delete_unmatched": True, "verbose": 3}
s = UploadSynchronizer(local, remote, opts)
s.run()`

I got error

from ftpsync.ftp_target import FTPTarget
ImportError: cannot import name 'FTPTarget' from 'ftpsync.ftp_target'

Please support.

Faild with FTP server with windows-1251 encoding

With code

from ftpsync.targets import FsTarget
from ftpsync.ftp_target import FtpTarget
from ftpsync.synchronizers import UploadSynchronizer
from ftpsync.synchronizers import DownloadSynchronizer
from ftpsync.synchronizers import BiDirSynchronizer
import ftpsync

import logging
import logging.handlers
from ftpsync.util import set_pyftpsync_logger

custom_logger = logging.getLogger("my.logger")
log_path = "D:\\TMP\\pyftpsync.log"
handler = logging.handlers.WatchedFileHandler(log_path)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
custom_logger.addHandler(handler)
set_pyftpsync_logger(custom_logger)

# import logging
# logger = logging.getLogger("pyftpsync")
# logger.setLevel(logging.DEBUG)

local = FsTarget("D:\\TMP\\ftpSync")
user = "blabla"
passwd = "blabla"

opts = {"encoding":"cp1251"}
remote = FtpTarget("/", "ftp.blabla.ru", username=user, password=passwd, extra_opts=opts)

opts = {"force": False, "delete_unmatched": True, "verbose": 6}
s = DownloadSynchronizer(local, remote, opts)
s.run()

I get an error:

INFO:my.logger:Encoding local: utf-8, remote: cp1251
Traceback (most recent call last):
  File "C:\Users\IR\PycharmProjects\untitled6\Sync_FTP.py", line 36, in <module>
    s.run()
  File "C:\Python39\lib\site-packages\ftpsync\synchronizers.py", line 1268, in run
    res = super(DownloadSynchronizer, self).run()
  File "C:\Python39\lib\site-packages\ftpsync\synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "C:\Python39\lib\site-packages\ftpsync\synchronizers.py", line 211, in run
    self.remote.open()
  File "C:\Python39\lib\site-packages\ftpsync\ftp_target.py", line 155, in open
    self.ftp.login(self.username, self.password)
  File "C:\Python39\lib\ftplib.py", line 407, in login
    resp = self.sendcmd('PASS ' + passwd)
  File "C:\Python39\lib\ftplib.py", line 279, in sendcmd
    return self.getresp()
  File "C:\Python39\lib\ftplib.py", line 242, in getresp
    resp = self.getmultiline()
  File "C:\Python39\lib\ftplib.py", line 228, in getmultiline
    line = self.getline()
  File "C:\Python39\lib\ftplib.py", line 210, in getline
    line = self.file.readline(self.maxline + 1)
  File "C:\Python39\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 4: invalid continuation byte

in codecs.py

    def decode(self, input, final=False):
        # decode input (taking the buffer into account)
        data = self.buffer + input
        (result, consumed) = self._buffer_decode(data, self.errors, final)
        # keep undecoded input until the next call
        self.buffer = data[consumed:]
        return result

error occure in (result, consumed) = self._buffer_decode(data, self.errors, final)
i have this data in input parameter:
b'230-\xc4\xee\xe1\xf0\xee \xef\xee\xe6\xe0\xeb\xee\xe2\xe0\xf2\xfc \xed\xe0 FTP \xf1\xe5\xf0\xe2\xe5\xf0 \xea\xee\xec\xef\xe0\xed\xe8\xe8 Maykor.\r\n230 User logged in, proceed.\r\n'

What decode as:
b'230-Добро пожаловать на FTP сервер компании Maykor.
230 User logged in, proceed.
'

Docs are wrong - opts requires "dry_run" : False

Without that dry_run set to False nothing happens.
opts = {"force": False, "delete_unmatched": True, "verbose": 3, "execute": True, "dry_run": False}
Thanks, it works great otherwise :)

from ftpsync.targets import FsTarget, UploadSynchronizer
from ftpsync.ftp_target import FtpTarget

local = FsTarget("~/temp")
user ="joe"
passwd = "secret"
remote = FtpTarget("/temp", "example.com", user, passwd)
opts = {"force": False, "delete_unmatched": True, "verbose": 3, "execute": True, "dry_run" : False}
s = UploadSynchronizer(local, remote, opts)
s.run()

ERROR - MLSD returned unsupported type: None

I'am running synchronization task by sceduler. Somtime this task start crushing with this error.
I fix this problem by delete meta file on remote ftp server. What is the root cause?

[2020-12-03 14:18:26,153] {{taskinstance.py:1128}} ERROR - MLSD returned unsupported type: None
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 966, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/get_cdr_SKPTUS.py", line 111, in export_to_asr
    s.run()
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/ftpsync/synchronizers.py", line 1106, in run
    res = super(UploadSynchronizer, self).run()
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/ftpsync/synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/ftpsync/synchronizers.py", line 213, in run
    res = self._sync_dir()
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/ftpsync/synchronizers.py", line 485, in _sync_dir
    remote_entries = self.remote.get_dir()
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/ftpsync/ftp_target.py", line 514, in get_dir
    self._ftp_retrlines_native("MLSD", _addline, encoding)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/ftpsync/ftp_target.py", line 772, in _ftp_retrlines_native
    self.ftp.retrbinary(command, _on_read_chunk)
  File "/usr/local/lib/python3.7/ftplib.py", line 447, in retrbinary
    callback(data)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/ftpsync/ftp_target.py", line 766, in _on_read_chunk
    _on_read_line(item)  # + LF)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/ftpsync/ftp_target.py", line 752, in _on_read_line
    callback(status, line_decoded)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/ftpsync/ftp_target.py", line 505, in _addline
    "MLSD returned unsupported type: {!r}".format(res_type)
NotImplementedError: MLSD returned unsupported type: None

Error if the local filesystem is read-only.

If I try to use upload from a read-only file system, the command fails with an exception.

pyftpsync upload --delete --resolve local /mnt/read-only-mount ftps://example.com/

Traceback (most recent call last):
  File "xxx/venv/bin/pyftpsync", line 8, in <module>
    sys.exit(run())
  File "xxx/venv/lib/python3.8/site-packages/ftpsync/pyftpsync.py", line 242, in run
    s.run()
  File "xxx/venv/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 1106, in run
    res = super(UploadSynchronizer, self).run()
  File "xxx/venv/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "xxx/venv/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 213, in run
    res = self._sync_dir()
  File "xxx/venv/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 571, in _sync_dir
    self.local.flush_meta()
  File "xxx/venv/lib/python3.8/site-packages/ftpsync/targets.py", line 404, in flush_meta
    self.cur_dir_meta.flush()
  File "xxx/venv/lib/python3.8/site-packages/ftpsync/metadata.py", line 223, in flush
    self.target.write_text(self.filename, s)
  File "xxx/venv/lib/python3.8/site-packages/ftpsync/targets.py", line 318, in write_text
    self.write_file(name, buf)
  File "xxx/venv/lib/python3.8/site-packages/ftpsync/targets.py", line 463, in write_file
    with open(os.path.join(self.cur_dir, name), "wb") as fp_dst:
OSError: [Errno 30] Read-only file system: '/mnt/read-only-mount/.pyftpsync-meta.json'

Crash when not setting verbose option

The following line should be a options.get

pyftpsync/ftpsync/synchronizers.py

Line 167 in 6711f2f

if self.options["verbose"] >= 3:

550 permission denied

on syncronize to a server with filezilla work, but when i use ftpsync thats a error is growing

guilherme.zulian@FPTI-023928:~$ python3 t.py 
INFO:keyring.backend:Loading KWallet
INFO:keyring.backend:Loading SecretService
INFO:keyring.backend:Loading Windows
INFO:keyring.backend:Loading chainer
INFO:keyring.backend:Loading macOS
INFO:pyftpsync:Download to /home/guilherme.zulian/teste
                from ftp://ftp.itaipu.gov.br/
INFO:pyftpsync:Encoding local: utf-8, remote: utf-8
INFO:pyftpsync:FEAT command failed: '550 Permission denied.'
WARNING:pyftpsync:Could not send 'OPTS UTF8 ON': '550 Permission denied.'
INFO:pyftpsync:Setting FTP encoding to utf-8 (was latin-1).
Traceback (most recent call last):
  File "t.py", line 11, in <module>
    s.run()
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 1268, in run
    res = super(DownloadSynchronizer, self).run()
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 213, in run
    res = self._sync_dir()
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 485, in _sync_dir
    remote_entries = self.remote.get_dir()
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/ftp_target.py", line 514, in get_dir
    self._ftp_retrlines_native("MLSD", _addline, encoding)
  File "/home/guilherme.zulian/.local/lib/python3.8/site-packages/ftpsync/ftp_target.py", line 772, in _ftp_retrlines_native
    self.ftp.retrbinary(command, _on_read_chunk)
  File "/usr/lib/python3.8/ftplib.py", line 425, in retrbinary
    with self.transfercmd(cmd, rest) as conn:
  File "/usr/lib/python3.8/ftplib.py", line 382, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/usr/lib/python3.8/ftplib.py", line 348, in ntransfercmd
    resp = self.sendcmd(cmd)
  File "/usr/lib/python3.8/ftplib.py", line 275, in sendcmd
    return self.getresp()
  File "/usr/lib/python3.8/ftplib.py", line 248, in getresp
    raise error_perm(resp)
ftplib.error_perm: 550 Permission denied.

Utf-8 encoding problems

Hi,
It looks like that there is a utf-8 encoding problem in writing/reading to json file as:

                  to ftp://192.168.43.230/tmp2
COPY NEW          >  01 - Születés.mp3          
SKIP DOWNLOAD     <  01 - Születés_.mp3          
Traceback (most recent call last):
  File "/usr/local/bin/pyftpsync", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python2.7/dist-packages/ftpsync/pyftpsync.py", line 174, in run
    s.run()
  File "/usr/local/lib/python2.7/dist-packages/ftpsync/synchronizers.py", line 972, in run
    res = super(UploadSynchronizer, self).run()
  File "/usr/local/lib/python2.7/dist-packages/ftpsync/synchronizers.py", line 723, in run
    res = super(BiDirSynchronizer, self).run()
  File "/usr/local/lib/python2.7/dist-packages/ftpsync/synchronizers.py", line 189, in run
    res = self._sync_dir()
  File "/usr/local/lib/python2.7/dist-packages/ftpsync/synchronizers.py", line 519, in _sync_dir
    self.local.flush_meta()
  File "/usr/local/lib/python2.7/dist-packages/ftpsync/targets.py", line 321, in flush_meta
    self.cur_dir_meta.flush()
  File "/usr/local/lib/python2.7/dist-packages/ftpsync/metadata.py", line 180, in flush
    s = json.dumps(self.dir, sort_keys=True)
  File "/usr/lib/python2.7/json/__init__.py", line 251, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 209, in encode
    chunks = list(chunks)
  File "/usr/lib/python2.7/json/encoder.py", line 434, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "/usr/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 361, in _iterencode_dict
    items = sorted(dct.items(), key=lambda kv: kv[0])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

the corresponding json looks like:

> {
>     "_disclaimer": "Generated by https://github.com/mar10/pyftpsync",
>     "_file_version": 2,
>     "_time": 1529133757.0,
>     "_time_str": "2018-06-16 10:22:37",
>     "_version": "2.0.0",
>     "mtimes": {},
>     "peer_sync": {
>         "192.168.43.230/tmp2": {
>             "01 - Sz\u00fclet\u00e9s.mp3": {
>                 "m": 1529137337.656211,
>                 "s": 13675540,
>                 "u": 1529137357.673848
>             },
>             ":last_sync": 1529137357.673848
>         }
>     }
> }
> 

Thanks,

Sync and delete synced files on remote?

Hi,

Is it possible to configure a unidirectional (remote to local) sync strategy like this?

1. Upon startup, sync all files on remote to local
2. Then delete all synced files on remote

Basically, the objective is to get all the files "new" files from remote to local.

ERROR: [Errno 17] File exists

Hi,

Python version: 3.8.10
pyftpsync version: 3.1.0 (also tested with 4.0.0-a3)

Started with the following Error from the Title:

$ pyftpsync download -v --progress . ftps://ftp.someserver.de
Download to /opt/workspace/working
                from ftps://ftp.someserver.de/
Encoding local: utf-8, remote: utf-8
No credentials found in keyring('pyftpsync', 'ftp.someserver.de').
Using credentials from .netrc file: user_for_sync:***.
Login as 'user_for_sync'.
Could not send 'OPTS UTF-8': '502 Unknown OPS format.'
Sent 'OPTS UTF8 ON'.
Setting FTP encoding to utf-8 (was latin-1).
Server time offset: -15.34 seconds.
EQUAL             =  [01_Nara_SP]
SKIP UPLOAD       >  transfer.log.all
EQUAL             =  [test]
SKIP UPLOAD       >  transfer-ftp-old.tar.gz
SKIP UPLOAD       >  transfers_non_nagios.log
COPY UNMODIFIED   <  [..]
ERROR: [Errno 17] File exists: '/opt/workspace'
    <EntryPair([..])>: (missing, unmodified) => copy_remote
Traceback (most recent call last):
  File "/home/engl/.local/bin/pyftpsync", line 8, in <module>
    sys.exit(run())
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/pyftpsync.py", line 242, in run
    s.run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 1268, in run
    res = super(DownloadSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 213, in run
    res = self._sync_dir()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 558, in _sync_dir
    res = handler(pair)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 851, in on_copy_remote
    self._copy_recursive(self.remote, self.local, remote_entry)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 333, in _copy_recursive
    dest.mkdir(dir_entry.name)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/targets.py", line 392, in mkdir
    os.mkdir(path)
FileExistsError: [Errno 17] File exists: '/opt/workspace'

Then I thought, well what if just have a Problem with relative paths like .. and .
Therefore I started pyftpsync again but this time with this exclude -x ".,..".
Now things getting stranger. I do get to the part where file operations happen. pyftpsync is creating the first folder, just to immediately afterwords trowing the same error again. But if you just restart he is going one folder further. So with every time I start the program I get one Folder closer to my remote structure.

$ pyftpsync download -v --progress -x ".,.." . ftps://ftp.someserver.de
Download to /opt/workspace/working
                from ftps://ftp.someserver.de/
Encoding local: utf-8, remote: utf-8
No credentials found in keyring('pyftpsync', 'ftp.someserver.de').
Using credentials from .netrc file: user_for_sync:***.
Login as 'user_for_sync'.
Could not send 'OPTS UTF-8': '502 Unknown OPS format.'
Sent 'OPTS UTF8 ON'.
Setting FTP encoding to utf-8 (was latin-1).
Server time offset: -15.37 seconds.
EQUAL             =  [01_Nara_SP]
SKIP UPLOAD       >  transfer.log.all
EQUAL             =  [test]
SKIP UPLOAD       >  transfer-ftp-old.tar.gz
SKIP UPLOAD       >  transfers_non_nagios.log
SKIP UNMATCHED    -  [..]
SKIP UNMATCHED    -  [.]
COPY UNMODIFIED   <  [02_Wakayama_SZ]
ERROR: [Errno 17] File exists: '/opt/workspace/working'
    <EntryPair([02_Wakayama_SZ])>: (missing, unmodified) => copy_remote
Traceback (most recent call last):
  File "/home/engl/.local/bin/pyftpsync", line 8, in <module>
    sys.exit(run())
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/pyftpsync.py", line 242, in run
    s.run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 1268, in run
    res = super(DownloadSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 213, in run
    res = self._sync_dir()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 558, in _sync_dir
    res = handler(pair)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 851, in on_copy_remote
    self._copy_recursive(self.remote, self.local, remote_entry)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 340, in _copy_recursive
    self._copy_recursive(src, dest, entry)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 333, in _copy_recursive
    dest.mkdir(dir_entry.name)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/targets.py", line 392, in mkdir
    os.mkdir(path)
FileExistsError: [Errno 17] File exists: '/opt/workspace/working'

$ pyftpsync download -v --progress -x ".,.." . ftps://ftp.someserver.de
Download to /opt/workspace/working
                from ftps://ftp.someserver.de/
Encoding local: utf-8, remote: utf-8
No credentials found in keyring('pyftpsync', 'ftp.someserver.de').
Using credentials from .netrc file: user_for_sync:***.
Login as 'user_for_sync'.
Could not send 'OPTS UTF-8': '502 Unknown OPS format.'
Sent 'OPTS UTF8 ON'.
Setting FTP encoding to utf-8 (was latin-1).
Server time offset: -15.33 seconds.
EQUAL             =  [01_Nara_SP]
EQUAL             =  [02_Wakayama_SZ]
SKIP UPLOAD       >  transfer.log.all
EQUAL             =  [test]
SKIP UPLOAD       >  transfer-ftp-old.tar.gz
SKIP UPLOAD       >  transfers_non_nagios.log
SKIP UNMATCHED    -  [..]
SKIP UNMATCHED    -  [.]
SKIP UNMATCHED    -  [.]
SKIP UNMATCHED    -  [01_Nara_SP]
COPY UNMODIFIED   <  [01_Nara_SP/02_T-Stop_Tester]
ERROR: [Errno 17] File exists: '/opt/workspace/working/01_Nara_SP'
    <EntryPair([01_Nara_SP/02_T-Stop_Tester])>: (missing, unmodified) => copy_remote
Changing to ftp root folder to remove lock file: /
Traceback (most recent call last):
  File "/home/engl/.local/bin/pyftpsync", line 8, in <module>
    sys.exit(run())
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/pyftpsync.py", line 242, in run
    s.run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 1268, in run
    res = super(DownloadSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 213, in run
    res = self._sync_dir()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 592, in _sync_dir
    self._sync_dir()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 558, in _sync_dir
    res = handler(pair)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 851, in on_copy_remote
    self._copy_recursive(self.remote, self.local, remote_entry)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 340, in _copy_recursive
    self._copy_recursive(src, dest, entry)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 333, in _copy_recursive
    dest.mkdir(dir_entry.name)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/targets.py", line 392, in mkdir
    os.mkdir(path)
FileExistsError: [Errno 17] File exists: '/opt/workspace/working/01_Nara_SP'

So I was like, okay If getting an error if the folder already exists. I add a only mkdir if not exists.
This looks like this:
in ftpsync/targets.py on line 389 I Changed the def mkdir:

    def mkdir(self, dir_name):
        self.check_write(dir_name)
        path = normpath_url(join_url(self.cur_dir, dir_name))
        if not self.dir_exists(path):
            os.mkdir(path)

    def dir_exists(self, path):
        return os.path.isdir(path)

But the result of the change was a Tried to navigate outside root error from def cwd. At this point I am not deep enough into this to fix the Problem.

I Hope I could explain the Problem.

Alex

Can't run in command line mode

Hi. I try to run this lib in command line mode, but a have trouble. I use python 3.6 on Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-91-generic x86_64).
Module is installed:
pip list | grep pyftpsync
pyftpsync (3.1.0)
Upgrade return segmentation fault
'pip install pyftpsync --upgrade'
[1] 10348 segmentation fault (core dumped) pip3 install pyftpsync --upgrade
I can't run this:

 ~  pyftpsync
zsh: command not found: pyftpsync

[suggestion] list of compatible ftp daemon

The pyftpsync is excellent. But the MLSD command is not implemented widely. At this moment, the results of google searching is not informative. It took me about 4 hours to figure out "pure-ftpd" works. I almost once gave up pyftpsync because that I thought there is no any linux ftpd supports MLSD.

Maybe it is worthy to have a compatible list of ftpd somewhere in the documentation.

Unable to navigate to working directory '' (Windows)

Both the local and remote computers are running on Windows 7. Running the script like that:

import pyftpsync.targets
import pyftpsync.ftp_target
import pyftpsync.synchronizers

local = pyftpsync.targets.FsTarget("temp")
user = "my_user"
passwd = "my_pass"
remote = pyftpsync.ftp_target.FtpTarget("/", "0.0.0.0", 18416, user, passwd)
opts = {
    "resolve": "remote",
    "verbose": 3,
    "dry_run": False
}
s = pyftpsync.synchronizers.DownloadSynchronizer(local, remote, opts)
s.run()

...and it gives me an RuntimeError:

Unable to navigate to working directory ""

Nothing changes if path = ''\\" or path = ".".

So I made a dirty and personal fix in ftp_target.py file (line 107 to 109) :

pwd = self.root_dir
# if pwd != self.root_dir:
#    raise RuntimeError("Unable to navigate to working directory %r" % self.root_dir)

Now it's working the way I wanted, but I think there may be a proper fix.

'new' not supported in interactive resolve in the DownloadSyncronizer

I think there is a bug in the download syncronizer.
It allows only for assert resolve in ("remote", "ask", "skip")
Then what we get in the on_conflict method is :

            elif resolve == "remote" or (is_newer and resolve == "old") or \
                    (not is_newer and resolve == "new"):
                self._log_action("copy", "conflict", "*<*", pair.local)
                self._copy_file(self.remote, self.local, pair.remote)

Which clearly just copies every files again and again since i kinda need to use remote all the time

[FEATURE] Storing metadata separate from the synced folder-pairs

I currently use pyftpsync to incremetally download a mailbox folder structure directly into a local mailbox folder, and if this metadata can speed up incremental functionality, i would not want them to be removed.

I would welcome some option to tell pyftpsync where to replicate the folderstructure and place these metadata in the "shadow-folderstructure"

if i could specify a meta-folder, where pyftpsync can store/look-for the meta files, and additionally the files could include their folder-path info, so there would be no need to create loads of folders.

Also for "virtually read-only" remotes you could store sync-metadata locally, which would eliminate the need for writable remote while doing download-functionality

This meta-data-folder solution is applied for example in offlineimap as a config value called metadatadir

how to make the program keep running and monitoring

I have two FTP servers need to sync with each other, and I run pyftpsync on my computer as a middle point, how to keep the program running so if each FTP server have file modification, it will be sync to the other FTP server immediately. I think of a scheduled task but if it's a big file that will take some time, how will I set the timing? if there is a build-in feature for this? thanks!

INTERMITTENT ERROR: MLSD returned unsupported type: {!r}".format(res_type)

I am not having much luck running pyftpsync from either the command line or in a python script.
I ran this today (I've replaced the real source and target folders with placeholders)
pyftpsync sync --dry-run --exclude=.* SOURCEFOLDER TARGETFTPFOLDER

It failed the first time.
Next time it worked and displayed in the terminal window a list of files it would
upload (sync) if not in dry run mode.
The next time I ran the code (removing dry run mode) I got the same error as the first time.
After that I got the error every time either in dry run or normal mode.
But it did seem to work once!
pyftpsync error.txt
Any ideas how to fix?

Question about usage

I'm looking for this library, but I have some trouble to find clear examples. I need to sync two folders in backbround. There exist a no blocking run() version of bidirectional sync? After set a source and dest to sync, is it possible to stop the sync and restart it with another couple of source and destination?

netrc

This line is confusing

pyftpsync/ftpsync/util.py

Line 284 in fd6471b

allow_netrc = not opts.get("no_netrc", False) and not force_user

Reads
👍if not no_netrc (so you didnt not specify no_netrc)
❓and not force_user (so you did NOT specify a user)
allow netrc.

This means netrc will only work if you DO NOT specify a username (sftp://machine.com)
It will not work if you do username (sftp://[email protected]) because not force_user

Not sure what the intention of this condition is, but it seems that not force_user just creates confusion and seems to work as intended without that condition.

If this is a desired function, can there be a log that is written of why netrc is being skipped?

Unable to navigate to working directory '/hetznertet' (now at '/hetznertet/')

While i tried to upload my directory to cdn push zone over FTP i geting this error
Unable to navigate to working directory '/hetznertet' (now at '/hetznertet/')

cdn zone patch is /hetznertet/ so as target i used cdn.com/hetznertet/ i tried cdn.com/hetznertet as well, but it give same errror

Special characters

I'm trying to upload some files with this example code:

from ftpsync.targets import FsTarget
from ftpsync.ftp_target import FtpTarget
from ftpsync.synchronizers import UploadSynchronizer

local = FsTarget("~/temp")
user ="joe"
passwd = "secret"
remote = FtpTarget("/temp", "example.com", username=user, password=passwd)
opts = {"force": False, "delete_unmatched": True, "verbose": 3, "dry_run" : False}
s = UploadSynchronizer(local, remote, opts)
s.run()

I've getting an error:
Traceback (most recent call last):..
File "upload.py", line 14, in
s.run()
File "C:\Python27\lib\site-packages\ftpsync\synchronizers.py", line 128, in run
res = self._sync_dir()
File "C:\Python27\lib\site-packages\ftpsync\synchronizers.py", line 471, in _sync_dir
self._sync_dir()
File "C:\Python27\lib\site-packages\ftpsync\synchronizers.py", line 456, in sync_dir
self.local.flush_meta()
File "C:\Python27\lib\site-packages\ftpsync\targets.py", line 547, in flush_meta
self.cur_dir_meta.flush()
File "C:\Python27\lib\site-packages\ftpsync\targets.py", line 351, in flush
s = json.dumps(self.dir, sort_keys=True)
File "C:\Python27\lib\json_init.py", line 251, in dumps
sort_keys=sort_keys, **kw).encode(obj)
File "C:\Python27\lib\json\encoder.py", line 209, in encode
chunks = list(chunks)
File "C:\Python27\lib\json\encoder.py", line 434, in _iterencode
for chunk in _iterencode_dict(o, _current_indent_level):
File "C:\Python27\lib\json\encoder.py", line 408, in _iterencode_dict
for chunk in chunks:
File "C:\Python27\lib\json\encoder.py", line 408, in _iterencode_dict
for chunk in chunks:
File "C:\Python27\lib\json\encoder.py", line 387, in _iterencode_dict
yield _encoder(key)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xaf in position 14: invalid start byte

Cant use this if build app use pyinstaller --windowed mode

Hi, thanks for this project.

I found that I can't use this module(version 3.1.0) if my app is builded by pyinstaller --windowed. I can run my app directly with command "python app.py".

I catch the exception:

File "ftpsync\util.py", line 104, in
OSError: [WinError 6] The handle is invalid

That line 104 in ftpsync\util.py is "IS_REDIRECTED = os.fstat(0) != os.fstat(1)", I change that to "IS_REDIRECTED = False" for temporary workaround.