your-tools / pycp

cp and mv with a progress bar

pycp: cp and mv with a progressbar

What it looks like:

See pycp --help for detailed usage.

Development happens on github

Installation

pycp works both for any version greater than Python 3.4, and is installable with pip.

For Archlinux, a PKGBUILD is also available on AUR

Notes

• Implementation heavily inspired by the wonderful library progressbar by Nilton Volpato.
• I also maintain a similar tool written in rust called rusync. It has a different set of features (so you may want to stick with pycpy), but is much faster.
• If you are looking for a ncurses-based solution, vcp maybe the right choice for you http://www.freebsdsoftware.org/sysutils/vcp.html
• If you are looking for a more general solution to display progress bars when performing command-line operations, see clpbar: http://clpbar.sourceforge.net/

pycp's Issues

man pages are gone

Installed binaries crash with pkg_resources.DistributionNotFound when called during packaging.

Probably best to write a real doc in sphinx and let sphinx generate the man pages instead.

-g flag crashes pycp

When I do pycp -g (directory) (directory) I get the following:

Traceback (most recent call last):
  File "/usr/sbin/pycp", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/pycp/main.py", line 114, in main
    errors = transfer_manager.do_transfer()
  File "/usr/lib/python3.6/site-packages/pycp/transfer.py", line 354, in do_transfer
    error = ftm.do_transfer()
  File "/usr/lib/python3.6/site-packages/pycp/transfer.py", line 188, in do_transfer
    self.transfer_file()
  File "/usr/lib/python3.6/site-packages/pycp/transfer.py", line 229, in transfer_file
    self.callback(transfered)
  File "/usr/lib/python3.6/site-packages/pycp/transfer.py", line 339, in on_file_transfer
    self.progress_indicator.on_progress(progress)
  File "/usr/lib/python3.6/site-packages/pycp/progress.py", line 517, in on_progress
    self._render_second_line(progress)
  File "/usr/lib/python3.6/site-packages/pycp/progress.py", line 511, in _render_second_line
    filename=progress.src,
  File "/usr/lib/python3.6/site-packages/pycp/progress.py", line 391, in render
    length, string = component.render(kwargs)
  File "/usr/lib/python3.6/site-packages/pycp/progress.py", line 252, in render
    return len(text), text
TypeError: object of type 'NoneType' has no len()

easy_install on mac results in SandboxViolation

Just did a sudo easy_install pycp on my mac and got this error:

Searching for pycp
Reading http://pypi.python.org/simple/pycp/
Best match: pycp 6.1.2
Downloading https://pypi.python.org/packages/source/p/pycp/pycp-6.1.2.tar.gz#md5=6f370ea0d19fbc495c00a9c8d0973e7a
Processing pycp-6.1.2.tar.gz
Running pycp-6.1.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-znaR6L/pycp-6.1.2/egg-dist-tmp-fawFiu
/bin/sh: help2man: command not found
/bin/sh: help2man: command not found
error: Setup script exited with error: SandboxViolation: open('/usr/share/man/man1/pycp.1', 'wb') {}

The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand.  Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.

Progressbar sometimes too long

Steps to reproduce:

pycp /veeeeeeeeeeeeeeerrrrrrrrrrrrry/looooooooooooooooooong/paaaaaaaaaath/{src,dest}

The problem is that the output of describe_transfer is not guaranteed to take less than the line width, and after that it's too late to know which segment of the line should be truncated

port to python3

TODO

• run 2to3
• check that the tests still pass

Implement --exclude or --skip-hidden ?

Not sure we need this.

Plus, what should we do with skipped files that may be left behind at the end of the transfer in case we are moving things ?

Implement pycp --continue

Like wget -c, or rsync --append pycp could optionally support appending stuff to the end of a partially copied file

pvcp & pvmv Directories doesn't create new Source directory in destination..

I use: pymv ~/Videos /media/storage

the files and folders within Videos appear directly in /media/storage

it should appear as /media/storage/Videos.

pymv doesn't preserve timestamp

i used pvmv to move files from one drive to another and was surprised to find out it didn't preserve my orginal time-stamped info ion the files.. please make it default behavior.

please add preserve timestamp switch/operator for pycp too.

Add pyrm

I really like pycp+pymv.

Would it be possible to add a pyrm that performs an rm?

automatic test fails on windows

Something to do with read only directories.

To be investigated ....

Add test_dir to sdist

This way we can add a check() to the PKGBUILD.

add countdown for number of files being copied or moved.

add a switch or operator to tally the number of files and count them in verbose results.
e.g [01/279] files

pycp crashes when file size changes during transfer

If the file size increases while it's being transferred, pycp may crash with an assertion failure here

def on_file_transfer():
    self.global_pbar.update(xferd) 

def fraction()
    res = float(curval) / maxval
    assert res > 0
    assert res < 1
    return res

We can't really assume that xferd will always be smaller than src file size !

pycp is slow

version 8 is slower than 7, which is slower than cp, and not by a small margin.

Ideas:

• Optimize usage of python-cli-ui
• Use 2 processes, one for the disk I/O, and an other one for the output
• Throttle console output
• Use async stuff ?
• Rewrite in an other language. Rust ? Javascript ?

But first we need to do some serious benchmarking ...

pycp takes a long time to start when transferring many files

It spends a lot of time computing size of every file to transfer, and then only starts to copy files.
We should start transferring immediately

pycp check() fails during packaging

Fails with:

  OSError: [Errno 25] Inappropriate ioctl for device

when using ioctl to check for terminal size

Colorization of output

Are there any possibilities to add colorization to this tool? Everything is brilliant but absent of colorized stdout is real issue when copying/moving multiple files in bulk.

cp cp --reflink=auto feature

I am using Fedora 35 XFCE edition. One thing I have noticed:

I copied a 27 GB folder in Thunar in the same drive, the GUI file manager of XFCE. It took 4 seconds to copy.

If I run sudo btrfs fi du ~

It shows 27 GB is shared. In other words, only metadata was copied, not the actual data.

But if I use pycp on the same folder, it takes a few minutes to copy to the same drive. It means pycp is duplicating the files. I can see the free space also decreases.

I found here

that we can use cp --reflink=auto to copy with deduplication.

Is it possible to integrate this behavior of Thunar in pycp where it copies with de-duplication?

Thanks! 🙏🏽