halilozercan / pget

Fast download in chunks

Deprecated

I abandoned this project almost 3 years ago when I also stopped using Python. Please use tools like curl, aria2, wget for file downloads using CLI. Also consider using requests library as a Python package alternative.

PGet

pget offers a simple yet functional API that enables you to save large files from bandwidth limited servers such as Google Drive, Yandex Disk, etc.

Basically, Downloader operates asynchronously and creates multithreads that connect to specified url from different ranges.

How to get

You can install pget from PyPI using pip

pip install pget

Also you can clone this repository and install it using setuptools

git clone https://github.com/halilozercan/pget
cd pget
python setup.py install

Usage

Pget is essentially designed to be a python module, but it also provides a nice executable to cater your download needs quickly.

PGet - A tool for fast downloads

positional arguments:
  http://filedownload.url/path?args=given
                        File URL
  filename.txt          File name

optional arguments:
  -h, --help            show this help message and exit
  --chunks CHUNKS, -C CHUNKS
                        Chunk count

This is the output of well-known argument parser of python. For example

pget "http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_60fps_normal.mp4" bunny.mp4 -C 8

In this example, we download big buck bunny video to bunny.mp4 file from 8 different ranges.

If we want to do this in a python code

from pget.down import Downloader
downloader = Downloader(url, filename, chunk_count)

downloader.start()
downloader.subscribe(callback, callback_threshold)
downloader.wait_for_finish()

• by using subscribe, we get a callback from downloader whenever callback_threshold kilobytes of data is downloaded.
• start and wait_for_finish is like starting a thread and waiting to join. You can also run pget downloader in sync mod by start_sync()
• During the download, downloader object is updated regularly. So you can use it to get feedback in your application.

pget's Issues

Show Only Progressive Bar

Hi There,

I liked your project and found very useful in my application. But i wanted to ask if i could show only the progressive bar (by using command line) instead of showing other options [like url , saving to , etc]. Moreover, I Am Getting This Error(Below) On Using pget via command line:

Traceback (most recent call last):
File "/home//.local/bin/pget", line 8, in
sys.exit(main())
File "/home//.local/lib/python3.9/site-packages/pget/bin.py", line 109, in main
run(sys.argv)
File "/home//.local/lib/python3.9/site-packages/pget/bin.py", line 103, in run
downloader.wait_for_finish()
File "/home//.local/lib/python3.9/site-packages/pget/down.py", line 141, in wait_for_finish
while self.thread.isAlive():
AttributeError: 'Thread' object has no attribute 'isAlive'

And After That, The Download Starts.

Support for hls streams

This tool is the only fast pure python lightweight download I could find.

I would like to use this in a project of mine but I need support for downloading hls (m3u8) streams.

Can you look into it?

Releases 0.5 and 0.5.0 are the same for `pip`

I have a project using pget that I want to use release 0.5.0. However, no matter how I try to specify the dependencies, pip continues to install 0.5 instead of 0.5.0.

Can someone re-release 0.5.0 as 0.5.1 or 0.6.0 so I can use the correct version?

Thanks,
Martin

What is callback, callback_threshold?

I'm not sure how to define callback and callback_threshold arguments. It is not clear from the help.

Allow resuming of downloads

Right now, the download starts over

How can I catch the errors during downloading?

When it raise RuntimeError, I can not catch this error, or is there any way to return the status code like 404 403 something like that?

Status reporting randomly stops

Status reporting randomly stops at 75% or 82% even though the download itself is happening.

ModuleNotFoundError: No module named 'down'

Starting PGet 0.5 I get:

File ".../python-environments/lib/python3.7/site-packages/pget/__init__.py", line 1, in <module>
    from down.py import Downloader
ModuleNotFoundError: No module named 'down'

Follow semantic versioning

The releases page and pypi releases do not match. PyPi releases indicate two version with same tag.

Detect the file was changed during the download

It turns out if the file changed during the download after the final assembly we will have the downloaded file broken.
Is it possible to remeber "etag" header or something like this when retrieving each chunk? If we see the file changed - start over again?

add timeout choice for Downloader __init__

requests.get(self.url, stream=True, headers=self.headers)
May cause eternal wait for some case, add timeout for it may be better.

something like:
requests.get(self.url, stream=True, headers=self.headers, timeout=timeout)