toqe / downloader Goto Github PK

Current download speed calculations are performed very often, what effect very high peaks up and down not really depend of how internet connection stable is.
Would be good to be able to control time for which average download speed is calculated.
I had a thought to sum up download speed every tick (1s) and ticks in while loop since its exact one second and then divide it but i maybe its better to do something directly in DownloadSpeedMonitor class for this.

If file does not exist or is not available on remote server Downloader throw throw new InvalidOperationException("Resuming not supported"); but it should inform that file is unavailable.

Hi

I am trying to split download a file from a ftp server. It seems that after it has downloaded its part, the thread doesn't disconnect from the server.

eg: the server I am downloading from, has a connection limit.

Once the parts are downloaded, then it is unable to continue with the rest of the file, as there are already too many connected connections to the server.

Thanks
Brendan

Did you ever considered to implement some trigger or variable which would allow user to control download speed? Multi-part downloader due to way of how is working is supposed to maximize download speed, but at same time can consume completly user bandwitch. That would be nice feature.

There is still issue correlated with large files.
This time when numberOfParts is bigger than 1 (try 4+) it take some time when file is created on hard drive.
Every second i can see progress of it looking on file size.
With 10GB file and SSD array it take around two minutes, then everything stuck.
I do not know if it is possible to create empty ghost file for speed up this process?
At this step if you restart downloading/app and keep same file name it appear like downloading start fine.

Is it possible to add calculation of remaining time, so user can see how much time left based on approx download speed connection?

• not correlated with Downloader, can be closed

Is there any way for resume downloading file after restart of application without starting from scratch?
This issue may appear when downloading big files due to reaching reconnect limit or closing application.
I assume need to provide alreadyDownloadedRanges but how to access it based on existing file?

The code is as follows:
public string SaveFile(HttpContext context = null, bool immediateSave = true)
{
var url = new Uri(GetFileRemoteUrl());
var requestBuilder = new SimpleWebRequestBuilder();
var dlChecker = new DownloadChecker();
var httpDlBuilder = new SimpleDownloadBuilder(requestBuilder, dlChecker);
var timeForHeartbeat = 3000;
var timeToRetry = 5000;
var maxRetries = 5;
var resumingDlBuilder = new ResumingDownloadBuilder(timeForHeartbeat, timeToRetry, maxRetries, httpDlBuilder);
List alreadyDownloadedRanges = null;
var bufferSize = 4096000;
var numberOfParts = 4;
var download = new MultiPartDownload(url, bufferSize, numberOfParts, resumingDlBuilder, requestBuilder, dlChecker, alreadyDownloadedRanges);
download.DownloadCompleted += Download_DownloadCompleted;
download.Start();
while ( !dlSaver.Complete )
{
Thread.Sleep(1000);
}
if ( dlSaver.IsError )
{
throw dlSaver.Exception;
}
return dlSaver.FilePath;
}

   ///This is always triggered
     
    private void Download_DownloadCompleted(Toqe.Downloader.Business.Contract.Events.DownloadEventArgs args)
    {

        args.Download.DetachAllHandlers();
    }

The console is no problem, what is the problem?

this software can user for product ？

Does this lib work with web servers that don't provide Content-Length, but respond with chunked encoding?
The content-length is either missing or set to -1, but transfer-encoding header comes with value "chunked".

Thanks

I searched nuget and couldn't find this project package.

Apparently downloader do not support large files.
When im trying to download 12 GB file it throw with throw new ArgumentException("offset < 0");.

hi
i would like to thankyout for creating this awesome library. i need to show some % to user how much its downloaded. can you please guide me how to do that i am following your multipart downlaod
thankyou

Large file downloads are evident in caton. I need help

Hello.
When initialize download with AlreadyDownloadedRanges with fully downloaded file
get NullReferenceException in SplitToDoRangesForNumberOfParts() - maxRange = null

var maxRange = this.ToDoRanges.FirstOrDefault(r => r.Length == this.ToDoRanges.Max(r2 => r2.Length));
this.ToDoRanges.Remove(maxRange);
var range1Start = maxRange.Start;

i fix like this

private void SplitToDoRangesForNumberOfParts()
{
	try
	{
		while (this.ToDoRanges.Count < this.numberOfParts)
		{
			var maxRange =
				this.ToDoRanges.FirstOrDefault(r => r.Length == this.ToDoRanges.Max(r2 => r2.Length));
			if (maxRange == null)
				return;
			this.ToDoRanges.Remove(maxRange);
			var range1Start = maxRange.Start;
			var range1Length = maxRange.Length / 2;
			var range2Start = range1Start + range1Length;
			var range2Length = maxRange.End - range2Start + 1;
			this.ToDoRanges.Add(new DownloadRange(range1Start, range1Length));
			this.ToDoRanges.Add(new DownloadRange(range2Start, range2Length));
		}
	}
	catch (Exception ex)
	{
		throw ex;
	}
}

private void DetermineFileSizeAndStartDownloads(DownloadCheckResult downloadCheck)
        {
            lock (this.monitor)
            {
                if (AlreadyDownloadedRanges.Count == 1 && AlreadyDownloadedRanges[0].Length == downloadCheck.Size)
                {
                    state = DownloadState.Finished;
                    OnDownloadCompleted(new DownloadEventArgs(this));
                    return;
                }

                this.ToDoRanges = this.DetermineToDoRanges(downloadCheck.Size, this.AlreadyDownloadedRanges);
                this.SplitToDoRangesForNumberOfParts();

                for (int i = 0; i < ToDoRanges.Count; i++)
                {
                    var todoRange = this.ToDoRanges[i];
                    StartDownload(todoRange);
                }
            }
        }

Hi Toqe,
Thank you for this excellent work.
I followed your example and succeeded to build a multipartdownload.
The problem is when i stop it and start it again, an exception comes out and says
InvalidOperationException: Invalid state: Running Toqe.Downloader.Business.Download.AbstractDownload.Start ()
Do you have any ideas?

I want to download two files at same time so i created two threads of Downloader.
I am unable in any way to start more than one thread of Downloader in same time.
After starting second one it just stay at 0% and do not open connection until first thread is finished.
I even tried to rename all variables and create separated void but result was same.
I noticed that if numberOfParts is set to 1 i can start second thread at same time and it downloading fine.

Large file downloads are evident in caton. I need help

hi
when I download file with 300-400 MB there is no problem
but when i use it to download 1.8 GB file its start slowing down after 59%

using System;
using System.Collections.Generic;
using System.Threading;
using Toqe.Downloader.Business.Contract;
using Toqe.Downloader.Business.Contract.Events;
using Toqe.Downloader.Business.Download;
using Toqe.Downloader.Business.DownloadBuilder;
using Toqe.Downloader.Business.Observer;
using Toqe.Downloader.Business.Utils;

namespace Downloader.Example
{
    public class Program
    {
        static bool isCompleted = false;

        public static void Main()
        {
            try
            {
                bool useDownloadSpeedThrottling = false;

                // Please insert an URL of a large file here, otherwise the download will be finished too quickly to really demonstrate the functionality.
                var url = new Uri("http://sqlserver.dlservice.microsoft.com/dl/download/B/8/0/B808AF59-7619-4A71-A447-F597DE74AC44/SQLFULL_CHS.iso");
                var file = new System.IO.FileInfo("npp_7.3.3_Installer.exe");
                var requestBuilder = new SimpleWebRequestBuilder();
                var dlChecker = new DownloadChecker();
                var httpDlBuilder = new SimpleDownloadBuilder(requestBuilder, dlChecker);
                var timeForHeartbeat = 3000;
                var timeToRetry = 5000;
                var maxRetries = 5;
                var resumingDlBuilder = new ResumingDownloadBuilder(timeForHeartbeat, timeToRetry, maxRetries, httpDlBuilder);
                List<DownloadRange> alreadyDownloadedRanges = null;
                var bufferSize = 4096;
                var numberOfParts = 4;
                var download = new MultiPartDownload(url, bufferSize, numberOfParts, resumingDlBuilder, requestBuilder, dlChecker, alreadyDownloadedRanges);
                var speedMonitor = new DownloadSpeedMonitor(maxSampleCount: 128);
                speedMonitor.Attach(download);
                var progressMonitor = new DownloadProgressMonitor();
                progressMonitor.Attach(download);

                if ( useDownloadSpeedThrottling )
                {
                    var downloadThrottling = new DownloadThrottling(maxBytesPerSecond: 200 * 1024, maxSampleCount: 128);
                    downloadThrottling.Attach(download);
                }

                var dlSaver = new DownloadToFileSaver(file);
                dlSaver.Attach(download);
                download.DownloadCompleted += OnCompleted;
                download.DownloadCancelled += Download_DownloadCancelled;
                download.DownloadStopped += Download_DownloadStopped;
                download.Start();

                while ( !isCompleted )
                {
                    Thread.Sleep(1000);

                    var alreadyDownloadedSizeInBytes = progressMonitor.GetCurrentProgressInBytes(download);
                    var totalDownloadSizeInBytes = progressMonitor.GetTotalFilesizeInBytes(download);
                    var currentSpeedInBytesPerSecond = speedMonitor.GetCurrentBytesPerSecond();

                    var currentProgressInPercent = progressMonitor.GetCurrentProgressPercentage(download) * 100;
                    var alreadyDownloadedSizeInKiB = (alreadyDownloadedSizeInBytes / 1024);
                    var totalDownloadSizeInKiB = (totalDownloadSizeInBytes / 1024);
                    var currentSpeedInKiBPerSecond = (currentSpeedInBytesPerSecond / 1024);
                    var remainingTimeInSeconds = currentSpeedInBytesPerSecond == 0 ? 0 : (totalDownloadSizeInBytes - alreadyDownloadedSizeInBytes) / currentSpeedInBytesPerSecond;

                    Console.WriteLine(
                        "Progress: " + currentProgressInPercent + "% " + "(" + alreadyDownloadedSizeInKiB + " of " + totalDownloadSizeInKiB + " KiB)" +
                        "   Speed: " + currentSpeedInKiBPerSecond + " KiB/sec." +
                        "   Remaining time: " + remainingTimeInSeconds + " sec.");
                }
            }
            catch ( Exception  ex)
            {
                Console.WriteLine("error:"+ex.Message);
            }
            Console.WriteLine("finished");
        }

        private static void Download_DownloadStopped(DownloadEventArgs args)
        {
            Console.WriteLine("DownloadStopped");
        }

        private static void Download_DownloadCancelled(DownloadCancelledEventArgs args)
        {
            Console.WriteLine("DownloadCancelled");
        }

        static void OnCompleted(DownloadEventArgs args)
        {
            // this is an important thing to do after a download isn't used anymore, otherwise you will run into a memory leak.
            args.Download.DetachAllHandlers();
            isCompleted = true;
        }
    }
}

In this example, I downloaded the resources that do not exist, the program supports hanging.
I registered the event did not trigger.
Another problem is that setting the timeout doesn't seem to work.

Such as https://github.com/Toqe/Downloader/archive/master.zip, when I start downloading this uri.
It will get exception.
Because those links cannot get response, it will crash in CheckDownload(Uri url, IWebRequestBuilder requestBuilder) .