slow / memory loading? about filedrop

[ProgerXP]

It's not supposed to happen... I am currently away and can't test this myself but it would help a lot if you could do a quick benchmark. For this make sure the demo (basic.html or the full one) uses non-minified filedrop.js and measure how long it takes at the following points in the script (e.g. set breakpoints and see what takes so long):

1 global.callAllOfObject(self, 'send', [files])
2 self.sendTo = function (url, opt) {
3 self.sendDataReadyTo = function (opt, e) {
4 self.onXhrSend = function (xhr, data) {
5 xhr.send(data)   // that's inside, when does it return?

[albertoa]

Argh. I need to do more testing, it may have something to do with how I did my test page.

Chrome crashes on the 1.7GB file with both: mine and basic.html with the perf. tracking code.

Firefox can handle the same large file. Here are the basic.html readings so far (run_1 was just a 17MB file so that I knew it was working, run_2 and run_3 are the 1.7GB file)

run_1 sendTo /api/0.3/uploadtest1 0.35365400000046066 ms after send event triggered

run_1 sendDataReadyTo 328.86348500000076 ms after send event triggered

run_1 callAllOfObject xhrSetup 329.88721500000065 ms after send event triggered

run_1 callAllOfObject xhrSend 329.993386000001 ms after send event triggered

run_1 onXhrSend pre-xhr.send 330.0686100000012 ms after send event triggered

run_1 onXhrSend post-xhr.send 345.6614440000003 ms after send event triggered

run_1 callAllOfObject progress 424.8156280000003 ms after send event triggered

run_1 callAllOfObject done 425.23010200000044 ms after send event triggered

run_1 done with profile.json 425.36242700000093 ms after send event triggered

run_1 callAllOfObject dragEnter 5852.188122000001 ms after send event triggered

run_1 callAllOfObject dragOver 5853.410059000002 ms after send event triggered

run_1 callAllOfObject dragOver 5865.696233000001 ms after send event triggered

run_1 callAllOfObject dragExit 5875.200305 ms after send event triggered

run_1 callAllOfObject dragLeave 5875.624634 ms after send event triggered

run_1 callAllOfObject dragEnter 6339.977332000002 ms after send event triggered

run_1 callAllOfObject dragOver 6340.871684000002 ms after send event triggered

run_1 callAllOfObject dragOver 6350.81112 ms after send event triggered

run_1 callAllOfObject dragOver 6355.827777 ms after send event triggered

run_1 callAllOfObject dragOver 6367.074907 ms after send event triggered

run_1 callAllOfObject dragOver 6371.810297 ms after send event triggered

run_1 callAllOfObject dragOver 6379.324185000001 ms after send event triggered

run_1 callAllOfObject dragOver 6488.876571000001 ms after send event triggered

run_1 callAllOfObject dragExit 6602.436978000002 ms after send event triggered

run_1 callAllOfObject dragEnter 6602.673027000001 ms after send event triggered

run_1 callAllOfObject dragLeave 6602.933510000001 ms after send event triggered

run_1 callAllOfObject dragOver 6603.464642999999 ms after send event triggered

run_1 callAllOfObject upload 6887.215845000001 ms after send event triggered

run_1 callAllOfObject fileSetup 7045.505580000001 ms after send event triggered

run_1 callAllOfObject send 7045.580889000001 ms after send event triggered

run_1 callAllOfObject inputSetup 7045.801383 ms after send event triggered

run_2 sendTo /api/0.3/uploadtest1 0.24048800000127812 ms after send event triggered

run_2 sendDataReadyTo 4432.3767990000015 ms after send event triggered

run_2 callAllOfObject xhrSetup 4433.614182000001 ms after send event triggered

run_2 callAllOfObject xhrSend 4433.828979999998 ms after send event triggered

run_2 onXhrSend pre-xhr.send 4433.908357000004 ms after send event triggered

run_2 onXhrSend post-xhr.send 6763.443578999999 ms after send event triggered

run_2 callAllOfObject progress 6854.878727000003 ms after send event triggered

run_2 callAllOfObject progress 7126.771561000001 ms after send event triggered

run_2 callAllOfObject progress 7488.581207000003 ms after send event triggered

run_2 callAllOfObject progress 8008.3040070000025 ms after send event triggered

run_2 callAllOfObject progress 8603.228421 ms after send event triggered

run_2 callAllOfObject progress 9387.424955000002 ms after send event triggered

run_2 callAllOfObject progress 10172.368819000003 ms after send event triggered

run_2 callAllOfObject progress 10799.514254000002 ms after send event triggered

run_2 callAllOfObject progress 10979.142178000002 ms after send event triggered

run_2 callAllOfObject done 10979.712576999998 ms after send event triggered

run_2 done with u8.vdi 10979.782821 ms after send event triggered

run_2 callAllOfObject dragEnter 237397.74590000004 ms after send event triggered

run_2 callAllOfObject dragOver 237398.74551700003 ms after send event triggered

run_2 callAllOfObject dragOver 237422.27389900002 ms after send event triggered

run_2 callAllOfObject dragOver 237424.13829700003 ms after send event triggered

run_2 callAllOfObject dragOver 237430.91055099998 ms after send event triggered

run_2 callAllOfObject dragOver 237444.25426800002 ms after send event triggered

run_2 callAllOfObject dragOver 237445.974972 ms after send event triggered

run_2 callAllOfObject dragOver 237453.458921 ms after send event triggered

run_2 callAllOfObject dragOver 237458.917129 ms after send event triggered

run_2 callAllOfObject dragOver 237467.144446 ms after send event triggered

run_2 callAllOfObject dragOver 237595.080702 ms after send event triggered

run_2 callAllOfObject dragOver 237607.154485 ms after send event triggered

run_2 callAllOfObject dragOver 237617.159552 ms after send event triggered

run_2 callAllOfObject dragOver 237689.305496 ms after send event triggered

run_2 callAllOfObject dragOver 237860.35113900003 ms after send event triggered

run_2 callAllOfObject upload 238014.182998 ms after send event triggered

run_2 callAllOfObject fileSetup 238107.15480199998 ms after send event triggered

run_2 callAllOfObject send 238107.330944 ms after send event triggered

run_2 callAllOfObject inputSetup 238108.59700299997 ms after send event triggered

run_3 sendTo /api/0.3/uploadtest1 0.26207199998316355 ms after send event triggered

run_3 sendDataReadyTo 2098.2072239999834 ms after send event triggered

run_3 callAllOfObject xhrSetup 2099.252622 ms after send event triggered

run_3 callAllOfObject xhrSend 2099.378131000005 ms after send event triggered

run_3 onXhrSend pre-xhr.send 2099.4666069999803 ms after send event triggered

run_3 onXhrSend post-xhr.send 3856.710815999977 ms after send event triggered

run_3 callAllOfObject progress 4068.5010380000167 ms after send event triggered

run_3 callAllOfObject progress 4429.60298299999 ms after send event triggered

run_3 callAllOfObject progress 5050.142741999996 ms after send event triggered

run_3 callAllOfObject progress 5638.7866260000155 ms after send event triggered

run_3 callAllOfObject progress 6127.268744000001 ms after send event triggered

run_3 callAllOfObject progress 6566.36986799998 ms after send event triggered

run_3 callAllOfObject progress 7313.744886 ms after send event triggered

run_3 callAllOfObject progress 7695.994991000014 ms after send event triggered

run_3 callAllOfObject progress 8043.283770000009 ms after send event triggered

run_3 callAllOfObject progress 8341.854453000007 ms after send event triggered

run_3 callAllOfObject progress 8536.250612999982 ms after send event triggered

run_3 callAllOfObject progress 8796.178563000023 ms after send event triggered

run_3 callAllOfObject progress 8816.903364999976 ms after send event triggered

run_3 callAllOfObject done 8817.391713000019 ms after send event triggered

run_3 done with u8.vdi 8817.446589999978 ms after send event triggered

I'll add the debug timing code to my test page and see if I can reproduce it there. I'll post the results later tonight.

[albertoa]

I think my problems may indeed be memory related (maybe it was taking long because I was running out of it).

Right now even my test form is working (although still no progress bar, but it may be too fast/something else).

Here is the memory consumption for firefox (from running top):

Page loaded no runs: 598MB
After run_1: 3.979GB
After run_2: 7.410GB
If I close that tab: 590MB

The timings are less than 10 seconds for each run.

It may be a garbage collection issue or a memory leak within the code. It does surprise me though that the memory grows around 2 * file_size each time I upload a file though.

[ProgerXP]

Have you tested partial reading (http://filedropjs.org/#partial)? This is specifically for avoiding loading entire file into the memory (now that I think about it it's normal that browser tries to load it entirely, it sends it in one request). With partial reading you can read smaller chunks and send them to the server.

Bad news is that FileDrop doesn't provide any easy way to handle these chunks - you'll need to merge the sliced data on the server side yourself (for this server modules exist, e.g. uploader module for nginx), as well as handle cases when a chunk gets lost or is corrupt. I've seen some other JS uploaders that supposedly make this easier; you might want to check them out.

[oytuntez]

This was a long time problem for me while using FileDrop JS. I was using the previous version for a long time, then we recently upgraded our application for the latest FileDrop version.

We have always had memory problems with large files, and I'm talking about files that are more than 100MB. And if you have the developer toolbar open in Chrome, it would already crash if the file size is around 60MB. This was completely impractical.

I dived into the source code and tried a couple of things. At the end, I found out that if the browser didn't have FileReader API (accessed via window.FileReader, line 2001), then everything worked well. It would still use AJAX to send the file, but it won't take the file into browser memory.

I assume this line is guilty:

    reader.readAsArrayBuffer(self.nativeFile)

So finally, I had to disable FileReader on our application:

    window.FileReader = null;

With this modification, FileDrop uses falls to line 2010, which has this comment:

    // Using early Chrome/Safari File API.

Disabling FileReader gained me:

• Page doesn't freeze anymore when selected a large file (it normally pauses for a short time even on 10MB files and the time increases with the file size)
• All other functionality such as progress events seems to be working well in all modern browsers.

I would highly appreciate any feedback on this.

[ProgerXP]

@oytuntez
I don't entirely understand your point. If you say that:

Disabling FileReader gained me:

Page doesn't freeze anymore when selected a large file (it normally pauses for a short time even on 10MB files and the time increases with the file size)
All other functionality such as progress events seems to be working well in all modern browsers.

...then what do you mean by:

So finally, I had to disable FileReader on our application: ...
With this modification, FileDrop uses falls to line 2010, which has this comment:

If you have disabled FileReader and it fails on sendDataReadyTo then how can you measure progress and such - it doesn't even start uploading?

I have already explained what I think is the cause of the problem with big files. You are trying to load the entire file into the memory. To be precise, it's not just a sequence of 60 MiB, it's an ArrayBuffer which must have some overhead (don't think it's supposed to hold this big amount of data at once). So big files are better uploaded in chunks. FileDrop doesn't provide you convenient way to do this (unlike other libraries that don't provide you fallback methods that FileDrop has) but it does have a versatile read() method that will let you read file parts and send them to the server that will merge them into one when done (there are server software modules for this, e.g. nginx' upload modules).

[oytuntez]

This line:

FileDrop uses falls to line 2010

should read "FileDrop falls to", which means "skips to" (in an IF statement). It doesn't say it "fails". So yeah, disabling FileReader also prevented loading the file into memory. I am able to upload GBs of files now. You may want to take a look at this...

[ProgerXP]

It doesn't say it "fails".

That's right, I've misread it.

I am able to upload GBs of files now. You may want to take a look at this...

That's interesting. Does it work reliably in Chrome, FF, IE?

[oytuntez]

Actually, yeah... All of them. No memory overload. Chrome and FF would normally just crash with large files.

However, you may want to do an extensive testing with this; because I am only sure that the parts I need are working fine (well, though, I should say I am using FileDrop's features very heavily...)

[ProgerXP]

Okay, I definitely need to do some research about it because FileReader is supposed to be the W3C way of doing the upload. It's strange that it's the bottleneck. It might take time though.

Thanks @oytuntez for bringing this to my attention.

[oytuntez]

You're welcome!

I'll also let you know if I see anything else related to FileReader. I was surprised as well; but everything seems to be working...

[adamteale]

@ProgerXP thank you for making FileDrop!
@oytuntez thanks for the temporary fix to get it uploading!

[davidrans]

Same issue here. If I upload a 180MB AVI file, Chrome consistently crashes with a memory error very soon after. The upload works fine on Firefox. I'll have to experiment with your solution @oytuntez.

[Pin0]

I have this issue when uploading a MP4 file of 765MB. I'll try the fix of @oytuntez and let you know if that works.

[ProgerXP]

I should definitely get to fix this...
@Pin0 any results?

[davidrans]

Disabling FileReader fixed the issue for me. Large uploads were always crashing Chrome with a memory error before that fix.

[Pin0]

Setting window.FileReader = null; did solve it for us. Unfortunately we now encounter memory issues on the server side but that is out of scope of this.

Setting filereader to null does disable the progress bar for us. So the users now have to look at a spinner and are unaware of when their file is uploaded.