a few tweaks to recent streaming changes about dfp_external_storage

[khoran]

I just discovered that the built-in File object sets the file_size type to "int", which can only represent a value going up to 2GB, so it is impossible to set the size of any file larger than that. So that changes the above problem from just "defensive coding" to a requirement, as the file_size field cannot be used at all for large (>2GB) files.

I did find that I can manually change the file_size field from int to float, giving me greater range. But that requires action on the users part.

[khoran]

An issue I found with changing the file_size field to a float is that the content_length header ends up getting set to a decimal value, ie, "100304505.0". This seems to be an invalid value, resulting in the browser not knowing the full size of the file and so not showing a proper progress bar. This can be fixed by casting the file_size value to an int:

response_values["headers"].append(("Content-Length", int(doc.file_size)))

[developmentforpeople]

Hi @khoran thanks again for your feedback and your work on this (and your pull request) :)

I just made a commit in develop solving these issues, could you test it at your side please?: dab1b34

Comments/answers by concept:

Another small issue I encountered was in the file function. The check if not response_values['response'] fails with a KeyError if the key 'response' is not even in the response_values dictionary. So an extra guard of 'response' not in response_values or ... would be good there.

Added the extra verification as you commented if "response" not in response_values or not response_values["response"]:. I added a log error too, to know at our side what makes that response not being set:

frappe.log_error(f"Error obtaining remote file content: {name}/{file}")

I just discovered that the built-in File object sets the file_size type to "int", which can only represent a value going up to 2GB, so it is impossible to set the size of any file larger than that. So that changes the above problem from just "defensive coding" to a requirement, as the file_size field cannot be used at all for large (>2GB) files.
I did find that I can manually change the file_size field from int to float, giving me greater range. But that requires action on the users part.

You are totally right, the INT(11) for file_size defined in File DocType limits the max file size to 2,048 MB. It would needed a BIGINT (maybe I am missing something that @gavindsouza (sorry for mention you! :) or any other from Frappe could answer us :). By the way I just added a new setting for the S3 Connection remote_size_enabled that allows enable the stat_object you implemented if you use files > 2GB, so if checked we use always the dfp_file_size that calls stat_object. More info in the recent commit: dab1b34

@khoran, I am not "sure" about changing the Int field to Float as solution. It helped as a temporal one, but I did not add the int casting to size :).

And nothing else, thank you very much for your time Kevin, have a good weekend!

[developmentforpeople]

Hi again @khoran just forgot to tell you that in your case you must check the new setting within bucket setup :):