Running Degoo api on Windows results in os.sep and os.path.abspath conflicts (and possibly others) about degoo

[DLBerger]

I've been working on these problems as I have been working under WSL2 on Windows 10.
I think I have code that is working well for most of these issues, but haven't tested it completely.
I can check-in my code on the working branch of the DLBerger fork if you want to see if it helps.
There is additional debugging that I added if you are at verbose 1 or above, so be aware of that.

[MikeBrownPhotographic]

I’ll try it later - sounds hopeful :-) Thanks On 11 Mar 2021, at 14:47, DLBerger ***@***.***> wrote:  I've been working on these problems as I have been working under WSL2 on Windows 10. I think I have code that is working well for most of these issues, but haven't tested it completely. I can check-in my code on the working branch of the DLBerger fork if you want to see if it helps. There is additional debugging that I added if you are at verbose 1 or above, so be aware of that. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#23 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATF2QQIKN6S26LILDSXCWUDTDDCVPANCNFSM4ZAASFSA>.

[DLBerger]

I've focused my work on 'put' and haven't looked at 'get' yet at all and have been trying to get important files saved. The browser API that Degoo provides doesn't handle 0-length files at all (I sent a bug report), but Bernd's interface does, though there are still issues as it often takes 3 or more times to actually get files under 1K to be accepted. Walking an entire tree with thousands of files is problematic as well, but I'm trying to work methodically, but is pretty slowly for now.

[MikeBrownPhotographic]

Downloaded and trying this out. I can degoo_put a file to the Degoo cwd, but I can't name a different target for the upload, as it seems it is still trying to treat C:\ as the root directory on Degoo:

C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working>py degoo_put test\req.txt \X51\test
Traceback (most recent call last):
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working\degoo_put", line 286, in
sys.exit(main())
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working\degoo_put", line 211, in main
result = degoo.put(args.local, args.remote, args.verbose, not args.force, args.dryrun, args.scheduled)
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working\degoo\util.py", line 1164, in put
return put_file(local_path, remote_folder, verbose, if_changed, dry_run, schedule)
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working\degoo\util.py", line 889, in put_file
dest = get_item(remote_folder)
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working\degoo\util.py", line 541, in get_item
raise DegooError(f"{p} does not exist in {path_str(part_id)}")
degoo.util.DegooError: C:\ does not exist in \X51\Backups
degoo_put: C:\ does not exist in \X51\Backups

This fails the same way with UNIX separators in the target:

py degoo_put test\req.txt /X51/test

[DLBerger]

Can you post the output from 'py degoo_pwd' and 'py degoo_ls /'?

[DLBerger]

And 'py degoo_put -v -v -v -v test\req.txt /X51/test'?

[DLBerger]

I see the problem. degoo_put allows you to copy a file to a remote folder, not to a remote file. You have to copy the file first and then use degoo_mv to rename it.

[MikeBrownPhotographic]

1. Degoo-working>py degoo_pwd
   Working Directory is /X51/Backups

2. Degoo-working>py degoo_put -v -v -v -v test\req.txt /X51/test
   test\req.txt is a file
   Traceback (most recent call last):
   File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working\degoo_put", line 286, in
   sys.exit(main())
   File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working\degoo_put", line 211, in main
   result = degoo.put(args.local, args.remote, args.verbose, not args.force, args.dryrun, args.scheduled)
   File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working\degoo\util.py", line 1164, in put
   return put_file(local_path, remote_folder, verbose, if_changed, dry_run, schedule)
   File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working\degoo\util.py", line 889, in put_file
   dest = get_item(remote_folder)
   File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-working\degoo\util.py", line 541, in get_item
   raise DegooError(f"{p} does not exist in {path_str(part_id)}")
   degoo.util.DegooError: C:\ does not exist in \X51\Backups
   degoo_put: C:\ does not exist in \X51\Backups

3. I am trying to copy to a remote folder which already exists, yes, so that should not be a problem here.

[MikeBrownPhotographic]

Here's an idea. Shouldn't the code include two separate versions of os? For example, os_local could work like at present, sensitive to the machine Python is running on, and os_degoo could be hard-coded to UNIX. Function calls would then have to decide which of the two was appropriate to call on any occasion. That approach ought to be portable and robust. What do you think?

I tried forcing os to use Posix, but it wasn't playing, as it unsurprisingly couldn't find the system values it was looking for on a Windows PC. Not sure what would be an elegant solution for that... Maybe someone with a UNIX machine could create a static array to hard code, or some such??

[DLBerger]

BAD IDEA. NEVER EVER hardwire "portable" code for a particular filesystem. Every programming language has filesystem-independent ways to access files. os.path.normpath() normalizes paths for the appropriate filesystem. I changed the code to use that for the source. There are still places where os.sep is being used and those need to be corrected by the use os.path.join() and other methods. Directly using os.sep is not a good idea in my opinion.

[MikeBrownPhotographic]

Ok. No need to shout :-) The challenge remains - how to make the code sensitive to the local os, without it assuming Degoo is running on the same os and so ruining all the remote path handling calls... On 11 Mar 2021, at 18:08, DLBerger ***@***.***> wrote:  BAD IDEA. NEVER EVER hardwire "portable" code for a particular filesystem. Every programming language has filesystem-independent ways to access files. os.path.normpath() normalizes paths for the appropriate filesystem. I changed the code to use that for the source. There are still places where os.sep is being used and those need to be corrected by the use os.path.join() and other methods. Directly using os.sep is not a good idea in my opinion. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#23 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATF2QQNDDMJAJN6LOQAKXLTTDD2JDANCNFSM4ZAASFSA>.

[DLBerger]

Sorry, but I've been working on portable systems for almost 40 years and getting everyone on the same page is often difficult.
If you look at the way Bernd communicates with Degoo, he always moves one directory at a time because you need the parent ID in order to create anything in the parent. I believe you'll find the various os.path functions work correctly on both local and remote paths. I added the os.path.normpath() on the source input to set an absolute path to the path, so all the following routines would work consistently (at least for the degoo_put command for now - I believe anyway).

[MikeBrownPhotographic]

I’m right with you on portability. But why is the code looking for C:\ as the root _remote_ directory? Something is still amiss. On 11 Mar 2021, at 18:34, DLBerger ***@***.***> wrote:  Sorry, but I've been working on portable systems for almost 40 years and getting everyone on the same page is often difficult. If you look at the way Bernd communicates with Degoo, he always moves one directory at a time because you need the parent ID in order to create anything in the parent. I believe you'll find the various os.path functions work correctly on both local and remote paths. I added the os.path.normpath() on the source input to set an absolute path to the path, so all the following routines would work consistently (at least for the degoo_put command for now - I believe anyway). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#23 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATF2QQKRZJYNFVU4F5OFB23TDD5IPANCNFSM4ZAASFSA>.

[MikeBrownPhotographic]

The erroneous "C:" prefix is being added to the path by the last line in lib.py, in the function absolute_remote_path():
Given a value of path = "\X51\test", this call returns "C:\X51\test":
return os.path.abspath(os.path.join(CWD["Path"], path))
It is the os.path.abspath() call which does this, because it is interpreting it as a Windows absolute path. But of course there is no C:\ on the remote Degoo drive. We need a UNIX abspath() at this point, not a Windows one, surely? But how to do that correctly...

[DLBerger]

Now that's a different problem.
Looking at the code, I think it should be simply: return os.path.join(CWD["Path"], path)

[DLBerger]

Well, that's not correct.

[MikeBrownPhotographic]

Yes, if I remove the two calls to os.path.abspath(), then degoo_put works fine. But I thought this call had been put in to fix some other problem?
Also, I have to use Windows-style "" folder separators on both the local (which is fine) and remote (which is odd) file paths. OK once you know.

[MikeBrownPhotographic]

Why do backslashes disappear from posts?? Let's try this -
...Windows-style "\" folder separators...

[DLBerger]

OK, I made a small change to degoo/lib.py. Could you see if it helps?

[MikeBrownPhotographic]

Sure thing - it will be tomorrow now - it’s getting late in the UK :-) On 11 Mar 2021, at 21:55, DLBerger ***@***.***> wrote:  OK, I made a small change to degoo/lib.py. Could you see if it helps? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#23 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATF2QQNCRLB4YG3FMWHOFJ3TDEU4LANCNFSM4ZAASFSA>.

[bernd-wechner]

It's great to see some folk looking at Windows compatibility as I admit I haven't ever, just assumed the os.path library would sort that ... but it does indeed look like os.path.abspath may be an interesting case. I'll take a look when I have time, but in the mean time wanted to add that I only added absolute_remote_path that line very recently for some reason, to generalise something else, it might easily do with a rework that works better across systems. Step 1 is to look at what os.path.abspath does.

https://docs.python.org/3/library/os.path.html#os.path.abspath

and yes it does look like it's ill suited for getting an remote abspath because the remote filesystem I've modelled with / as a root with devices as top level directories then directories under that (note that Degoo themselves don't have a "remote file system" they have a a hierarchical view on your data which consists of items which have an id and parent_id.

Devices lack a parent_id but have an id. And so represent the root of any given hierarchy, but not liking that I gave an imagined root the id of 0 that sits above devices. Which lends me a more natural view of my data across the two devices I have (web and phone).

Given that, and that os.path.abspath returns a drive letter one windows systems that line should really be smartened up.

It might be the normpath or realpath are better candidates:

https://docs.python.org/3/library/os.path.html#os.path.normpath
https://docs.python.org/3/library/os.path.html#os.path.realpath

Reading the abspath doc, it looks like it was a bungle on my part to use it at all as it is the equivalent of normpath(join(os.getcwd(), path)) and it's os.getcwd that is the culprit because it is a local CWD not the remote. I'm guessing it's as simples using: normpath(join(CWD, path)) instead! Give that a whir.

I will when I have time ...

[bernd-wechner]

@DLBerger, if you have success with any Windows support tweaks, please feel free to send a Pull Request through and I'll merge them in. For now, all I'm aware of is this one reported here and I'm fairly confident the fix I outlined above is fine.

[bernd-wechner]

Hopefully fixed in 4654acc.

Please test and report back and close if so. I don't have a Windows box handy to try it on.

[MikeBrownPhotographic]

normpath is good, but it still formats using the local op.sep for the join(), which generates paths like /X51\Test. These then fail in get_item, which fetches the UNIX-formatted list of folders from Degoo.
(At risk of worrying @DLBerger again) I can make this work by making to following changes:
In lib.py:
def absolute_remote_path(CWD, path):
'''
Convert a give path string to an absolute one (if it's relative).

:param path: The path to convert.
:returns: The absolute version of path
'''
degoo_sep = "/"  #Mike: Otherwise on a Windows machine, it looks for "\\" in a remote UNIX file path and fails to find it
if path and path[0] == degoo_sep:
    return os.path.normpath(path.rstrip(os.sep)).replace("\\","/")   #Mike: Otherwise it creates Windows paths for Degoo UNIX folders
else:
    return os.path.normpath(os.path.join(CWD["Path"], path.rstrip(os.sep))).replace("\\","/")   #Mike: Otherwise it creates Windows paths for Degoo UNIX folders

The python code remains portable for the system it is running on. The Degoo part must not be "portable", but rather static, as it is always going to be UNIX-style (as far as we know). Perhaps there is a more elegant solution, but this at least works.

With these changes, degoo_cd and degoo_pwd seem working consistently when I use UNIX separators to navigate Degoo. However, I had to manually edit my existing cwd.json file, as it had stored a Windows-formatted path for the CWD, and this was breaking the code. Once changed to UNIX separators, everything works.

Later today, I hope to try degoo_put and degoo_get...

[DLBerger]

Try my latest update to degoo/lib.py

[MikeBrownPhotographic]

In absolute_remote_path that should have been
...replace(os.sep,"/")
to be portable, of course :-)

[bernd-wechner]

Well spotted! I wold suggest using os.altsep though where you can avoiding string literals:

https://docs.python.org/2.7/library/os.html#os.altsep

Nothing is super clean mind you.

For Windows specific code, if any is needed you might wrap it in:

if os.name == 'nt':

[DLBerger]

I believe this fixes it all in a system-independent way:

import pathlib
...
def absolute_remote_path(CWD, path):
'''
Convert a given path string to an absolute one (if it's relative).

:param path: The path to convert.
:returns: The absolute version of path in Posix format
'''
# normpath() will remove clean up the path and the PurePosixPath() will strip trailing /, if present
return str(pathlib.PurePosixPath(os.path.normpath(os.path.join(CWD["Path"], path))).as_posix())

[MikeBrownPhotographic]

My apologies, guys. I think I've been pushing in the wrong direction. I had convinced myself that the remote system was UNIX, and we should stay consistent with that. But how the code exposes the folder structure is internally constructed, isn't it? I realised that I was just making life hard for myself when it comes to using these calls in a live script, as I would have to transform all the target folders in the script into Posix format first. So I've switched to investigating what is stopping the whole system using the local patterns, and the answer is "not a lot now".

1. Remove the new .replace() calls in absolute_remote_path
2. Fix a minor bug in API.py around line 426, where an os.sep is missing:
   was: properties["FilePath"] = f"{os.sep}{prefix}{properties['FilePath'].replace('/',os.sep)}"
   is: properties["FilePath"] = f"{os.sep}{prefix}{os.sep}{properties['FilePath'].replace('/',os.sep)}"
   This was losing the first separator in a path including child folders.
3. Edit cwd.json to use backslashes
4. And then degoo_cd works on Windows using backslashes, all except to the root folder, which still gets stored as a forward slash (I'm still trying to track that down)
5. get and put seem OK, too, using backslashes throughout.
   I'll try out the other functions as I get time. Sorry again for the confusion.

[MikeBrownPhotographic]

Back on the case here...
Current issue with degoo_put is an empty URL being returned after a successful upload. Looks like this may have happened before, given a commented out snippet in util.py around line 1060 or so. Any recollection of what that issue might be?

C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-master>py degoo_put -f test\test.txt
...
Google Response Content:
Nothing, Nil, Nada, Empty
...
props: {'ID': 14656990105, 'MetadataID': 8244476512, 'UserID': 20899872, 'DeviceID': 23712214, 'MetadataKey': 'ChTLPhEBHS46cSknx7mOuLjl9mDl-hAA', 'Name': 'test.txt', 'FilePath': '\X51\Test\test.txt', 'LocalPath': '', 'LastUploadTime': '1615893946377', 'LastModificationTime': '1615893946377', 'ParentID': 14623625937, 'Category': 6, 'Size': 10582, 'Platform': None, 'Distance': None, 'URL': '', 'OptimizedURL': None, 'ThumbnailURL': '', 'CreationTime': '2021-03-16T11:25:47+00:00', 'IsSelfLiked': False, 'Likes': 0, 'IsHidden': False, 'IsInRecycleBin': False, 'Description': '', 'Country': None, 'Province': None, 'Place': None, 'Location': None, 'GeoLocation': None, 'Data': '', 'DataBlock': '', 'CompressionParameters': '', 'IsShared': False, 'ShareTime': None, '__typename': 'ContentView', 'CategoryName': 'Document', 'Time_Created': '2021-03-16 11:25:47', 'Time_LastModified': '2021-03-16 11:25:46', 'Time_LastUpload': '2021-03-16 11:25:46'}

[bernd-wechner]

I see nothing around 1060, but here's where a comment starts:

Degoo/degoo/util.py

Line 993 in 8ba93bd

# # Empirically the download URL seems fairly predictable from the inputs we have.

Note you can share permalinks to the code like that for better clarity.

I'm not sure what's up, or expected here off hand but the URL is returned later at:

Degoo/degoo/util.py

Line 1039 in 8ba93bd

props = api.getOverlay3(degoo_id)

By Degoo. Can't recall if Google is providing one, you can see from the commented blocks I was experimenting a bit. if memory serves me Google don't provide a URL, because the URL is signed using the private key Degoo have for which same reason I can't construct such a URL.

[MikeBrownPhotographic]

Something weird was going on with test text file I was using. I switched to a jpeg file and ran degoo_put OK. It returned a URL, and I see the file listed in the Degoo web app. However, when I try to use that URL with degoo_get, it can't find it. And I can't download it using the Degoo web app either, so there was some sort of problem with the upload, I guess. Here is the URL as returned, and used in the get request:

https://c.degoo.biz/degoo-production-large-file-us-east1.degoo.me/QC7DNo/hZynCw/jpg/ChSWTS0kU4lB7YooOJ1PZVMFW2TJjxAA.jpg?GoogleAccessId=GOOG1ERGS5Y62VUMTEDIF6DORMJGWTJNXVR4GZLNW6KFP7E4PMCAYMA5BR6RA&Expires=1617190872&Signature=8rur7FdG54wkep6GDD3%2BBq6zZkQ%3D&use-cf-cache=true

Here is the error message:
Traceback (most recent call last):
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-master\degoo_get", line 286, in
sys.exit(main())
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-master\degoo_get", line 201, in main
degoo.get(args.remote, args.local, args.verbose, not args.force, args.dryrun, args.scheduled)
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-master\degoo\util.py", line 871, in get
return get_file(item['ID'], local_directory, verbose, if_missing, dry_run, schedule)
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\Degoo-master\degoo\util.py", line 764, in get_file
wget.download(URL, out=Name, size=Size, headers={'User-Agent': "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/83.0.4103.61 Chrome/83.0.4103.61 Safari/537.36"})
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\lib\site-packages\wget.py", line 317, in download
(tmpfile, headers) = opener.retrieve(url, tmpfile, callback)
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1820, in retrieve
fp = self.open(url, data)
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1786, in open
return getattr(self, name)(url)
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1995, in open_https
return self._open_generic_http(self._https_connection, url, data)
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1958, in _open_generic_http
return self.http_error(
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1980, in http_error
return self.http_error_default(url, fp, errcode, errmsg, headers)
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\lib\site-packages\wget.py", line 284, in http_error_default
raise Exception("{0}: {1}".format(errcode, errmsg))
Exception: 404: Not Found
degoo_get: 404: Not Found

When I paste the URL into a web browser, I get this:

NoSuchKey
The specified key does not exist.

No such object: degoo-production-large-file-us-east1.degoo.me/QC7DNo/hZynCw/jpg/ChSWTS0kU4lB7YooOJ1PZVMFW2TJjxAA.jpg

Any ideas what's failing here?

[MikeBrownPhotographic]

Hey guys
Well, I'm stuck. I can put a file, and it appears in the Degoo web UI, but there is a problem with what gets stored, and the download URL is invalid. I've run out of ideas, so I'm afraid I'm going to have to give up on making this script work on Windows for the time being.
Thanks for all your patience and help getting this far.

[bernd-wechner]

I'll give it a spring when I can find time. I can with some motivation even test it on Windows ;-). In mean time I wonder if @DLBerger has any issues uploading and downloading from Windows?

[bernd-wechner]

I just tried an upload and download of my test data set:

$ ./degoo_get test_data
Downloading /Web/Backup/test_data/Audio/Song 1.mp3 to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Audio/Song 1.mp3
 27% [###################                                                     ]            2M / 10M
Downloading /Web/Backup/test_data/Audio/Song 2.m4a to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Audio/Song 2.m4a
100% [########################################################################]           17M / 17M
Downloading /Web/Backup/test_data/Audio/Song 3.wma to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Audio/Song 3.wma
100% [########################################################################]           11M / 11M
Downloading /Web/Backup/test_data/Data/Data 1.scad to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Data/Data 1.scad
100% [##############################################################################]     14K / 14K
Downloading /Web/Backup/test_data/Data/Data 2.obj to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Data/Data 2.obj
100% [########################################################################]           36M / 36M
Downloading /Web/Backup/test_data/Data/Data 3.stl to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Data/Data 3.stl
100% [########################################################################]           14M / 14M
Downloading /Web/Backup/test_data/Data/Data 4.m3lib to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Data/Data 4.m3lib
100% [######################################################################]           172M / 172M
Downloading /Web/Backup/test_data/Document/Document 1 to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 1
100% [################################################################################]     5K / 5K
Downloading /Web/Backup/test_data/Document/Document 10.note to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 10.note
100% [##################################################################################] 699B / 699B
Downloading /Web/Backup/test_data/Document/Document 11.xml to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 11.xml
100% [################################################################################]     2K / 2K
Downloading /Web/Backup/test_data/Document/Document 12.xcf to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 12.xcf
100% [############################################################################]     290K / 290K
Downloading /Web/Backup/test_data/Document/Document 13.kml to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 13.kml
100% [################################################################################]     6K / 6K
Downloading /Web/Backup/test_data/Document/Document 14.odt to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 14.odt
100% [##############################################################################]     20K / 20K
Downloading /Web/Backup/test_data/Document/Document 15.xls to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 15.xls
100% [##############################################################################]     39K / 39K
Downloading /Web/Backup/test_data/Document/Document 16.dwg to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 16.dwg
100% [############################################################################]     384K / 384K
Downloading /Web/Backup/test_data/Document/Document 17.dxf to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 17.dxf
100% [##########################################################################]           2M / 2M
Downloading /Web/Backup/test_data/Document/Document 18.log to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 18.log
100% [########################################################################]           55M / 55M
Downloading /Web/Backup/test_data/Document/Document 19.py to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 19.py
100% [################################################################################]     1K / 1K
Downloading /Web/Backup/test_data/Document/Document 2.pdf to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 2.pdf
100% [##########################################################################]           1M / 1M
Downloading /Web/Backup/test_data/Document/Document 20 to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 20
0 / unknown
Downloading /Web/Backup/test_data/Document/Document 3.docx to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 3.docx
100% [##############################################################################]     17K / 17K
Downloading /Web/Backup/test_data/Document/Document 4.pages to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 4.pages
100% [##########################################################################]           1M / 1M
Downloading /Web/Backup/test_data/Document/Document 5.txt to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 5.txt
100% [##################################################################################] 104B / 104B
Downloading /Web/Backup/test_data/Document/Document 6.ods to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 6.ods
100% [##############################################################################]     33K / 33K
Downloading /Web/Backup/test_data/Document/Document 7.htm to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 7.htm
100% [##############################################################################]     13K / 13K
Downloading /Web/Backup/test_data/Document/Document 8.html to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 8.html
100% [##############################################################################]     13K / 13K
Downloading /Web/Backup/test_data/Document/Document 9.reg to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Document/Document 9.reg
100% [##################################################################################] 384B / 384B
Downloading /Web/Backup/test_data/Image/Image 1.png to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Image/Image 1.png
100% [##############################################################################]     79K / 79K
Downloading /Web/Backup/test_data/Image/Image 2.jpg to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Image/Image 2.jpg
100% [##############################################################################]     18K / 18K
Downloading /Web/Backup/test_data/Image/Image 3.jpg to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Image/Image 3.jpg
100% [################################################################################]     7K / 7K
Downloading /Web/Backup/test_data/Image/Image 4.svg to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Image/Image 4.svg
100% [############################################################################]     810K / 810K
Downloading /Web/Backup/test_data/Image/Image 5.dwg to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Image/Image 5.dwg
100% [############################################################################]     384K / 384K
Downloading /Web/Backup/test_data/Image/Image 6.jpeg to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Image/Image 6.jpeg
100% [##############################################################################]     47K / 47K
Downloading /Web/Backup/test_data/Image/Image 7.bmp to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Image/Image 7.bmp
100% [##########################################################################]           1M / 1M
Downloading /Web/Backup/test_data/Video/Video 1.mp4 to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Video/Video 1.mp4
100% [########################################################################]           82M / 82M
Downloading /Web/Backup/test_data/Video/Video 2.mkv to /media/Data/Workspace/Eclipse/Degoo/tmp/test_data/Video/Video 2.mkv
Traceback (most recent call last):
  File "../degoo_get", line 286, in <module>
  File "../degoo_get", line 201, in main
  File "/media/Data/Workspace/Eclipse/Degoo/degoo/util.py", line 852, in get
    return get_directory(item['ID'], local_directory, verbose, if_missing, dry_run, schedule)
  File "/media/Data/Workspace/Eclipse/Degoo/degoo/util.py", line 832, in get_directory
    get_directory(f['ID'], local_directory, verbose, if_missing, dry_run, schedule)
  File "/media/Data/Workspace/Eclipse/Degoo/degoo/util.py", line 823, in get_directory
    get_file(f['ID'], local_directory, verbose, if_missing, dry_run, schedule)
  File "/media/Data/Workspace/Eclipse/Degoo/degoo/util.py", line 747, in get_file
    wget.download(URL, out=Name, size=Size, headers={'User-Agent': "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/83.0.4103.61 Chrome/83.0.4103.61 Safari/537.36"})
  File "/home/bernd/.local/lib/python3.8/site-packages/wget.py", line 317, in download
    (tmpfile, headers) = opener.retrieve(url, tmpfile, callback)
  File "/usr/lib/python3.8/urllib/request.py", line 1824, in retrieve
    fp = self.open(url, data)
  File "/usr/lib/python3.8/urllib/request.py", line 1790, in open
    return getattr(self, name)(url)
  File "/usr/lib/python3.8/urllib/request.py", line 1999, in open_https
    return self._open_generic_http(self._https_connection, url, data)
  File "/usr/lib/python3.8/urllib/request.py", line 1962, in _open_generic_http
    return self.http_error(
  File "/usr/lib/python3.8/urllib/request.py", line 1984, in http_error
    return self.http_error_default(url, fp, errcode, errmsg, headers)
  File "/home/bernd/.local/lib/python3.8/site-packages/wget.py", line 284, in http_error_default
    raise Exception("{0}: {1}".format(errcode, errmsg))
Exception: 302: Moved Temporarily
degoo_get: 302: Moved Temporarily
           for help use --help

Of note were:

a) a huge delay at start and the first one having a progress bar stall at 27%
b) bombing at one file with a 302 error

Need to check that one out of course and handle it. A 302 is a redirect and could be totally legit and needs to honoured. That is, the script is fetching some URL that is feeding back that we should fetch it from some other URL.

So that is on the TO FIX list before retesting.

EItehr way htis is not the 404 error you're seeing which is File does not exist error and is weird. UNless I can reproduce it (which I haven't I can't diagnose but to diagnose the steps are not tooo hard:

1. Download same file using their web app.
2. If it doesn't work, bad news, that's a Degoo being weird all round and disconcerting ...
3. If it does, then press F12 in your browser, click the network tab, clcik the clear button (usuall reound cicle with slash)
4. Download the file again.
5. Save everything that appears on the network tab (I save it all in HAR format typically).
6. Study that or send it to me ;-).
7. To study it I make a copy (so as to keep an oringal reference), load it in Atom (my favourite text editor), flag it as JSON data, and beautify it then step through each request deleting the noise (requests that are irrelnvant to us - there are usually a lot of them) and narrowing it down to the one, two or three requests of importance. Then I sdudy those.

[MikeBrownPhotographic]

Thanks for trying this. I could not download the file from Degoo web app at all. The web ui showed an error: "Unable to download file. Please try again". I guess that could be the equivalent of saying file not found, or invalid stored URL.

[MikeBrownPhotographic]

As I can't complete a request for a file I have uploaded with the script, here is a HAR file for a download of a file previously uploaded via the web app. The file is called 00fallingpetals.gif.
I'm hoping that you might be able to spot something here which might be at variance with what you would expect from a Mac.
app.degoo.com.har.zip

[MikeBrownPhotographic]

I can see that from the script, a typical (failing) generated url was
https://c.degoo.biz/degoo-production-large-file-us-east1.degoo.me/etc
but here it is
https://c.degoo.media/degoo-production-medium-file-us-east1.degoo.me/etc
Not sure if this is significant, or in any way controlled by the Python code?

[bernd-wechner]

Yep, that suggests a Degoo problem.

But it may have been caused by an upload from this script (not to spec). And so that is a concern. But unless we can reproduce it, very hard to fix. To wit, one strategy is to ignore this file (try deleting it on the cloud drive) and see if you can reliably reproduce it, a successful upload and failed download.

The put is a four step process:

        # The steps involved in an upload are 4 and as follows:
        #
        # 1. Call getBucketWriteAuth4 to get the URL and parameters we need for upload
        # 2. Post the actual file to the BaseURL provided by that
        # 3. Call setUploadFile3 to inform Degoo it worked and create the Degoo item that maps to it
        # 4. Call getOverlay3 to fetch the Degoo item this created so we can see that it worked (and return the download URL)

One possible break is if step 2 fails and step 3 succeeds. That is we don't upload the file, but do tell Degoo it was uploaded. It's possible to image the remote filesystem thinking the file exists, but it doesn't. I see no clear way for such a break to happen as Step 2 returns with a response code (from Google) and we proceed to Step 3 only after:

        if response.ok and response.status_code == 204:

that is, an OK response with code 204 from Google.

If something goes wrong and we don't get that, we don't do step 3. So it seems secure.

Another point of possible weakness is their API changing. You will see we call setUploadFile3, but if they have moved to setUploadFile4 but haven't disabled setUploadFile3 we might be creating a dodgy record on Degoo.

If any of this is a problem then a HAR of the upload session is more useful, to check that they still do what they did, or if they have upgraded something.

I think it might be worth adding a debug command line option (-D, --debug) that sees put and get at least dumping a trace of all transactions to a log file for analysis and submission.

Finally re:

https://c.degoo.biz/degoo-production-large-file-us-east1.degoo.me/etc
vs.
https://c.degoo.media/degoo-production-medium-file-us-east1.degoo.me/etc

You may well have hit the jackpot even. As you can see a shift from degoo.biz to degoo.media.

It depends on what you mean by generated URL though, generated by whom where? Where are you seeing that? The only URL I know is the one returned by getOverlay3 here:

Degoo/degoo/util.py

Line 1042 in 8ba93bd

URL = props['URL']

But I do note that it returns two URLs:

Degoo/degoo/util.py

Line 697 in 8ba93bd

URL = item.get("OptimizedURL", None)

And it would interesting to see them both. You can tweak there which one to use.

These URLS are all generated by Degoo and the returned by getOverlay3 but again they may have moved to getOverlay4 and that's the sort of thing to check.

Took a quick look at your HAR, but it only seems to call getFileChildren3 which cna return file URLs too.

[MikeBrownPhotographic]

After degoo_put, I can view the file in the web app, so it did actually upload. I just can't download it again, either by degoo_get or from the web app.
OptimizedURL is always "None", so there is no real choice available there. Should it have a value normally? Perhaps that is a clue?

Also, trying degoo_put -f today, I get this url:
c.degoo.media/degoo-production-large-file-us-east1.degoo.me/etc
But it still fails the same way. So it is likely the media vs biz thing was not the answer.

BTW When I try degoo_put without -f, it's failing from the util.py call:
LastUploadTime = datetime.utcfromtimestamp(0).replace(tzinfo=tzutc()).astimezone(tzlocal())
...
File "C:\Users\mike\AppData\Local\Programs\Python\Python39\lib\site-packages\dateutil\tz\tz.py", line 260, in _naive_is_dst
return time.localtime(timestamp + time.timezone).tm_isdst
OSError: [Errno 22] Invalid argument
This is failing in the Python library code, which is odd. Makes me suspicious of my install. I'm on version 3.9.2, which is supposed to be current for Windows.

[MikeBrownPhotographic]

Here is a HAR file for a web app upload of the same file
webappupload.degoo.com.har.zip

[bernd-wechner]

That upload trace is not a complete upload. The getBucketWriteAuth4 call returns:

        "text": "{\"data\":{\"getBucketWriteAuth4\":[{\"AuthData\":null,\"Error\":\"Already exist!\",\"__typename\":\"BucketWriteAuthInfo\"}]}}"

and not a BaseURL for upload.

The python error is odd, and looks like time corruption. I'd need to break that the line:

return time.localtime(timestamp + time.timezone).tm_isdst

and inspect the arguments timestamp and time.timezone

I just uploaded a video file which reports:

$ ./degoo_put Video\ 2.mkv 
Uploading Video 2.mkv to /Web/Backup
100% [######################################################################]           223M / 223M
Uploaded Video 2.mkv to /Web/Backup/Video 2.mkv with Degoo ID: 14566475583 and Download URL
https://c.degoo.info/degoo-production-large-file-us-east1.degoo.me/gCkuIp/tISlDA/mkv/ChSG0Vj9eZfJAKahAgTsskwgpqQdmhAA.mkv?GoogleAccessId=GOOG1ERGS5Y62VUMTEDIF6DORMJGWTJNXVR4GZLNW6KFP7E4PMCAYMA5BR6RA&Expires=1617627148&Signature=38LZHOGRmC1CjuMgPNsODE37xLU%3D

(that BTW is the kind of info to share as the reader can see what you did, what happened and what you're talking about)

I can see I'm getting c.degoo.info as a base domain for the download. If I surf to that link I get

302 Found
cloudflare

in my browser. Normally I get the file downloading, and cloudlfare is new to me, but an Amazon system that clearly Degoo now use for something. And yet degoo_get is downloading it, at a snails pace ... no complaint, just sluggish as all heck).

That is odd as my browser still says 302 which is very odd. So I did:

$ ./degoo_props Video\ 2.mkv 
Properties of Video 2.mkv:
	ID: 14566475583
	MetadataID: 6643390221
	UserID: None
	DeviceID: 25772596
	MetadataKey: ChSG0Vj9eZfJAKahAgTsskwgpqQdmhAA
	Name: Video 2.mkv
	FilePath: /Web/Backup/Video 2.mkv
	LocalPath: 
	URL: https://c.degoo.eu/degoo-production-large-file-us-east1.degoo.me/gCkuIp/tISlDA/mkv/ChSG0Vj9eZfJAKahAgTsskwgpqQdmhAA.mkv?GoogleAccessId=GOOG1ERGS5Y62VUMTEDIF6DORMJGWTJNXVR4GZLNW6KFP7E4PMCAYMA5BR6RA&Expires=1617628242&Signature=bhYuTkXgdufGYw3sjWAcWuR%2BRCg%3D
	OptimizedURL: None
	ThumbnailURL: https://lh3.googleusercontent.com/W_QKjTjXWf_joI5lBgwTOtR0wBq4esOvtWJuLixLMxQOTIVlD1PxLihc3oEEiVgEsZLym5fhwnI510ZDnyOP-TQ
	CreationTime: 2021-03-22T12:52:25+00:00
	LastModificationTime: 1616417545611
	LastUploadTime: 1616417545611
	ParentID: 13497575511
	Category: 4
	Size: 234488567
	Platform: None
	Distance: None
	IsSelfLiked: None
	Likes: None
	IsHidden: None
	IsInRecycleBin: False
	Description: None
	Country: None
	Province: None
	Place: None
	Location: None
	Location2: None
	GeoLocation: None
	Data: 
	DataBlock: 
	CompressionParameters: 
	IsShared: False
	ShareTime: None
	Shareinfo: None
	__typename: ContentView
	CategoryName: Video
	Time_Created: 2021-03-22 12:52:25
	Time_LastModified: 2021-03-22 12:52:25
	Time_LastUpload: 2021-03-22 12:52:25

And you can see it is now at c.degoo.eu and if I surf to that URL in my browser I get the file downloading. And MUCH faster than degoo_get!

Like wow. Tentative conclusions:

1. The Useragent I'm using for the CLI is being penalized on bandwidth. Though I'm using:
   headers={'User-Agent': "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/83.0.4103.61 Chrome/83.0.4103.61 Safari/537.36"})
   Still, TODO try using my browsers and see if it compares.

2. the URL changes over time, and quite possibly as it changes there's 302 redirect left behind for a little while. I have to catch one of those in code to see how to handle it properly.

Anyhow, got way too late here, so I'll sign off for the day. But there's clearly some testing to do here on download performance and why my browser downloaded a 223MB video near instantly and I have two degoo_gets running for a looooong time and crawling alog at a snaiiiiils pace.

Something to do with our request, and the wget and requests libs. I know the default user agent they used is blocked by Degoo from which I infer they take lengths to prevent scripts from crawling their site etc. Maybe they or Google found another way to or reason to hamstring the get from the script. Easy to find out by sniffing the respective GET requests. But another time.

[bernd-wechner]

Seems a pile of transient issues and today:

$ ./degoo_get -f Video\ 2.mkv 
Downloading /Web/Backup/Video 2.mkv to /media/Data/Workspace/Eclipse/Degoo/tmp/Video 2.mkv
100% [######################################################################]           223M / 223M

came down near instantly.

but I notice the URL changed again:

$ ./degoo_props Video\ 2.mkv 
Properties of Video 2.mkv:
	ID: 14566475583
	MetadataID: 6643390221
	UserID: None
	DeviceID: 25772596
	MetadataKey: ChSG0Vj9eZfJAKahAgTsskwgpqQdmhAA
	Name: Video 2.mkv
	FilePath: /Web/Backup/Video 2.mkv
	LocalPath: 
	URL: https://c.degoo.biz/degoo-production-large-file-us-east1.degoo.me/gCkuIp/tISlDA/mkv/ChSG0Vj9eZfJAKahAgTsskwgpqQdmhAA.mkv?GoogleAccessId=GOOG1ERGS5Y62VUMTEDIF6DORMJGWTJNXVR4GZLNW6KFP7E4PMCAYMA5BR6RA&Expires=1617715343&Signature=AfDtB0p%2B8GhFAmzc7eVKp9qMGH0%3D
	OptimizedURL: None
	ThumbnailURL: https://lh3.googleusercontent.com/W_QKjTjXWf_joI5lBgwTOtR0wBq4esOvtWJuLixLMxQOTIVlD1PxLihc3oEEiVgEsZLym5fhwnI510ZDnyOP-TQ
	CreationTime: 2021-03-22T12:52:25+00:00
	LastModificationTime: 1616417545611
	LastUploadTime: 1616417545611
	ParentID: 13497575511
	Category: 4
	Size: 234488567
	Platform: None
	Distance: None
	IsSelfLiked: None
	Likes: None
	IsHidden: None
	IsInRecycleBin: False
	Description: None
	Country: None
	Province: None
	Place: None
	Location: None
	Location2: None
	GeoLocation: None
	Data: 
	DataBlock: 
	CompressionParameters: 
	IsShared: False
	ShareTime: None
	Shareinfo: None
	__typename: ContentView
	CategoryName: Video
	Time_Created: 2021-03-22 12:52:25
	Time_LastModified: 2021-03-22 12:52:25
	Time_LastUpload: 2021-03-22 12:52:25

from degoo.eu yesterday to degoo.biz today.

Moreover if memory serves me these used to be google addresses, and now look to be Amazon addresses. There's a chance it seems Degoo have moved their back end from Google cloud services to Amazon.

[bernd-wechner]

Tried this again today, and all downloads seem to stop at 127MB. Looks fishy to me. Not sure what's up.

[bernd-wechner]

OK, can confirm upload a video, via web app, download via web app and it too truncates at 128MB. A Degoo bug. I filed a ticket. Fingers crossed.

[bernd-wechner]

That bug is now fixed at the Degoo end. Downloads appear to be fully functional here currently. I have also done a complete round trip test again and it seems fine. I suspect this can close, and you can reopen it if there are any further os.sep issues