bernd-wechner / degoo Goto Github PK

I found this the hard way.
Degoo sees MyFile and MYFILE as the same file/directory.
Like Windows it retains the filename as created, but trying to create any other variants causes an error.
Not sure why, but it leads to a host of potential problems.

Would anyone have direct contact with someone at Degoo; at least, at the business level.

I ask (party in ignorance): wouldn't they benefit from allowing authenticated access to one's storage via the Google API, where various tools could be used at that level? For example, I use "rclone" for interacting with Google Drive and S3; it's fast, efficient.

Pardon if this is the wrong place to ask this, I didn't see a mailing list.

Ubuntu Server 20.04 LTS
Running pip install -r requirements.txt
Receive error: ERROR: launchpadlib 1.10.13 requires testresources, which is not installed.

Fix: pip install testresources before installing the requirements, but maybe it needs to be added to them?

I think you added this to the repository by accident.

I tried installing everything but I get this error

Login failed with: 403: Forbidden
Login failed.

Had to add the following to requirements.txt before build.py would run:

clint==0.5.1
requests-toolbelt==0.9.1

When running the code on Windows, the os is set to "NT", which is correct for local disk access. But Degoo is running remotely on a Linux environment. This results in error messages for calls to degoo_cd etc.

A kludge workaround is to override os.sep in util.py, lib.py and API.py as soon as it is imported:
if os.sep == "\":
os.sep = "/"

This at least gives me back degoo_cd, but there are likely other undesirable side effects.

I have still had to comment out part of lib.absolute_remote_path, as this is still failing:
if path and path[0] == os.sep:
# Mike: This call is still a problem
#return os.path.abspath(path.rstrip(os.sep))
return path
For example, if cwd on Degoo was /X51/LightroomOther/2005_extra and I executed "py degoo_cd /X51", I got the following error:
degoo.util.DegooError: C:\ does not exist in /X51LightroomOther/2005_extra
This implies abspath is picking up the default root directory for Windows, rather than UNIX, and also the rstrip was removing a / unnecessarily.

Also relative cd is not working at all if it includes ".." in the path., e.g. degoo_cd .. , or degoo_cd ../Backups. Is it likely this is because I have commented out the call to os.path.abspath, or because it is unsupported?

Any help in making this work in a robust way is appreciated.
Mike

Hello, I am using the Ultimate plan of Degoo. I overcame the issue of 429, by modifying the login part using cloudscrapper, but now I face another issue whenever I try to list items in a subfolder. I get the following stacktrace:

Traceback (most recent call last):
File "./degoo_ll", line 328, in
sys.exit(main())
File "./degoo_ll", line 116, in main
degoo.ls(args.folder, args.long, args.human, args.recursive)
File "/mnt/devel/Degoo/degoo/util.py", line 1210, in ls
items = get_children(directory)
File "/mnt/devel/Degoo/degoo/util.py", line 604, in get_children
CACHE_CONTENTS[dir_id] = api.getFileChildren3(dir_id)
File "/mnt/devel/Degoo/degoo/API.py", line 625, in getFileChildren3
prefix = self.devices[i['DeviceID']] + os.sep + "Recycle Bin" if binned else self.devices[i['DeviceID']]
KeyError: 27437396
degoo_ll: 27437396

I have no problems issuing the same command on the root folder, but all other folders return the same error.

What can I do to troubleshoot the issue?
Thanks!

Francisco

I am currently setting degoo on my RPI and have the following issue:

zeus@rpi-3b ~/src/Degoo (master*) $ python build.py
Traceback (most recent call last):
File "build.py", line 20, in
import degoo
File "/home/zeus/src/Degoo/degoo/init.py", line 41, in
from .util import *
File "/home/zeus/src/Degoo/degoo/util.py", line 136
print(f"Waiting until {until.strftime('%A, %d/%m/%Y %H:%M:%S')}")

How should I proceed?

Could you add any guide for Linux beginners please ?
Maybe video tutorial for installation and use ?

This seems to be a bug at the Degoo end. Am curious if it is global, if others are experiencing same (or account specific or such). I have filed a support ticket with Degoo.

The issue is easily reproduced:

1. Upload a large file (I use video files)
2. Download it again. Only the first 128MB are downloaded.

This turns out to be true using degoo_put and degoo_get from this repo, but also using the Degoo web app. Hence the conclusion it is at the Degoo end. it is the wget library:

https://github.com/bernd-wechner/python3-wget

That is simply returning, with no error, before the file is complete, consistently at about 128MB irrespective of the file size (as long as it's over 128MB ;-).

I am using Cryptomator to encrypt my files before uploading to Degoo service.
Upload using degoo_put works (after making a small change in code):

index 8043121..c27976a 100644
--- a/degoo/util.py
+++ b/degoo/util.py
@@ -1162,7 +1162,7 @@ def put_directory(local_directory, remote_folder, verbose=0, if_changed=False, d
             relative_root = root
 
         for name in dirs:
-            Name = os.path.join(root, name)
+            Name = os.path.join(relative_root, name)
 
             IDs[Name] = mkdir(name, IDs[relative_root], verbose - 1, dry_run)

python degoo_put /mnt/kuvat /Web

After successfull upload I try to download the same directory to another folder:

python degoo_get /Web/kuvat /mnt/kuvat2

Some of the files are downloaded correctly while some of them output error:
<file> apparently has no URL to download from.

Has anyone faced such problems in the past? Any help how to debug the issue is appreciated.
Zip package of files I have used (to try to reproduce the problem) can be downloaded from here

Currently I've built this so it have a load of distinct command line utilities. That's nice in a way. They include:

degoo_cd
degoo_get
degoo_ll
degoo_login
degoo_ls
degoo_mkdir
degoo_path
degoo_props
degoo_put
degoo_pwd
degoo_rm
degoo_test
degoo_tree
degoo_user

It would be nice and others might also appreciate if all these had a way of being invoked through one CLI command for example:

degoo cd
degoo get
degoo ll
degoo login
degoo ls
degoo mkdir
degoo path
degoo props
degoo put
degoo pwd
degoo rm
degoo test
degoo tree
degoo user

Should be easy to support both methods. Just writing this note to remind myself to do it some time.

It's awesome that we now have progress bars on download and upload but they have distinct styles and it would be nice to bring them into conformance. The differences are presently as follows:

1. Style:

   Get:
   100% [########################################################################] 82M / 82M

   Put:
   [##########          ] 80044032/234490033 - 00:00:58

   Get is nicer. But Put has an ETA which is awesome.

   Both are lacking the filename.

   They apply file by file when doing directory puts and gets but don't show the current filename. This should be fixed.

2. History:

   Get shows the history in that every new file starts a new line and you see a load of 100% progress bars above the current one running. Put overwrites each progress bar and is only ever displaying the currently running one so no history is kept on screen.

   The two use different packages to achieve their result, and eitehr one may be more or less configurable.

   The get progress bar is delivered by the wget package:

   https://github.com/bernd-wechner/python3-wget/blob/e38979ffdf462a5cec05c5a92ff35a7630c6435b/wget.py#L286

   and the put progress bar is delivered by the clint package:

   https://github.com/kennethreitz-archive/clint/blob/9d3693d644b8587d985972b6075d970096f6439e/clint/textui/progress.py#L33

I suspect the best approach is to:

1. Generalise the wget progress bar extracting it from the wget library and including it as a degoo cli function: https://github.com/bernd-wechner/python3-wget/blob/e38979ffdf462a5cec05c5a92ff35a7630c6435b/wget.py#L166
2. Extend it to accept a filename
3. plug this bar in to the wget call: wget.download(bar= ...) and the MultipartEncoderMonitor(callback=...)

We'd then have consistent bars and behaviour and the best look (I am thinking the wget adaptive bar plus an ETA added and a filename).

I seem to get for following on many files:

Traceback (most recent call last):
File "/home/dberger/degoo.DLBerger/./degoo_put", line 286, in
sys.exit(main())
File "/home/dberger/degoo.DLBerger/./degoo_put", line 211, in main
result = degoo.put(args.local, args.remote, args.verbose, not args.force, args.dryrun, args.scheduled)
File "/home/dberger/degoo.DLBerger/degoo/util.py", line 1156, in put
return put_directory(local_path, remote_folder, verbose, if_changed, dry_run, schedule)
File "/home/dberger/degoo.DLBerger/degoo/util.py", line 1134, in put_directory
put_file(Name, IDs[relative_root], verbose, if_changed, dry_run, schedule)
File "/home/dberger/degoo.DLBerger/degoo/util.py", line 1035, in put_file
degoo_id = api.setUploadFile3(filename, dir_id, verbose, Size, Checksum)
File "/home/dberger/degoo.DLBerger/degoo/API.py", line 795, in setUploadFile3
raise self.Error(f"setUploadFile3 failed with: {response}")
degoo.API.API.Error: setUploadFile3 failed with: <Response [504]>
degoo_put: setUploadFile3 failed with: <Response [504]>
for help use --help

Any idea what it means?

Could you give an description on how to do this after installing the req's? Thanks a lot

from appdirs import user_config_dir
ModuleNotFoundError: No module named 'appdirs'

I have confirmed appdirs was installed so not sure what's going on or if these tool even works anymore ?

When I try to download a file I get the error download() got an unexpected keyword argument 'size', both in Windows and Ubuntu. I have seen that you need to install https://github.com/bernd-wechner/python3-wget

Honestly, I haven't tried to install this package, but I think that to avoid this error for new users you could download it as standard with requests:

      if not dry_run:
          # _ = wget.download(URL, out=Name, size=Size, headers={'User-Agent': "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/83.0.4103.61 Chrome/83.0.4103.61 Safari/537.36"})
          silent = False # It could be a new argument
          with requests.get(URL, stream=True) as r:
              r.raise_for_status()
              with open(Name, 'wb') as f:
                  if silent:
                      shutil.copyfileobj(r.raw, f)
                  else:
                      dl = 0
                      total_length = int(r.headers.get('content-length'))
                      if not total_length:
                          total_length = Size
                      for data in r.iter_content(chunk_size=4096):
                          dl += len(data)
                          f.write(data)
                          done = int(50 * dl / total_length)
                          sys.stdout.write("\r[%s%s]" % ('=' * done, ' ' * (50-done)))
                          sys.stdout.flush()

I hope it will be helpful.

I cloned & pip install -r requirements & python or python3 build.py but
raise ImportError('failed to find libmagic. Check your installation')
ImportError: failed to find libmagic. Check your installation
This happens everytime. I dont have pc & I want to download all of my degoo data plz help

Hello, this project is a few months old and has most likely been abandoned. However, I'm not actually able to find clear instructions or documentation. I'd love it if the devs could help me out.
How exactly do I use it? So far I've gathered that I:

1. Enter the directory with cd ~/Downloads/degoo/
2. Add the packages with pip install -r requirements.txt
3. Build it with python3.9 build.py

But what do I do that? Now that I've built the commands, I need to know what the commands are! For example, if I wanted to upload a file located in the path ~/image.jpg to my degoo account, how would I do this?
Where can I get any docs related to this?

Got this generic issue: most of the commands give the same error 'int() argument must be a string, a bytes-like object or a number, not 'NoneType''

• ./degoo_login, ./degoo-pwd and ./degoo_user give the expected output (Successfuly logged in. [typo here though > Succesfully], Working Directory is / & user info respectively)
• ./degoo_mkdir test outputs to degoo_mkdir: mkdir: No parent_id provided.
• Most of the others output to int() argument must be a string, a bytes-like object or a number, not 'NoneType'.

Some misconfiguration (by me)?

Hi there, I really appreciate such project exists. I really regret using Degoo and it is getting more and more ridiculous every day.

I've encountered an error where all the command doesn't work except degoo_login, degoo_pwd and degoo_user. I'm using Python 3.9.1 on Ubuntu 20.10 and Debian 10 just for reference. Whenever a command is executed it always returns with an error getFileChildren3 failed with: <Response [400]>. All python module requirements are installed. Adding the verbose flag doesn't return any extra debug info except for the error above. Below is an image that I have attached.

When I try to upload large files, a MemoryError exception occurs when executing request.post. I have tried with a 1Gb file. I managed to fix it by changing this:

# Perform the upload
# TODO: Can we get a progress bar on the this? Web app has one.
response = requests.post(BaseURL, files=parts, headers=heads)

By this:

from requests_toolbelt import MultipartEncoder

multipart = MultipartEncoder(fields=dict(parts))
heads['content-type'] = multipart.content_type
response = requests.post(BaseURL, data=multipart, headers=heads)

Greetings.

For large files in particular there's a real risk that a transfer is interrupted. For that reason most web tech today and the Google Cloud Store in particular support restartable downloads and uploads.

This CLI would need to do something in prep to exploit that, which needs a little research and testing of ideas. It may fall flat if it requires the Google Cloud Store private key to work, as that is in Degoo's hands, we can't interact freely with the cloud store only via the Degoo GraphQL interface and that's only doable if their web interface implements a restartable download/upload that can be watched. I'm not sure it does.

Hi there,
I cannot login.

I get the error 400

./degoo_login                            
username: [email protected]                                             
Password:                                                               
Login failed with: 400:                                                 
Login failed.                                                           

Running it with -vvv doesn't produce any additional log.

Can you support?

This error occurs when trying to upload a file with a time zone other than UTC (in my case +1). The problem is in the has_changed method, specifically when trying to calculate LastUploadTime:

LastUploadTime = datetime.datetime.utcfromtimestamp(0).replace(tzinfo=tz.UTC).astimezone(tz.tzlocal()) # Internally it tries to subtract 0 - 3600, which causes the exception

I think this can be fixed by changing the following, working on Windows as on Ubuntu:
LastUploadTime = datetime.datetime.now(tz.tzlocal())

I have created a new branch in which I have created the commands rename and mv. I hope this will help to complete all the available commands.

Thanks! This looks great, as I theoretically 13TB lifetime with them, which I can't use because they changed/abandoned their client software. (Don't get, why they don't at least expose a WebDAV interface to their storage.)

That said, 13TB, even with CLI tools aren't usable for much without lots of wasted time uploading, checking, etc.

Ideal would be a FUSE interface ( https://wiki.osdev.org/FUSE ), so the storage can be mounted as a network drive, and then used with random backup, file-sync, etc. software, without degoo specific code. Is that a direction you plan to take your code in?

The reason I ask: you seem to have already implemented the type of operations that sftp offers, and there's a FUSE abstraction layer based on those ( https://github.com/libfuse/sshfs )

I'm having problems logging in. Has Degoo changed their login protocols?

On login attempt, I get no response (error, confirmation, or other). I see no usage guideance for further troubleshooting. Where do I enter in my username and password?

# ls
LICENSE.md  build.py     default_properties.txt  degoo_cd   degoo_ll     degoo_ls     degoo_path   degoo_put  degoo_rm    degoo_tree 
README.md   commands.py  degoo                   degoo_get  degoo_login  degoo_mkdir  degoo_props  degoo_pwd  degoo_test  degoo_user
# ./commands.py help
# ./commands.py login
# ./commands.py user
# ./commands.py ls

I'm getting this every time now it was working fine a few weeks ago. I updated to the newest version and still get this.

Traceback (most recent call last):
  File "/tmp/degoo2/./degoo_ls", line 328, in <module>
    sys.exit(main())
  File "/tmp/degoo2/./degoo_ls", line 116, in main
    degoo.ls(args.folder, args.long, args.human, args.recursive)
  File "/tmp/degoo2/degoo/util.py", line 1210, in ls
    items = get_children(directory)
  File "/tmp/degoo2/degoo/util.py", line 604, in get_children
    __CACHE_CONTENTS__[dir_id] = api.getFileChildren3Ex(dir_id)
  File "/tmp/degoo2/degoo/API.py", line 659, in getFileChildren3Ex
    (next_items, next_token) = self.getFileChildren3(dir_id, next_token)
  File "/tmp/degoo2/degoo/API.py", line 650, in getFileChildren3
    raise self.Error(f"getFileChildren3 failed with: {response}")
degoo.API.API.Error: getFileChildren3 failed with: <Response [400]>
degoo_ls: getFileChildren3 failed with: <Response [400]>
          for help use --help

Thank you for contributing this. I got sucked in to Degoo, only to realize the frustration that they offer. This is something we all need.

Consider listing some dependencies that this requires?

From python3.8 (MacPorts, MacOS Big Sur):

Traceback (most recent call last):
File "build.py", line 20, in
import os, stat, degoo
File "/usr/local/src/degoo/degoo/init.py", line 23, in
import os, sys, csv, json, time, datetime, requests, wget, magic, humanize, humanfriendly, hashlib, base64
ModuleNotFoundError: No module named 'requests'

Degoo metadata does not include a remote hash. We can calculate one locally of course. But there is a good chance we can coax one out of the backed Google File Store that Degoo are using:

Here's a place to start reading:

https://stackoverflow.com/questions/25234515/how-to-get-the-md5-of-an-object-that-has-already-been-stored

I have tried every possible command other than this.
Please anyone give a hint. or just an example.

degoo_put

root@093ff746f382:/Degoo# python3 degoo_put /Degoo/testuploadfile /
Traceback (most recent call last):
  File "degoo_put", line 286, in <module>
    sys.exit(main())
  File "degoo_put", line 211, in main
    result = degoo.put(args.local, args.remote, args.verbose, not args.force, args.dryrun, args.scheduled)
  File "/Degoo/degoo/util.py", line 1129, in put
    return put_file(local_path, remote_folder, verbose, if_changed, dry_run, schedule)
  File "/Degoo/degoo/util.py", line 886, in put_file
    dest = get_item(remote_folder)
  File "/Degoo/degoo/util.py", line 535, in get_item
    contents = get_children(part_id)
  File "/Degoo/degoo/util.py", line 604, in get_children
    __CACHE_CONTENTS__[dir_id] = api.getFileChildren3(dir_id)
  File "/Degoo/degoo/API.py", line 541, in getFileChildren3
    raise self.Error(f"getFileChildren3 failed with: {message}")
degoo.API.API.Error: getFileChildren3 failed with: Too large input!
degoo_put: getFileChildren3 failed with: Too large input!
           for help use --help

degoo_user

root@093ff746f382:/Degoo# python3 degoo_user    
Logged in user:
        Name: 
        Email: [email protected]
        Phone: None
        AvatarURL: 
        AccountType: Pro 500 GB
        UsedQuota: 0 Bytes
        TotalQuota: 107.4 GB

Currently, when trying to log in, the following error is returned with a 403

{"message":"Missing Authentication Token"}

(I added debug messages to line 169 of degoo/__init__.py)

    if CREDS != {}:
        response = requests.post(URL_login, data=json.dumps(CREDS))
        print(response.headers)
        print(response.text)

I was trying to use this tool to backup my NAS to Degoo. Started with cloning this repo to python:3-alpine, installed libmagic and all the requirements.

When trying to login using python commands.py login, I get a 429 like so:

/usr/src/Degoo # python commands.py login
username: <emailID>
Password:
Login failed with: 429: Too Many Requests
Login failed.
/usr/src/Degoo #

Am I doing something wrong here?