bernd-wechner / degoo

CLI tool(s) for working with Degoo cloud storage

Degoo CLI Util(s)

Degoo are a cloud storage provider based in Sweden, who provide fairly good phone apps and web interface along with affordable plans, of up 10 TB storage.

https://degoo.com/

Unfortunately the apps (phone and web) are pitched to a very particular demography of user, and are rounded with great tools for storing photos, videos, music and documents in the cloud.

This makes it very difficult to use the cloud storage for flexible backup of data.

Here's a fairly impartial review you might find useful: Degoo Review 2020 - This Is Why You Shouldn't Use It

Support seems hit and miss. They are a small company: https://degoo.com/about

And have (only) two people on Customer Support so if they have 15 million users, then clearly they'd struggle to deliver customer support well. It's called outgrowing your boots.

They have also chopped and churned, originally using P2P storage then moving away from that after a load of poor reviews. They had a Windows desktop client, but no more. They are still, it seems clearly trying to find their niche in this market and establish a service model that secures a lasting future.

My interest is in keeping server data backed-up in the cloud.

And so, by studying their web app (which is written in Angular JS and managed with Webpack, communicating with a backend over a graphQL interface) I've written a simple CLI (command line interface) to the cloud storage.

It is written in Python and being developed under Linux. Being Python it's very likely highly portable, but there may be some small issues running on other systems. The only issues I can think of currently are:

• It tries to use os.sep intelligently to give you a natural feel if say you're using Windows where it's \ rather than Linux of MacOS where it's /. But that's untested so far on Windows.

• It uses python-magic to determine file types (needed for upload, as the API seems to demand this metadata). That may have some system dependencies.

• It uses a custom patched version of python-wget as the one that pip provides is lacking some crucial features and the package seems sadly unmaintained and dead to the world. This patched version stands as an open Pull Request on the upstream with no action.

A work in progress, it's not complete but at present it can reliably:

• log you in (if you provide valid credentials)

• list files and folders on the cloud drive (ls, tree)

• navigate the cloud drive (cd, pwd)

• manipulate the cloud drive (mkdir, rm)

• download files from the cloud drive (get)

• upload files to the cloud drive (put)

Not implemented yet:

• Top Secret Cloud Storage

  • Degoo provide a good security focused solution with their Top Secret vault, that they claim is 100% NSA proof. Only available on their phone app for now, not the web app. Will take some effort to analyze the client-server interactions to provide CLI support.

• Device creation

  • Top level directories on your Degoo cloud drive are reserved for devices. Different licenses provide different numbers of devices. Currently you can delete a device but there is no facility for adding one again, or if you're an Ultimate license holder adding new devices (which should be possible, but the web interface provides no such facility).

Using these tools

This is a work in progress (WIP) still and may or may not work well. Works for me ;-). But here are some quick tips if you're wanting to try it.

• Requires Python 3.9

• Requires the python packages in requirements.txt, install them with pip install -r requirements.txt

• The core of it is all implemented in three files currently:

  • degoo/API.py which defines the basic Python API
  • degoo/util.py which defines the functions for a CLI
  • commands.py which implements a command line tool (that is sensitive to its name)

• To build the command line tools there:

  • build.py - which just creates a pile of links to commands.py named as a command line tools. That's a dirty trick of sorts I used to give me a pile of CLI commands to work with so I can write bash scripts etc.

• There's no system installer yet, it's all working in the local dir as I work on it. I haven't yet put this to use as a serious backup strategy and am working on some areas to get there (slowly, when time permits)

• If you want to debug, personally I can't recommend Eclipse+PyDev more highly, that's what I use. PyCharm is popular but freemium; it might be easier to get started with.

• If you're wanting to look at how it's been engineered:

  • Surf to degoo.com in Firefox or Chrome
  • Open the Developer tools (F12)
  • Click the Network tab
  • Log in to degoo and watch the traffic.
  • You can save all that into text files and then start diagnosing.

degoo's Issues

Possible login issue

I'm having problems logging in. Has Degoo changed their login protocols?

guide

Could you add any guide for Linux beginners please ?
Maybe video tutorial for installation and use ?

Getting lots of <Response 504> errors.

I seem to get for following on many files:

Traceback (most recent call last):
File "/home/dberger/degoo.DLBerger/./degoo_put", line 286, in
sys.exit(main())
File "/home/dberger/degoo.DLBerger/./degoo_put", line 211, in main
result = degoo.put(args.local, args.remote, args.verbose, not args.force, args.dryrun, args.scheduled)
File "/home/dberger/degoo.DLBerger/degoo/util.py", line 1156, in put
return put_directory(local_path, remote_folder, verbose, if_changed, dry_run, schedule)
File "/home/dberger/degoo.DLBerger/degoo/util.py", line 1134, in put_directory
put_file(Name, IDs[relative_root], verbose, if_changed, dry_run, schedule)
File "/home/dberger/degoo.DLBerger/degoo/util.py", line 1035, in put_file
degoo_id = api.setUploadFile3(filename, dir_id, verbose, Size, Checksum)
File "/home/dberger/degoo.DLBerger/degoo/API.py", line 795, in setUploadFile3
raise self.Error(f"setUploadFile3 failed with: {response}")
degoo.API.API.Error: setUploadFile3 failed with: <Response [504]>
degoo_put: setUploadFile3 failed with: <Response [504]>
for help use --help

Any idea what it means?

List of commands that can be used?

Hello, this project is a few months old and has most likely been abandoned. However, I'm not actually able to find clear instructions or documentation. I'd love it if the devs could help me out.
How exactly do I use it? So far I've gathered that I:

1. Enter the directory with cd ~/Downloads/degoo/
2. Add the packages with pip install -r requirements.txt
3. Build it with python3.9 build.py

But what do I do that? Now that I've built the commands, I need to know what the commands are! For example, if I wanted to upload a file located in the path ~/image.jpg to my degoo account, how would I do this?
Where can I get any docs related to this?

Support a single CLI command with options

Currently I've built this so it have a load of distinct command line utilities. That's nice in a way. They include:

degoo_cd
degoo_get
degoo_ll
degoo_login
degoo_ls
degoo_mkdir
degoo_path
degoo_props
degoo_put
degoo_pwd
degoo_rm
degoo_test
degoo_tree
degoo_user

It would be nice and others might also appreciate if all these had a way of being invoked through one CLI command for example:

degoo cd
degoo get
degoo ll
degoo login
degoo ls
degoo mkdir
degoo path
degoo props
degoo put
degoo pwd
degoo rm
degoo test
degoo tree
degoo user

Should be easy to support both methods. Just writing this note to remind myself to do it some time.

403 Error when trying to login

I tried installing everything but I get this error

Login failed with: 403: Forbidden
Login failed.

How to install this in termux & use it

I cloned & pip install -r requirements & python or python3 build.py but
raise ImportError('failed to find libmagic. Check your installation')
ImportError: failed to find libmagic. Check your installation
This happens everytime. I dont have pc & I want to download all of my degoo data plz help

No download URL

I am using Cryptomator to encrypt my files before uploading to Degoo service.
Upload using degoo_put works (after making a small change in code):

index 8043121..c27976a 100644
--- a/degoo/util.py
+++ b/degoo/util.py
@@ -1162,7 +1162,7 @@ def put_directory(local_directory, remote_folder, verbose=0, if_changed=False, d
             relative_root = root
 
         for name in dirs:
-            Name = os.path.join(root, name)
+            Name = os.path.join(relative_root, name)
 
             IDs[Name] = mkdir(name, IDs[relative_root], verbose - 1, dry_run)

python degoo_put /mnt/kuvat /Web

After successfull upload I try to download the same directory to another folder:

python degoo_get /Web/kuvat /mnt/kuvat2

Some of the files are downloaded correctly while some of them output error:
<file> apparently has no URL to download from.

Has anyone faced such problems in the past? Any help how to debug the issue is appreciated.
Zip package of files I have used (to try to reproduce the problem) can be downloaded from here

degoo_put: [Errno 22] Invalid argument

This error occurs when trying to upload a file with a time zone other than UTC (in my case +1). The problem is in the has_changed method, specifically when trying to calculate LastUploadTime:

LastUploadTime = datetime.datetime.utcfromtimestamp(0).replace(tzinfo=tz.UTC).astimezone(tz.tzlocal()) # Internally it tries to subtract 0 - 3600, which causes the exception

I think this can be fixed by changing the following, working on Windows as on Ubuntu:
LastUploadTime = datetime.datetime.now(tz.tzlocal())

Any further development?

Make uploads and downloads restartable

For large files in particular there's a real risk that a transfer is interrupted. For that reason most web tech today and the Google Cloud Store in particular support restartable downloads and uploads.

This CLI would need to do something in prep to exploit that, which needs a little research and testing of ideas. It may fall flat if it requires the Google Cloud Store private key to work, as that is in Degoo's hands, we can't interact freely with the cloud store only via the Degoo GraphQL interface and that's only doable if their web interface implements a restartable download/upload that can be watched. I'm not sure it does.

How to use?

I am currently setting degoo on my RPI and have the following issue:

zeus@rpi-3b ~/src/Degoo (master*) $ python build.py
Traceback (most recent call last):
File "build.py", line 20, in
import degoo
File "/home/zeus/src/Degoo/degoo/init.py", line 41, in
from .util import *
File "/home/zeus/src/Degoo/degoo/util.py", line 136
print(f"Waiting until {until.strftime('%A, %d/%m/%Y %H:%M:%S')}")

How should I proceed?

degoo_ll: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Got this generic issue: most of the commands give the same error 'int() argument must be a string, a bytes-like object or a number, not 'NoneType''

• ./degoo_login, ./degoo-pwd and ./degoo_user give the expected output (Successfuly logged in. [typo here though > Succesfully], Working Directory is / & user info respectively)
• ./degoo_mkdir test outputs to degoo_mkdir: mkdir: No parent_id provided.
• Most of the others output to int() argument must be a string, a bytes-like object or a number, not 'NoneType'.

Some misconfiguration (by me)?

Missing Python Requirements

Had to add the following to requirements.txt before build.py would run:

clint==0.5.1
requests-toolbelt==0.9.1

FUSE?

Thanks! This looks great, as I theoretically 13TB lifetime with them, which I can't use because they changed/abandoned their client software. (Don't get, why they don't at least expose a WebDAV interface to their storage.)

That said, 13TB, even with CLI tools aren't usable for much without lots of wasted time uploading, checking, etc.

Ideal would be a FUSE interface ( https://wiki.osdev.org/FUSE ), so the storage can be mounted as a network drive, and then used with random backup, file-sync, etc. software, without degoo specific code. Is that a direction you plan to take your code in?

The reason I ask: you seem to have already implemented the type of operations that sftp offers, and there's a FUSE abstraction layer based on those ( https://github.com/libfuse/sshfs )

Trying to run in python env on mac get : ModuleNotFoundError: No module named 'appdirs'

from appdirs import user_config_dir
ModuleNotFoundError: No module named 'appdirs'

I have confirmed appdirs was installed so not sure what's going on or if these tool even works anymore ?

Login reports missing token

Currently, when trying to log in, the following error is returned with a 403

{"message":"Missing Authentication Token"}

(I added debug messages to line 169 of degoo/__init__.py)

    if CREDS != {}:
        response = requests.post(URL_login, data=json.dumps(CREDS))
        print(response.headers)
        print(response.text)

Bring download and upload toolbars into conformance

It's awesome that we now have progress bars on download and upload but they have distinct styles and it would be nice to bring them into conformance. The differences are presently as follows:

1. Style:

   Get:
   100% [########################################################################] 82M / 82M

   Put:
   [##########          ] 80044032/234490033 - 00:00:58

   Get is nicer. But Put has an ETA which is awesome.

   Both are lacking the filename.

   They apply file by file when doing directory puts and gets but don't show the current filename. This should be fixed.

2. History:

   Get shows the history in that every new file starts a new line and you see a load of 100% progress bars above the current one running. Put overwrites each progress bar and is only ever displaying the currently running one so no history is kept on screen.

   The two use different packages to achieve their result, and eitehr one may be more or less configurable.

   The get progress bar is delivered by the wget package:

   https://github.com/bernd-wechner/python3-wget/blob/e38979ffdf462a5cec05c5a92ff35a7630c6435b/wget.py#L286

   and the put progress bar is delivered by the clint package:

   https://github.com/kennethreitz-archive/clint/blob/9d3693d644b8587d985972b6075d970096f6439e/clint/textui/progress.py#L33

I suspect the best approach is to:

1. Generalise the wget progress bar extracting it from the wget library and including it as a degoo cli function: https://github.com/bernd-wechner/python3-wget/blob/e38979ffdf462a5cec05c5a92ff35a7630c6435b/wget.py#L166
2. Extend it to accept a filename
3. plug this bar in to the wget call: wget.download(bar= ...) and the MultipartEncoderMonitor(callback=...)

We'd then have consistent bars and behaviour and the best look (I am thinking the wget adaptive bar plus an ETA added and a filename).

Downloads seem to stop at 128MB

This seems to be a bug at the Degoo end. Am curious if it is global, if others are experiencing same (or account specific or such). I have filed a support ticket with Degoo.

The issue is easily reproduced:

1. Upload a large file (I use video files)
2. Download it again. Only the first 128MB are downloaded.

This turns out to be true using degoo_put and degoo_get from this repo, but also using the Degoo web app. Hence the conclusion it is at the Degoo end. it is the wget library:

https://github.com/bernd-wechner/python3-wget

That is simply returning, with no error, before the file is complete, consistently at about 128MB irrespective of the file size (as long as it's over 128MB ;-).

Missing Dependency

Ubuntu Server 20.04 LTS
Running pip install -r requirements.txt
Receive error: ERROR: launchpadlib 1.10.13 requires testresources, which is not installed.

Fix: pip install testresources before installing the requirements, but maybe it needs to be added to them?

degoo_mv

I think you added this to the repository by accident.

429 on login

I was trying to use this tool to backup my NAS to Degoo. Started with cloning this repo to python:3-alpine, installed libmagic and all the requirements.

When trying to login using python commands.py login, I get a 429 like so:

/usr/src/Degoo # python commands.py login
username: <emailID>
Password:
Login failed with: 429: Too Many Requests
Login failed.
/usr/src/Degoo #

Am I doing something wrong here?

How to Use degoo_put command ?

I have tried every possible command other than this.
Please anyone give a hint. or just an example.

degoo_put

root@093ff746f382:/Degoo# python3 degoo_put /Degoo/testuploadfile /
Traceback (most recent call last):
  File "degoo_put", line 286, in <module>
    sys.exit(main())
  File "degoo_put", line 211, in main
    result = degoo.put(args.local, args.remote, args.verbose, not args.force, args.dryrun, args.scheduled)
  File "/Degoo/degoo/util.py", line 1129, in put
    return put_file(local_path, remote_folder, verbose, if_changed, dry_run, schedule)
  File "/Degoo/degoo/util.py", line 886, in put_file
    dest = get_item(remote_folder)
  File "/Degoo/degoo/util.py", line 535, in get_item
    contents = get_children(part_id)
  File "/Degoo/degoo/util.py", line 604, in get_children
    __CACHE_CONTENTS__[dir_id] = api.getFileChildren3(dir_id)
  File "/Degoo/degoo/API.py", line 541, in getFileChildren3
    raise self.Error(f"getFileChildren3 failed with: {message}")
degoo.API.API.Error: getFileChildren3 failed with: Too large input!
degoo_put: getFileChildren3 failed with: Too large input!
           for help use --help

degoo_user

root@093ff746f382:/Degoo# python3 degoo_user    
Logged in user:
        Name: 
        Email: [email protected]
        Phone: None
        AvatarURL: 
        AccountType: Pro 500 GB
        UsedQuota: 0 Bytes
        TotalQuota: 107.4 GB

response 400 for all commands

I'm getting this every time now it was working fine a few weeks ago. I updated to the newest version and still get this.

Traceback (most recent call last):
  File "/tmp/degoo2/./degoo_ls", line 328, in <module>
    sys.exit(main())
  File "/tmp/degoo2/./degoo_ls", line 116, in main
    degoo.ls(args.folder, args.long, args.human, args.recursive)
  File "/tmp/degoo2/degoo/util.py", line 1210, in ls
    items = get_children(directory)
  File "/tmp/degoo2/degoo/util.py", line 604, in get_children
    __CACHE_CONTENTS__[dir_id] = api.getFileChildren3Ex(dir_id)
  File "/tmp/degoo2/degoo/API.py", line 659, in getFileChildren3Ex
    (next_items, next_token) = self.getFileChildren3(dir_id, next_token)
  File "/tmp/degoo2/degoo/API.py", line 650, in getFileChildren3
    raise self.Error(f"getFileChildren3 failed with: {response}")
degoo.API.API.Error: getFileChildren3 failed with: <Response [400]>
degoo_ls: getFileChildren3 failed with: <Response [400]>
          for help use --help

Add hash comparison to the has_changed() function

Degoo metadata does not include a remote hash. We can calculate one locally of course. But there is a good chance we can coax one out of the backed Google File Store that Degoo are using:

Here's a place to start reading:

https://stackoverflow.com/questions/25234515/how-to-get-the-md5-of-an-object-that-has-already-been-stored

getFileChildren3 failed with: <Response [400]>

Hi there, I really appreciate such project exists. I really regret using Degoo and it is getting more and more ridiculous every day.

I've encountered an error where all the command doesn't work except degoo_login, degoo_pwd and degoo_user. I'm using Python 3.9.1 on Ubuntu 20.10 and Debian 10 just for reference. Whenever a command is executed it always returns with an error getFileChildren3 failed with: <Response [400]>. All python module requirements are installed. Adding the verbose flag doesn't return any extra debug info except for the error above. Below is an image that I have attached.

List dependencies

Thank you for contributing this. I got sucked in to Degoo, only to realize the frustration that they offer. This is something we all need.

Consider listing some dependencies that this requires?

From python3.8 (MacPorts, MacOS Big Sur):

Traceback (most recent call last):
File "build.py", line 20, in
import os, stat, degoo
File "/usr/local/src/degoo/degoo/init.py", line 23, in
import os, sys, csv, json, time, datetime, requests, wget, magic, humanize, humanfriendly, hashlib, base64
ModuleNotFoundError: No module named 'requests'

Move and rename commands

I have created a new branch in which I have created the commands rename and mv. I hope this will help to complete all the available commands.

Stack-trace while listing subfolder items

Hello, I am using the Ultimate plan of Degoo. I overcame the issue of 429, by modifying the login part using cloudscrapper, but now I face another issue whenever I try to list items in a subfolder. I get the following stacktrace:

Traceback (most recent call last):
File "./degoo_ll", line 328, in
sys.exit(main())
File "./degoo_ll", line 116, in main
degoo.ls(args.folder, args.long, args.human, args.recursive)
File "/mnt/devel/Degoo/degoo/util.py", line 1210, in ls
items = get_children(directory)
File "/mnt/devel/Degoo/degoo/util.py", line 604, in get_children
CACHE_CONTENTS[dir_id] = api.getFileChildren3(dir_id)
File "/mnt/devel/Degoo/degoo/API.py", line 625, in getFileChildren3
prefix = self.devices[i['DeviceID']] + os.sep + "Recycle Bin" if binned else self.devices[i['DeviceID']]
KeyError: 27437396
degoo_ll: 27437396

I have no problems issuing the same command on the root folder, but all other folders return the same error.

What can I do to troubleshoot the issue?
Thanks!

Francisco

Degoo uses Windows filenaming conventions and not Unix

I found this the hard way.
Degoo sees MyFile and MYFILE as the same file/directory.
Like Windows it retains the filename as created, but trying to create any other variants causes an error.
Not sure why, but it leads to a host of potential problems.

degoo_get: download() got an unexpected keyword argument 'size'

When I try to download a file I get the error download() got an unexpected keyword argument 'size', both in Windows and Ubuntu. I have seen that you need to install https://github.com/bernd-wechner/python3-wget

Honestly, I haven't tried to install this package, but I think that to avoid this error for new users you could download it as standard with requests:

      if not dry_run:
          # _ = wget.download(URL, out=Name, size=Size, headers={'User-Agent': "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/83.0.4103.61 Chrome/83.0.4103.61 Safari/537.36"})
          silent = False # It could be a new argument
          with requests.get(URL, stream=True) as r:
              r.raise_for_status()
              with open(Name, 'wb') as f:
                  if silent:
                      shutil.copyfileobj(r.raw, f)
                  else:
                      dl = 0
                      total_length = int(r.headers.get('content-length'))
                      if not total_length:
                          total_length = Size
                      for data in r.iter_content(chunk_size=4096):
                          dl += len(data)
                          f.write(data)
                          done = int(50 * dl / total_length)
                          sys.stdout.write("\r[%s%s]" % ('=' * done, ' ' * (50-done)))
                          sys.stdout.flush()

I hope it will be helpful.

Running Degoo api on Windows results in os.sep and os.path.abspath conflicts (and possibly others)

When running the code on Windows, the os is set to "NT", which is correct for local disk access. But Degoo is running remotely on a Linux environment. This results in error messages for calls to degoo_cd etc.

A kludge workaround is to override os.sep in util.py, lib.py and API.py as soon as it is imported:
if os.sep == "\":
os.sep = "/"

This at least gives me back degoo_cd, but there are likely other undesirable side effects.

I have still had to comment out part of lib.absolute_remote_path, as this is still failing:
if path and path[0] == os.sep:
# Mike: This call is still a problem
#return os.path.abspath(path.rstrip(os.sep))
return path
For example, if cwd on Degoo was /X51/LightroomOther/2005_extra and I executed "py degoo_cd /X51", I got the following error:
degoo.util.DegooError: C:\ does not exist in /X51LightroomOther/2005_extra
This implies abspath is picking up the default root directory for Windows, rather than UNIX, and also the rstrip was removing a / unnecessarily.

Also relative cd is not working at all if it includes ".." in the path., e.g. degoo_cd .. , or degoo_cd ../Backups. Is it likely this is because I have commented out the call to os.path.abspath, or because it is unsupported?

Any help in making this work in a robust way is appreciated.
Mike

Degoo and Google API

Would anyone have direct contact with someone at Degoo; at least, at the business level.

I ask (party in ignorance): wouldn't they benefit from allowing authenticated access to one's storage via the Google API, where various tools could be used at that level? For example, I use "rclone" for interacting with Google Drive and S3; it's fast, efficient.

Pardon if this is the wrong place to ask this, I didn't see a mailing list.

How do I download all Degoo files at once?

Could you give an description on how to do this after installing the req's? Thanks a lot

Login Failed with: 400:

Hi there,
I cannot login.

I get the error 400

./degoo_login                            
username: [email protected]                                             
Password:                                                               
Login failed with: 400:                                                 
Login failed.                                                           

Running it with -vvv doesn't produce any additional log.

Can you support?

Issue uploading large files

When I try to upload large files, a MemoryError exception occurs when executing request.post. I have tried with a 1Gb file. I managed to fix it by changing this:

# Perform the upload
# TODO: Can we get a progress bar on the this? Web app has one.
response = requests.post(BaseURL, files=parts, headers=heads)

By this:

from requests_toolbelt import MultipartEncoder

multipart = MultipartEncoder(fields=dict(parts))
heads['content-type'] = multipart.content_type
response = requests.post(BaseURL, data=multipart, headers=heads)

Greetings.

Bug: On first run fails to create the degoo app folder when trying to create cred_file

On login attempt, I get no response (error, confirmation, or other). I see no usage guideance for further troubleshooting. Where do I enter in my username and password?

# ls
LICENSE.md  build.py     default_properties.txt  degoo_cd   degoo_ll     degoo_ls     degoo_path   degoo_put  degoo_rm    degoo_tree 
README.md   commands.py  degoo                   degoo_get  degoo_login  degoo_mkdir  degoo_props  degoo_pwd  degoo_test  degoo_user
# ./commands.py help
# ./commands.py login
# ./commands.py user
# ./commands.py ls