JSON decode error deleting uploads about ytmusic-deleter

[zebrapurring]

I'm able to reproduce the same error on Linux and Windows.

[zebrapurring]

Installing the brotli package fixes the issue for me:

pip install brotli

So it's definitely related to the server using an unsupported encoding...

[apastel]

So I've confirmed the issue arises with the way the headers_auth.json is composed.

I created a fresh venv in Windows, and manually created a headers_auth.json file using the instructions for "Manual file creation" in ytmusicapi found here: https://ytmusicapi.readthedocs.io/en/stable/setup.html#manual-file-creation

Tested delete-uploads and it worked.

Then I deleted that headers_auth.json file and created a new one by pasting @zebrapurring 's json example from this thread, and replaced the "XXX" values with my own request header values. Then tested delete-uploads and got the following error:

[2023-03-13 21:39:06] Looking for C:\Users\alexr\Downloads\headers_auth.json"
[2023-03-13 21:39:06] Found C:\Users\alexr\Downloads\headers_auth.json"
[2023-03-13 21:39:06] Retrieving all uploaded albums...
Traceback (most recent call last):
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\Scripts\ytmusic-deleter.exe\__main__.py", line 7, in <module>
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\site-packages\ytmusic_deleter\cli.py", line 82, in delete_uploads
    (albums_deleted, albums_total) = delete_uploaded_albums(ctx, add_to_library)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\site-packages\ytmusic_deleter\cli.py", line 96, in delete_uploaded_albums
    uploaded_albums = youtube_auth.get_library_upload_albums(sys.maxsize)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\site-packages\ytmusicapi\mixins\uploads.py", line 75, in get_library_upload_albums
    response = self._send_request(endpoint, body)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\site-packages\ytmusicapi\ytmusic.py", line 139, in _send_request
    response_text = json.loads(response.text)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\alexr\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

So the error is very likely because of one or more of the fields included in the headers_auth.json that both of you created manually. Give it a try using the instructions I just linked -- you should only have to replace the Cookie field. I will update my README to mention that macOS users may have to create the headers_auth.json file manually and to follow the instructions here: https://ytmusicapi.readthedocs.io/en/stable/setup.html#manual-file-creation

Thank you both for bringing it to my attention since I don't have a machine running macOS to test on.

EDIT: After more testing it looks like the specific field that was causing the problem in your manual headers_auth.json file was the accept-encoding as @erikw figured out. Simply remove that property from your headers_auth.json altogether because the ytmusicapi generates it on its own, and it omits br. If br is present, it will not work, even if you add utf-8. Just remove br or delete the accept-encoding property altogether.

[apastel]

Added note to README about manual file creation for macOS users.

https://github.com/apastel/ytmusic-deleter/blob/master/README.md#authenticating-to-your-account

[apastel]

Hmm, interesting. Well the latest version of ytmusicapi is v0.25.0, and if you're using the latest version of my ytmusic-deleter, it uses that version. I'm looking at this issue on the ytmusicapi repo which seems related: sigma67/ytmusicapi#268

Most likely the server returned some error and that's why ytmusicapi is unable to parse it at JSON. I wonder if it's an authentication issue like the person in the thread was having. Can you try deleting the headers_auth.json file that should've been generated when you first went through the authentication process, and do that process again and see if you still get the issue? I did not have this issue when testing it just now on my Windows machine.

[zebrapurring]

I was unable to create headers_auth.json from the command prompt, when I paste the headers copied from the browser (which use format header: value), the input freezes and pressing CTRL+D has no effect.

I had to create the file manually, this is what it looks like:

{
    "Authorization": "XXX",
    "Connection": "keep-alive",
    "Origin": "https://music.youtube.com",
    "X-Goog-AuthUser": "0",
    "X-Youtube-Bootstrap-Logged-In": "true",
    "Cookie": "XXX",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0",
    "X-Goog-Visitor-Id": "XXX",
    "Content-Type": "application/json",
    "Sec-Fetch-Mode": "same-origin",
    "Host": "music.youtube.com",
    "X-Origin": "https://music.youtube.com",
    "Referer": "https://music.youtube.com/",
    "Alt-Used": "music.youtube.com",
    "Content-Length": "2132",
    "Accept": "*/*",
    "X-Youtube-Client-Version": "1.20230306.01.00",
    "Sec-Fetch-Dest": "empty",
    "Accept-Encoding": "gzip, deflate, br",
    "Sec-Fetch-Site": "same-origin",
    "X-Youtube-Client-Name": "67",
    "TE": "trailers"
}

[erikw]

I'm getting the same 2 issues as well:

• The request header input hangs (not possible to Ctrl+d). The workaround of manually creating a `headers_auth.json works
• The call to 'https://music.youtube.com/youtubei/v1/browse?alt=json&key=REDACT' fails with the same stacktrace for me
  • Fails here https://github.com/sigma67/ytmusicapi/blob/1ad3dc2312e464e709d82cea8b0ae89672012a72/ytmusicapi/ytmusic.py#L139 because response.text is jibberish (need to be decoded it looks like)

Edit.
I see from the response that it had response.headers: 'Content-Encoding': 'br',. The latest version of python Requests should handle the brotli encoding automatically according to this post:
https://stackoverflow.com/a/69721272

It might be a matter of updating the project dependencies to use the latest Requests version.

Edit.2 WORKAROUND
A tip I found here https://stackoverflow.com/a/67799052
is to change the request headers instead to ask for UTF-8 directly.

I went to my system's pip installation of
https://github.com/sigma67/ytmusicapi/blob/1ad3dc2312e464e709d82cea8b0ae89672012a72/ytmusicapi/helpers.py#L14
and changed it like this

-         "accept-encoding": "gzip, deflate",
+         "accept-encoding": "gzip, deflate, utf-8",

and now ytmusic-deleter delete-uploads runs fine!

Just as a temporary hack to workaround the issue

[apastel]

Thanks @erikw for the detailed analysis. Are you also on a Mac? Wondering why this issue wasn't happening for me or why I don't see any issues for it in ytmusicapi. I don't directly depend on requests; just ytmusicapi of which I am running the latest.

[erikw]

Yes @apastel I'm on macOS as well as @zebrapurring

[zebrapurring]

Also tried @erikw's workaround adding UTF-8 to the Accept-Encoding header, but still no luck 😓

[erikw]

@zebrapurring hmm and you're sure you edited the exact right ytmusicapi pip installation used by this tool?

From your original stacktrace that should be

./ytdelete/.venv/lib/python3.9/site-packages/ytmusicapi/helpers.py

[erikw]

@zebrapurring funny because I also tried installing the brotli package! But it didn't solve it for me so I went change the request instead of the response

[zebrapurring]

@zebrapurring hmm and you're sure you edited the exact right ytmusicapi pip installation used by this tool?

@erikw yes, adding utf-8 had no effect whatsoever for me. But after installing brotli it runs with no issues.

[apastel]

Glad you guys were both able to get it working but I'll try to see if I can replicate this and fix it in all scenarios, although it seems more like it's an issue for ytmusicapi. Either way, I don't want ytmusic-deleter to have a bug if it's possible to fix it here. I wonder if the manual creation of the headers_auth.json file has anything to do with it. Apparently MacOS terminal applications can only accept 1024 characters pasted to stdin (see the dropdown section here entitled "MacOS special pasting instructions" https://ytmusicapi.readthedocs.io/en/stable/setup.html#).

[erikw]

Ah it all makes sense now!!