vuetubeapp / vuetube-extractor Goto Github PK

Describe your suggested feature

Hi there,

It seems like the service worker endpoint (sw.js) will sometimes fail to return the config data (depending on where/when the extractor is running).

This can be fixed by using /sw.js_data instead, which doesn't return any js/html, and is likely what provides data to sw.js.

Other details

See this issue for more context.

Acknowledgements

• I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
• I have written a short but informative title.
• I will fill out all of the requested information in this form.

Ask your question

Can a roadmap be added and a guide be added too to this project (VueTube Extractor). I want to contribute but don't know how this can be integrated and what is expected.

Aditional information

No response

Acknowledgements

• I have searched the existing issues and this is a new question, NOT a duplicate or related to another open issue.
• I have written a short but informative title.
• I will fill out all of the requested information in this form.
• My question isn't asked in FAQ (Frequently Asked Questions).

This one is slightly complicated by how Youtube's android/ios API responds. Whereas most other aspects of the API respond in a fairly straightforward manner, the order of the comment and the data for the comments are separated.

For example, given this request

curl --request POST \
  --url 'https://www.youtube.com/youtubei/v1/next?key=AIzaSyDCU8hByM-4DrUqRUYnGn-3llEO78bcxq8' \
  --header 'Content-Type: application/json' \
  --data '{
    "context": {
        "client": {
            "clientName": "ANDROID",
            "clientVersion": "17.20",
            "hl": "en"
        }
    },
    "continuation": "Eg0SC2RRdzR3OVdnWGNRGAYyJSIRIgtkUXc0dzlXZ1hjUTAAeAJCEGNvbW1lbnRzLXNlY3Rpb24%3D"
}'

The response will be in two parts as follows (parts are cut for readability)

{
  "continuationContents": {
    "itemSectionContinuation": {
      "contents": [
        {
          "elementRenderer": {
            "newElement": {
              "childElements": [
                {
                  "type": {
                    "componentType": {
                      "model": {
                        "commentThreadModel": {
                          "hidePinnedComments": true,
                          "trackingParams": "CK0REMJ1GI4CIhMIzvLZhsmM_QIVXKFLBR1JegGU",
                          "hideSeparator": false,
                          "commentKey": "EhpVZ3phcnFqYWFQQzdUYkZJTk54NEFhQUJBZyAoKAE%3D",
                          "sharedKey": "EgAgKygB",
                          "surfaceKeys": {
                            "commentSurfaceKey": "Eh1VZ3phcnFqYWFQQzdUYkZJTk54NEFhQUJBZy8xMiBPKAE%3D",
                            "toolbarSurfaceKey": "Eh1VZ3phcnFqYWFQQzdUYkZJTk54NEFhQUJBZy8xMiBGKAE%3D"
                          },
                          "pinnedStateKey": "EhpVZ3phcnFqYWFQQzdUYkZJTk54NEFhQUJBZyBNKAE%3D",
                          "composerDraftEntityKey": "N/A",
                          "inlineRepliesKey": "empty_inline_replies"
                        }
                      }
                    }
                  }
                }
              ]
            }
          }
        }
      ]
    }
  },
  "frameworkUpdates": {
    "entityBatchUpdate": {
      "mutations": [
        {
          "entityKey": "EhpVZ3phcnFqYWFQQzdUYkZJTk54NEFhQUJBZyAoKAE%3D",
          "type": "ENTITY_MUTATION_TYPE_REPLACE",
          "payload": {
            "commentEntityPayload": {
              "key": "EhpVZ3phcnFqYWFQQzdUYkZJTk54NEFhQUJBZyAoKAE%3D",
              "properties": {
                "commentId": "UgzarqjaaPC7TbFINNx4AaABAg",
                "content": {
                  "content": "1 BILLION views for Never Gonna Give You Up!  Amazing, crazy, wonderful! Rick ♥️"
                },
                "publishedTime": "1y ago",
                "replyLevel": 0,
                "authorButtonA11y": "Rick Astley",
                "toolbarStateKey": "EhpVZ3phcnFqYWFQQzdUYkZJTk54NEFhQUJBZyAsKAE%3D",
                "translateButtonEntityKey": "EhpVZ3phcnFqYWFQQzdUYkZJTk54NEFhQUJBZyD_ASgB"
              },
              "author": {
                "channelId": "UCuAXFkgsw1L7xaCfnd5JJOw",
                "displayName": "Rick Astley",
                "avatarThumbnailUrl": "https://yt3.ggpht.com/BbWaWU-qyR5nfxxXclxsI8zepppYL5x1agIPGfRdXFm5fPEewDsRRWg4x6P6fdKNhj84GoUpUI4=s88-c-k-c0x00ffffff-no-rj",
                "isVerified": false,
                "isCurrentUser": false,
                "isCreator": true,
                "channelCommand": {
                  "innertubeCommand": {
                    "browseEndpoint": {
                      "browseId": "UCuAXFkgsw1L7xaCfnd5JJOw",
                      "canonicalBaseUrl": "/channel/UCuAXFkgsw1L7xaCfnd5JJOw"
                    }
                  }
                },
                "innerBadgeA11y": "Rick Astley, Official Artist Channel",
                "isArtist": true
              },
              "toolbar": {
                "likeCountLiked": "1.1M",
                "likeCountNotliked": "1.1M",
                "replyCount": "500",
                "creatorThumbnailUrl": "https://yt3.ggpht.com/BbWaWU-qyR5nfxxXclxsI8zepppYL5x1agIPGfRdXFm5fPEewDsRRWg4x6P6fdKNhj84GoUpUI4=s88-c-k-c0x00ffffff-no-rj",
                "likeButtonA11y": "Like this comment along with 1.1M other people",
                "engagementToolbarStyle": {
                  "value": "ENGAGEMENT_TOOLBAR_STYLE_VALUE_DEFAULT"
                },
                "likeCountA11y": "1.1M likes",
                "replyCountA11y": "500 replies"
              },
              "avatar": {
                "image": {
                  "sources": [
                    {
                      "url": "https://yt3.ggpht.com/BbWaWU-qyR5nfxxXclxsI8zepppYL5x1agIPGfRdXFm5fPEewDsRRWg4x6P6fdKNhj84GoUpUI4=s88-c-k-c0x00ffffff-no-rj",
                      "width": 88,
                      "height": 88
                    }
                  ]
                },
                "accessibilityText": "Rick Astley",
                "avatarImageSize": "AVATAR_SIZE_XS",
                "endpoint": {
                  "innertubeCommand": {
                    "browseEndpoint": {
                      "browseId": "UCuAXFkgsw1L7xaCfnd5JJOw",
                      "canonicalBaseUrl": "/channel/UCuAXFkgsw1L7xaCfnd5JJOw"
                    }
                  }
                }
              }
            }
          }
        }
      ]
    }
  }
}

Note the commentKey and entityKey fields under continuationContents and frameworkUpdates respectively. I am not sure how best to parse this, so any input will be appreciated.

Of course, there is the workaround of calling the web version of the API instead, but that will mean the comments in VueTube will not be able to access some app-exclusive features from the Youtube app without some serious workarounds.

This issue tracks the state of core systems for the extractor rewrite (located in the youtube-new directory).

Parser functionalities

Features both planned and implemented are described in the youtube-new documentation

• Optional Properties
• Default Values
• Type Checking
• Strict Mode
• Sub-Rules
• Aliases
• Sub-rule auto application
• Key Mapping
• Result flattening
• JSONPath Dot Notation
• Arrays
• Conditional Rule Application
• Functional Conditions
• Rule Conditions

Requester functionalities

The requester covers any feature that interacts with remote APIs

• Initialising values (API key, context value, visitor data, etc)
• Retrieve Data from the Home Page (/browse endpoint)
• Retrieve basic Information of a video (/player endpoint)
• Retrieve information from the watch page (/next endpoint, ryd API, sponsorblock)
• Retrieve information from playlists (/browse endpoint)
• Retrieve information from search (/search endpoint)
• #60
• Retrieve information for comment replies (/browse endpoint)
• Retrieve search suggestions (suggestqueries.google.com)
• Send watch information to the youtube server (/api/stats/qoe endpoint)
• Retrieve information from channels (/browse endpoint)
• Pagination support for various features (/next endpoint)

Overall progress

• Search
• Search Filters
• Search Suggestions
• Home Page
• Home Page Continuation
• Video Information
• Comments
• Replies
• Channel Information

Steps to reproduce

See #299, #298, #297, etc.

For some reason all requester tests fail with github actions. This behavior cannot be replicated on a local environment.

This might be a problem with how GH actions is set up, or it may be something else. Further investigation is warranted.

Expected behavior

Tests should pass across all platforms

Actual behavior

Tests fail on github only

Extractor version

0.1.0

Node version

v14.x

Nation or locality

No response

Other details

No response

Acknowledgements

• I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
• I have written a short but informative title.
• I have updated to the latest version of the extractor and confirmed this issue still exists.
• I will fill out all of the requested information in this form.

Steps to reproduce

It would appear visitorData has a few complications and that not all randomly generated 11 character string is valid. This results in not so uncommon cases where the home page test fails despite everything else working as expected

Expected behavior

home page tests should work all the time

Actual behavior

sometimes home page tests fails with error: " No section contents"

Other details

No response

Acknowledgements

• I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
• I have written a short but informative title.
• I have confirmed that this is an issue with VueTube Extractor, and not the main app.
• I will fill out all of the requested information in this form.

Stable Release Checklist

This issue will outline the needed features before VueTube Extractor is able to completely replace the current solution employed by VueTube.

This will be updated often. Stay tuned.

Update: Due to the new rewrite, this checklist no longer reflects the actual state of the extractor. However, it will continue to be updated as the rewrite catches up to version 1's feature sets

Needed Features

• Search
• Search Filters
• Search Suggestions
• Home Page
• Home Page Continuation
• Video Information
• Comments
• Replies
• Channel Information