russell-newton / tiktokpy

Extract data from TikTok without needing any login information or API keys.

Hello! I'm Russell Newton

Check out my portfolio for the most up-to-date information about me and my projects!

tiktokpy's Issues

Get Video URL

Is there a way to get the video url from the Video object

Exception trying to use an API instance

Hello, it seems that any use of an API instance always gets me the following exception:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.8/site-packages/playwright/_impl/_page.py", line 255, in _on_route
    handled = await route_handler.handle(route)
  File "/home/user/.local/lib/python3.8/site-packages/playwright/_impl/_helper.py", line 290, in handle
    g.switch()
  File "/home/user/.local/lib/python3.8/site-packages/playwright/_impl/_helper.py", line 280, in impl
    result = cast(
  File "/home/user/.local/lib/python3.8/site-packages/playwright/_impl/_impl_to_api_mapping.py", line 123, in wrapper_func
    return handler(
  File "/home/user/.local/lib/python3.8/site-packages/tiktokapipy/api.py", line 292, in capture_api_extras
    response = route.fetch()
AttributeError: 'Route' object has no attribute 'fetch'

It wasn't causing me any trouble before as it doesn't stop execution of the code, but as I started using the multiprocessing library, the processes just die when this exception appears. But even when I was not using multiple processes, it was still very annoying and made it very difficult to debug my code. A try-except block around the API call doesn't seem to solve the issue.

Without Watermark?

Hey..
Its was great repository.

What is the response from get video and also the challenge without a watermark?

[QUESTION] Is there a way get a random video using api.challenge?

Ask your question
I would like to randomly sample videos from a bunch of predefined challenges. Currently when running api.challenge(challenge_name=hashtag, video_limit=1), it will always return the same video back.

E.g. with challenge_name = 'fyp', I always get this video.

How does the API currently choose which videos to grab from a challenge?

[BUG] using tiktok challange failed

Describe the bug
i try run and filter for video under 60 seconds only. After found few data like 5-10 its failed. below is error

Traceback (most recent call last):
  File "C:\Python311\Lib\site-packages\tiktokapipy\api.py", line 363, in _scrape_data
    data = self._extract_and_dump_data(content, extras_json, data_model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\tiktokapipy\api.py", line 394, in _extract_and_dump_data
    parsed = data_model.parse_raw(data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic\main.py", line 549, in pydantic.main.BaseModel.parse_raw
  File "C:\Python311\Lib\site-packages\tiktokapipy\models\raw_data.py", line 130, in parse_obj
    return super().parse_obj(obj)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic\main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for VideoResponse
ItemModule -> 7195422072552639771 -> music -> authorName
  field required (type=value_error.missing)
Traceback (most recent call last):
  File "C:\Python311\Lib\site-packages\tiktokapipy\api.py", line 363, in _scrape_data
    data = self._extract_and_dump_data(content, extras_json, data_model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\tiktokapipy\api.py", line 394, in _extract_and_dump_data
    parsed = data_model.parse_raw(data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic\main.py", line 549, in pydantic.main.BaseModel.parse_raw
  File "C:\Python311\Lib\site-packages\tiktokapipy\models\raw_data.py", line 130, in parse_obj
    return super().parse_obj(obj)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic\main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for VideoResponse
ItemModule -> 7195422072552639771 -> music -> authorName
  field required (type=value_error.missing)
Traceback (most recent call last):
  File "C:\xampp8\htdocs\tiktokupload\runs.py", line 94, in <module>
    fromchallage(keyword_value,length_value)
  File "C:\xampp8\htdocs\tiktokupload\fungsi.py", line 70, in fromchallage
    for video in challenge.videos:
  File "C:\Python311\Lib\site-packages\tiktokapipy\api.py", line 90, in __next__
    out = self.fetch(self._next_up)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\tiktokapipy\api.py", line 103, in fetch
    return self._api.video(video_link(self.light_models[idx].id))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\tiktokapipy\api.py", line 316, in video
    response, api_extras = self._scrape_data(
                           ^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\tiktokapipy\api.py", line 370, in _scrape_data
    raise TikTokAPIError(
tiktokapipy.TikTokAPIError: Data scraping unable to complete in 30.0s (retries: 1)

i try use:

• navigation_retries=1 /sometime i set to 5
• navigation_timeout=30 /sometime i set to 90
• also try use proxy, my proxy is working fine. before that i use my real ip meaning didn't use any proxy. after i use proxy in code, error still same after found a few data.

proxy setting:

proxy={
   "server": "http://my proxy ip here:8080",
   "username": "",
   "password": ""

Additional context
Yesterday and few days ago i use its working fine. no error. but starting today got error. can i know how to solve it?

Proxy settings not being used

Hi there,

I've added a proxy={"server":"http" ...} argument to the API but it's not actually sending the requests through the proxy.

Is there anything else I can do to test or ensure it's working through the proxy?

Thanks

[BUG] Scrolling is not working anymore

Describe the bug
It tries to scroll but page loads nothing, as if Tiktok had detected an automatism.
I tried Firefox & Chromium.

To Reproduce

from tiktokapipy.api import TikTokAPI
with TikTokAPI(scroll_down_time=10) as api:
        challenges = api.challenge("test")      

Expected behavior
It should scroll down

Version Information
0.1.11

Additional context
I add a similar problem when scraping TikTok with Selenium without an undetected webdriver and trying to scroll down.

[BUG] ValidationError: 1 validation error for VideoPage

not sure what causes it but i am seeing this alot

Failed: ValidationError: 1 validation error for VideoPage
itemInfo.itemStruct.music.playUrl
  Field required [type=missing, input_value={'authorName': '"removed"...itle': 'original sound'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.1.2/v/missing
Traceback (most recent call last):
  File "/home/user/TwitchBot/bot/modules/tiktok.py", line 27, in get_tiktoks
    async for video in user.videos:
  File "/home/user/.venv/lib/python3.10/site-packages/tiktokapipy/util/deferred_collectors.py", line 90, in __anext__
    await self._fetch_async()
  File "/home/user/.venv/lib/python3.10/site-packages/tiktokapipy/util/deferred_collectors.py", line 223, in _fetch_async
    self._collected_values.append(await self._api.video(video.id))
  File "/home/user/.venv/lib/python3.10/site-packages/tiktokapipy/async_api.py", line 163, in video
    response = VideoPage.model_validate(
  File "/home/user/.venv/lib/python3.10/site-packages/pydantic/main.py", line 480, in model_validate
    return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for VideoPage
itemInfo.itemStruct.music.playUrl
  Field required [type=missing, input_value={'authorName': '"removed"...itle': 'original sound'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.1.2/v/missing

"removed" after 'authorName': was me replacing the username with that the '...itle' is part of original output

[QUESTION] How to retrieve more than 30 videos

I have two needs now

• Get all videos by username
• Get all videos by tag

The code for the first case is as follows

        with TikTokAPI() as api:
            user = api.user("morifuntool", video_limit=2000)

            for video in user.videos:
                print(f"URL：https://www.tiktok.com/@morifuntool/video/{video.id},Time:{video.create_time}")

The video data can be obtained normally, but there are only 30 videos.
How can I get all videos?

The code for the second case is as follows

         with TikTokAPI() as api:
            challenge = api.challenge("OutfitIdeas")
            for video in challenge.videos:
                print(f"https://www.tiktok.com/@morifuntool/video/{video.id}")
                print(video.create_time)

The video data can be obtained normally, but there are only 40+ videos.
How can I get all videos?

[FEATURE REQUEST] Ability to Retrieve Liked Videos of Users

[BUG] Timeout with firefox and Emulate Mobile

Describe the bug
The only way that I could download comments it's when I use
with TikTokAPI(navigator_type="firefox",emulate_mobile=True) as api:
This code, it works when I run in my Macbook and VsCode
But, when I put this code in EC2 or other server, I get a timeout message and only works if I use emulate_mobile=False, but the problem with this config is I don't take the comments.
I checked all version os libs and everything looks the same
I checked firewall and other security issues in EC2, there´s nothing else to do.

Anyone has this issue?

To Reproduce
Steps to reproduce the behavior:
Run this code in EC2
with TikTokAPI(navigator_type="firefox",navigation_retries=1,navigation_timeout=0,emulate_mobile=True) as api: video = api.video("https://vm.tiktok.com/ZMYrkvoqy") frase = video.comments[0].text.capitalize() print(frase)

See the error below...
`Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/tiktokapipy/api.py", line 355, in _scrape_data
page.wait_for_selector("#SIGI_STATE", state="attached")
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 8213, in wait_for_selector
self._sync(
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 104, in _sync
return task.result()
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_page.py", line 364, in wait_for_selector
return await self._main_frame.wait_for_selector(**locals_to_params(locals()))
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 322, in wait_for_selector
await self._channel.send("waitForSelector", locals_to_params(locals()))
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 44, in send
return await self._connection.wrap_api_call(
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 419, in wrap_api_call
return await cb()
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 79, in inner_send
result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator("#SIGI_STATE")

Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/tiktokapipy/api.py", line 355, in _scrape_data
page.wait_for_selector("#SIGI_STATE", state="attached")
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 8213, in wait_for_selector
self._sync(
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 104, in _sync
return task.result()
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_page.py", line 364, in wait_for_selector
return await self._main_frame.wait_for_selector(**locals_to_params(locals()))
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 322, in wait_for_selector
await self._channel.send("waitForSelector", locals_to_params(locals()))
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 44, in send
return await self._connection.wrap_api_call(
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 419, in wrap_api_call
return await cb()
File "/home/ubuntu/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 79, in inner_send
result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator("#SIGI_STATE")

Traceback (most recent call last):
File "/home/ubuntu/bla.py", line 4, in
video = api.video("https://vm.tiktok.com/ZMYrkvoqy")
File "/home/ubuntu/.local/lib/python3.10/site-packages/tiktokapipy/api.py", line 316, in video
response, api_extras = self._scrape_data(
File "/home/ubuntu/.local/lib/python3.10/site-packages/tiktokapipy/api.py", line 370, in _scrape_data
raise TikTokAPIError(
tiktokapipy.TikTokAPIError: Data scraping unable to complete in 0.0s (retries: 1)
`

Expected behavior
Expect return json with video data and comments

Version Information
tiktokapipy==0.1.11
pydantic==1.10.4
playwright==1.30.0

Additional context
Nothing else..

Data scraping unable to complete in 30.0s

I'm trying to use this package to scrape user profiles and it returns the error below. Any reason why it's happening and how to fix it?

Traceback (most recent call last):
  File "/home/usr1/.local/lib/python3.8/site-packages/tiktokapipy/api.py", line 318, in _scrape_data
    page.wait_for_selector("#SIGI_STATE", state="attached")
  File "/home/usr1/.local/lib/python3.8/site-packages/playwright/sync_api/_generated.py", line 7991, in wait_for_selector
    self._sync(
  File "/home/usr1/.local/lib/python3.8/site-packages/playwright/_impl/_sync_base.py", line 104, in _sync
    return task.result()
  File "/home/usr1/.local/lib/python3.8/site-packages/playwright/_impl/_page.py", line 364, in wait_for_selector
    return await self._main_frame.wait_for_selector(**locals_to_params(locals()))
  File "/home/usr1/.local/lib/python3.8/site-packages/playwright/_impl/_frame.py", line 322, in wait_for_selector
    await self._channel.send("waitForSelector", locals_to_params(locals()))
  File "/home/usr1/.local/lib/python3.8/site-packages/playwright/_impl/_connection.py", line 44, in send
    return await self._connection.wrap_api_call(
  File "/home/usr1/.local/lib/python3.8/site-packages/playwright/_impl/_connection.py", line 419, in wrap_api_call
    return await cb()
  File "/home/usr1/.local/lib/python3.8/site-packages/playwright/_impl/_connection.py", line 79, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator("#SIGI_STATE")
============================================================
Traceback (most recent call last):
  File "main.py", line 40, in <module>
    user = api.user("andrew.araya", "3")
  File "/home/usr1/.local/lib/python3.8/site-packages/tiktokapipy/api.py", line 269, in user
    response, api_extras = self._scrape_data(link, self._user_response_type)
  File "/home/usr1/.local/lib/python3.8/site-packages/tiktokapipy/api.py", line 333, in _scrape_data
    raise TikTokAPIError(
tiktokapipy.TikTokAPIError: Data scraping unable to complete in 30.0s (retries: 0)

Here's my code:

from tiktokapipy.api import TikTokAPI

with TikTokAPI() as api:
    user = api.user("therock", "3")
    print(user)

Some video links are returning 'Forbidden'

Some video links are returning '403 Forbidden' when I try to open them. After sending one more request to get video information, it fixes but it wasting time.

Here is my code:

def get_tiktok_media(tiktok_username: str, max_count: int):
    try:
        result = []
        with TikTokAPI() as api:
            user = api.user(tiktok_username, max_count)
            for video in user.videos:
                result_video = video
                while result_video.video.play_addr.endswith("chain_token") and result_video.video.download_addr.endswith("chain_token"):
                    result_video = api.video(f"https://www.tiktok.com/@{video.author}/video/{result_video.id}")

                result.append(result_video)
        return result
    except Exception as e:
        return e

I realize patterns between working videos and not working videos.

Sample working video link:
https://v16-webapp.tiktok.com/612dc7b3bd400a2ccd84af58fdbba950/63d15716/video/tos/useast2a/tos-useast2a-pve-0037c001-aiso/o8DYgoPAliHy5aGSOe2Erh0KfZrfkOEQAHGZeC/?a=1988&ch=0&cr=0&dr=0&lr=tiktok&cd=0%7C0%7C1%7C0&cv=1&br=4260&bt=2130&cs=0&ds=3&ft=4b~OyMlh8Zmo0Cd6I64jVhLWypWrKsdm&mime_type=video_mp4&qs=0&rc=MzY7NmY4OGZnZjY8ZzxmZkBpam87NWU6ZnA6aTMzZjgzM0A0YGIyM182XjExLWExNDUvYSNhLmwzcjRnMTVgLS1kL2Nzcw%3D%3D&l=20230125102117B681AB00030843ACF23F&btag=80000

Sample not working video link:
https://v16-webapp-prime.tiktok.com/video/tos/useast2a/tos-useast2a-pve-0037-aiso/oEbqy7f2KjVLQQDyUec8GCmnBHD80oVhnWPAgD/?a=1988&ch=0&cr=0&dr=0&lr=tiktok&cd=0%7C0%7C1%7C0&cv=1&br=2520&bt=1260&cs=0&ds=3&ft=4fUEKMlh8Zmo03d6I64jVVO1ZpWrKsdm&mime_type=video_mp4&qs=0&rc=NmY4NjNlZDM3M2VnOGU3aUBpM3d2czg6ZnRkaTMzZjgzM0BfMl9gLl9hXzQxMzUxXzQzYSNzcF9kcjRnZzVgLS1kL2Nzcw%3D%3D&btag=80000&expire=1674663675&l=202301251021026E4C2DACC26490AAE1A6&ply_type=2&policy=2&signature=e303281836dfc4b2b8d8acfe4beb9963&tk=tt_chain_token

not working videos are ends with chain_token

I'll be happy if anybody can help me about this.

tiktokapipy version is 0.1.10.post1

Cannot grab comments

UserWarning: Was unable to collect comments. A second attempt might work.

from tiktokapipy.api import TikTokAPI

video_url = "https://www.tiktok.com/@lindamaittm/video/7183294139855850794"

with TikTokAPI(navigation_retries=2, navigation_timeout=10, scroll_down_time=60) as api:
    video = api.video(video_url)
    print(video.comments)

Is there a way to download video without tiktok watermark?

Hey, I've followed all the setup on downloading videos in the docs and it all works great! However, I can't figure out how to download the regular videos without watermark.

Slideshows only contain the raw images so those are completely fine, but after dumping the regular video content, I can't seem to find any address which does not contain the watermark.

Any insight?

[FEATURE REQUEST] Set unable to collect comments back to warnings.warn

Can we get this set back to warnings.warn so it stops flooding us with its message

 if not video.comments:
            print(
                "Was unable to collect comments.\n"
                "A second attempt or setting a nonzero value for scroll_down_time might work.",
                file=sys.stderr,
            )

[QUESTION] Scroll time on Docker

Ask your question
I tried containerizing my script with this package in Docker (Dockerfile below). When it runs, I am able to get user information back, but it seems that the scroll time is not taken into account? When I set a high scroll time running on my host locally, it returns all of a users videos, even if they have a lot. When running the same code on my container, it only returns a fraction of the data (first 30 videos). I am using the data_dump_file (I can see the file size is much smaller on the data file when running through Docker) Any ideas?

# Use an official Python runtime as a parent image
FROM python:3.10-slim

# Set the working directory to /app
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install any needed packages specified in requirements.txt
RUN pip install --trusted-host pypi.python.org -r requirements.txt

# Make port 80 available to the world outside this container
EXPOSE 80

# Install system dependencies for Playwright
RUN playwright install-deps

# Install Playwright browser dependencies
RUN python -m playwright install

# Run run.py when the container launches
CMD ["python", "run.py"]

[QUESTION]

C:\Users\Administrator\Documents\TikTokGrab>TikTokGrabAlphav0.2.py
Traceback (most recent call last):
File "C:\Users\Administrator\Documents\TikTokGrab\TikTokGrabAlphav0.2.py", line 1, in
from tiktokapipy.api import TikTokAPI
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python311\Lib\site-packages\tiktokapipy\api.py", line 32, in
from tiktokapipy.models import DeferredIterator, TikTokDataModel
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python311\Lib\site-packages\tiktokapipy\models_init_.py", line 24, in
from pydantic.main import ModelMetaclass
ImportError: cannot import name 'ModelMetaclass' from 'pydantic.main' (C:\Users\Administrator\AppData\Local\Programs\Python\Python311\Lib\site-packages\pydantic\main.py)

The code was working great until I installed your 0.2 version of the api to be able to scrap more user's videos.
I had an issue with it so I get back to the previous version of the api. But now I have errors with pydantic. I tried to uninstall it and the api too. But still the error occurs. Do you have any idea how to fix ?

Tks

Allow selection of exactly which parts of a model need to be retrieved

I'm trying to write a program to give me desktop notifications when someone I'm following posts a new TikTok, which means I need to fetch a lot of users and videos and it takes around 10 minutes to search through them all meaning it would be significantly quicker to just do it manually. But I think a lot of time is spent retrieving data I'm not interested in:

• I already have all of the data needed to produce the User model, because it is given by the same endpoint I'm using to fetch the users I'm following, but in order for TikTokPy to retrieve their videos, it seems I can only give TikTokPy the unique ID and it fetches the rest of the information again itself. Most of this information isn't even needed for retrieving the videos!
• I only need the ID for each video to identify videos I've already watched and to produce a link for me to watch it, but the only way I know to obtain the list of videos is iterating over User.videos and doing so also retrieves information such as the video's stats, comments, description and tags.

I'd imagine that if I was able to restrict TikTokPy into only retrieving the data I need, it would significantly improve the speed at which I can find new videos, and I doubt this is some niche case that only applies to me because there's few examples I can think of where a program would need all of the available data.

[QUESTION] Video Downloading Error 403

Cannot download any video by down_addr from challenge collection. It returns 403 Forbidden error

for video in challenge.videos
requests.get(video.video.download_addr)

How to fix that?

Cannot get createTime of the video

I use def get_challenge_videos_scroll_down() challenge by tag_name and can not get createTime of the video

output :

Update attribute scroll_down_time

Hello,

I try to get the stats of the videos corresponding to a challenge, for that I use the option to scroll in order to have all my videos for that challenge. It works very well but for each video in the for loop, it also scrolls for X seconds (30 seconds here) so it takes a long time.

with TikTokAPI(scroll_down_time=30) as api:
    challenge = api.challenge('fyp')
    # challenge.scroll_down_time = 0 ?
    for video in challenge.videos:
            play_count = video.stats.play_count

Is there a way to update the scroll_down_time value or an alternative idea?

[BUG] ModuleNotFoundError: No module named 'pydantic._internal'

Getting this with version 0.2.0.post1

from tiktokapipy.async_api import AsyncTikTokAPI

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/django-test/.venv/lib/python3.10/site-packages/tiktokapipy/async_api.py", line 14, in <module>
    from tiktokapipy.api import TikTokAPI
  File "/home/user/django-test/.venv/lib/python3.10/site-packages/tiktokapipy/api.py", line 15, in <module>
    from tiktokapipy.models.challenge import Challenge
  File "/home/user/django-test/.venv/lib/python3.10/site-packages/tiktokapipy/models/__init__.py", line 12, in <module>
    from pydantic._internal._model_construction import ModelMetaclass
ModuleNotFoundError: No module named 'pydantic._internal'

pip freeze versions
pydantic==1.10.4 tiktokapipy==0.2.0.post1

Empty response on User stats

Hello Russell-Newton,

Thank you for your work. I am trying to use TikTokPy for stats. Unfortunately, it doesn't seem to receive any answers.
Maybe I am using the function UserStats wrong?! I just want stats of all videos of one user. Thx!

from tiktokapipy.api import TikTokAPI
result = []

def get_stats():
with TikTokAPI() as api:
user = api.user(rihanna)
for video in user.videos:
num_comments = video.stats.comment_count
num_likes = video.stats.digg_count
num_views = video.stats.play_count
num_shares = video.stats.share_count
result.append(num_comments,num_likes,num_views,num_shares)
return result

[QUESTION] Access Denied Error When Downloading Video

Hello. I'm certain this is a simple issue of just passing through my s_v_web_id and tt_chain_token, but I'm having trouble doing so.

Basically the code runs without error, but I get an Access Denied error when trying to download a TikTok Video. What's the work around here?

import io
import glob
import os
import urllib.request
from os import path

import aiohttp
from tiktokapipy.async_api import AsyncTikTokAPI
from tiktokapipy.models.video import Video

link = "https://www.tiktok.com/@prestoneckhardt/video/7203852306889133358"
directory = r"C:\Users\peckh\Desktop\ECKHARDT CAPITAL LLC"

async def save_video(video: Video):
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0;Win64) AppleWebkit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36'}
    async with aiohttp.ClientSession() as session:
        async with session.get(video.video.download_addr, headers=headers) as resp:
            data = await resp.read()
            print(data)  # print the response data
            with open(os.path.join(directory, f'{video.id}.mp4'), 'wb') as f:
                f.write(data)
            print(f"Video {video.id} has been downloaded.")

async def download_video():
    async with AsyncTikTokAPI() as api:
        video: Video = await api.video(link)
        if video.image_post:
            downloaded = await save_slideshow(video)
        else:
            downloaded = await save_video(video)

asyncio.run(download_video())```

Is there a way to ignore slideshows or Error in video extraction: status code 100004?

I believe I'm getting and error because I'm trying to extract data from a slideshow. I know I could utilize emulate_mobile=True, however this severely limits the number of videos I am able to pull data on. Traceback on the error is below:

Traceback (most recent call last):
  File "~/myproject.py", line 52, in <module>
    for video in challenge.videos:
  File "~/venv/lib/python3.10/site-packages/tiktokapipy/api.py", line 90, in __next__
    out = self.fetch(self._next_up)
  File "~/venv/lib/python3.10/site-packages/tiktokapipy/api.py", line 103, in fetch
    return self._api.video(video_link(self._light_models[idx].id))
  File "~/venv/lib/python3.10/site-packages/tiktokapipy/api.py", line 312, in video
    return self._extract_video_from_response(response, api_extras)
  File "~/venv/lib/python3.10/site-packages/tiktokapipy/api.py", line 450, in _extract_video_from_response
    raise TikTokAPIError(
tiktokapipy.TikTokAPIError: Error in video extraction: status code 100004

I can't seem to figure out where to put my except: statement since even calling the video in the for videos in challenge.videos causes the error, so I'm not sure how to move to the next iteration.

Any help is appreciated! Again, I apologize if there is a simple solution I am just missing.

Filter by view count

Is it possible to select videos based on the number of views in a Challenge? I want to find videos which is not popular .

Comments no loading on both chromium and firefox

It seems comments are not loading again, for both firefox and chromium navigator types.
Only confirmed locally on windows wsl environment if it makes any difference.

new feature: help add proxy

hi, due to region limitations, some ppl have to visit tiktok with a network proxy,
could you help add proxy while navigating tiktok
thanx

[QUESTION] How to retrieve more than 10 videos ?

Hello,

pip freeze | findstr tiktokapipy tiktokapipy==0.2.0
pip freeze | findstr pydantic pydantic==2.0.1 pydantic_core==2.0.2

I just switched to v 0.2 but I can't return more than 10 videos, please how can I return all user's videos ?

from tiktokapipy.api import TikTokAPI

# Fonction pour récupérer les vidéos d'un compte TikTok spécifié
def get_videos(username):
    videos = []
    with TikTokAPI() as api:
        user = api.user(username)
        for video in user.videos:
            videos.append({
                "video_id": video.id,
                "views": video.stats.play_count,
                "likes": video.stats.digg_count
            })
    return videos

# Fonction pour calculer l'engagement d'une vidéo (pourcentage de likes par rapport au nombre de vues)
def calculate_engagement(video):
    if video["views"] > 0:
        engagement = (video["likes"] / video["views"]) * 100
    else:
        engagement = 0
    return engagement

username = ""  

# Récupérer les vidéos du compte TikTok spécifié avec les nombres de vues et de likes
videos = get_videos(username)

# Calculer l'engagement de chaque vidéo et ajouter la valeur dans un nouveau champ "engagement"
for video in videos:
    video["engagement"] = calculate_engagement(video)

# Trier les vidéos par engagement (du plus élevé au plus faible)
videos_sorted = sorted(videos, key=lambda x: x["engagement"], reverse=True)

# Afficher le pourcentage d'engagement et le lien vers les vidéos classées par engagement
for video in videos_sorted:
    engagement = video["engagement"]
    video_id = video["video_id"]
    link = f"https://www.tiktok.com/@{username}/video/{video_id}"
    print(f"Engagement: {engagement}%")
    print(f"Video Link: {link}")
    print()

I tested several users and it returns 10 videos, what's wrong with my code ?
Thanks

how to get the basic video info LightVideo from user.videos and avoid query other time consuming data

hi,
how can I get the basic video info LightVideo from user.videos and avoid query data in Video.
I found comment data easy to throw exceptions and time-consuming.

Maybe I can firstly get the basic video id set of a user and decide what I want next steps. Then query other video details by the video id I have got.

[QUESTION] Data scraping unable to complete in 30.0s

Hi to all
Until yesteday this piece of code was working well:

_with TikTokAPI() as api:
    user = api.user(username)
    for video in user.videos:_

but now i'm always received a error message:
Data scraping unable to complete in 30.0s (retries: 0)

Do you have any idea? Is possible TikTok change something in his webpage?
I've modified the timeout to 300s and continues the same error.

K.R.

Suppressing certain warnings and errors

Is there an easy way to suppress some of the warnings related to the comments not loading? I am not trying to collect any comments, so suppressing these would make things easier.

For instance, when I get the following when I start using the API:

site-packages/tiktokapipy/api.py:470: UserWarning: Was unable to collect comments.
A second attempt might work.
  warnings.warn(

Additionally, I get the following for some videos:

  File "pydantic/main.py", line 527, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 19 validation errors for APIResponse
comments -> 0 -> id
  field required (type=value_error.missing)
comments -> 1 -> id
  field required (type=value_error.missing)
comments -> 2 -> id
  field required (type=value_error.missing)

Any suggestions would be appreciated! Sorry if this is obvious to others.

[BUG] Unable to do any kind of request

Describe the bug
I am unable to do any kind of request. This error pops up:
tiktokapipy.TikTokAPIError: Data scraping unable to complete in 30.0s (retries: 0)

To Reproduce
Here is the code you can run:

from tiktokapipy.api import TikTokAPI 

with TikTokAPI() as api:
    video = api.video('https://www.tiktok.com/@ahormozi/video/7242371997324315946')
    print(video)

Expected behavior
print out the scraped Video object

Version Information
pydantic==1.10.9
playwright==1.35.0
playwright==1.35.0
tiktokapipy==0.1.13.post1
Please include what versions of pydantic, playwright, and tiktokapipy you have installed (can be found with pip freeze).

Additional context
Here is the full traceback:

Traceback (most recent call last):
File "C:\Users\path\venv\lib\site-packages\tiktokapipy\api.py", line 430, in _scrape_data
data = self._extract_and_dump_data(content, extras_json, data_model)
File "C:\Users\path\venv\lib\site-packages\tiktokapipy\api.py", line 469, in _extract_and_dump_data
parsed = data_model.parse_raw(data)
File "pydantic\main.py", line 549, in pydantic.main.BaseModel.parse_raw
File "C:\Users\path\venv\lib\site-packages\tiktokapipy\models\raw_data.py", line 130, in parse_obj
return super().parse_obj(obj)
File "pydantic\main.py", line 526, in pydantic.main.BaseModel.parse_obj
File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for VideoResponse
VideoPage
field required (type=value_error.missing)
Traceback (most recent call last):
File "c:\Users\path\main.py", line 11, in
video = api.video('https://www.tiktok.com/@ahormozi/video/7242371997324315946')
File "C:\Users\path\venv\lib\site-packages\tiktokapipy\api.py", line 348, in video
response, api_extras = self._scrape_data(
File "C:\Users\path\venv\lib\site-packages\tiktokapipy\api.py", line 445, in _scrape_data
raise TikTokAPIError(
tiktokapipy.TikTokAPIError: Data scraping unable to complete in 30.0s (retries: 0)

How to use it

Love your dedication but i don't know how to use it, can you write a guide to use it. I'm try to collect data for study only. Thanks a lot

Json decode error

Getting this error with the 0.1.6 release

Exception in callback AsyncIOEventEmitter._emit_run.<locals>.callback(<Task finishe... 1 (char 0)')>) at /home/user/.local/lib/python3.10/site-packages/pyee/asyncio.py:65
handle: <Handle AsyncIOEventEmitter._emit_run.<locals>.callback(<Task finishe... 1 (char 0)')>) at /home/user/.local/lib/python3.10/site-packages/pyee/asyncio.py:65>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/user/.local/lib/python3.10/site-packages/pyee/asyncio.py", line 71, in callback
    self.emit("error", exc)
  File "/home/user/.local/lib/python3.10/site-packages/pyee/base.py", line 179, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/home/user/.local/lib/python3.10/site-packages/pyee/base.py", line 139, in _emit_handle_potential_error
    raise error
  File "/home/user/.local/lib/python3.10/site-packages/playwright/_impl/_page.py", line 255, in _on_route
    handled = await route_handler.handle(route)
  File "/home/user/.local/lib/python3.10/site-packages/playwright/_impl/_helper.py", line 287, in handle
    [handled, *_] = await asyncio.gather(handled_future, *handler_task)
  File "/home/user/.local/lib/python3.10/site-packages/tiktokapipy/async_api.py", line 146, in capture_api_extras
    _data = await response.json()
  File "/home/user/.local/lib/python3.10/site-packages/playwright/async_api/_generated.py", line 16965, in json
    return mapping.from_maybe_impl(await self._impl_obj.json())
  File "/home/user/.local/lib/python3.10/site-packages/playwright/_impl/_fetch.py", line 467, in json
    return json.loads(content)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

code thats giving the error with real username removed:

async def tester():
     async with AsyncTikTokAPI() as api:
             user = await api.user("some_user", video_limit=1)
             async for video in user.videos:
                     print(video)

asyncio.run(tester())

video info is being printed

Add tiktok url to the video object

Is your feature request related to a problem? Please describe.
Since we are not able to download the videos from this library, at least I want to use it to fetch the tiktok url of a user profile and then use a website to download that tiktok link.

Describe the solution you'd like
For me, it would be useful to add some sort of "url" param to the user.videos.video

Describe alternatives you've considered
Tried requesting url from the play_addr, but as mentioned in other issues, it's not possible to get that url

[TODO] - Make more API calls, load less pages

At the moment, TikTokPy loads TikTok's website to grab data. This circumvents the need for a verify_fp cookie, but it increases runtime and hinders scalability.

Changes could be made such that API calls are made directly instead of loading pages. A page should only be loaded completely when the API call doesn't work. This could reload the cookies necessary to make future API calls.

Steps towards accomplishing goal:

1. Identify TikTok API calls necessary to get the information currently expected by TikTokPy
2. Migrate TikTokPy calls to using API calls unless overridden manually
3. Create automatic override determined by cookie staleness

get data by username

hello can i have an example of code to get by username number of likes shares comments of each video?

Data is the same

I have been testing this API and notice that all the data returned is the same.
I have been playing around with the challenge method at different periods of time and I notice that when you use a hashtag to gather some data and you use it a second time, it will return the same data as the first time regardless of the time lapse between the 2 requests.

Just in case, you can find the code attached below.

    def hashtags(self, hashtag: str='funny', video_count: int=10):
        """
        Retrun information about a TikTok hashtag
        including video
        
        hashtag: str
        video_count: int
        """
        now = datetime.now()
        current_time = now.strftime(f"%H_%M ({datetime.today().strftime('%Y-%m-%d')})")
        
        with TikTokAPI() as api:
            data = {}
            challenge = api.challenge(hashtag, video_limit=video_count, )
            
             # Evaluate if the user has a folder in the database_json folder
            if(not os.path.isdir(f"Backend/hashtag_database_json/{challenge.title}")):
                print(os.path.exists(f"Backend/hashtag_database_json/{challenge.title}"))
                os.makedirs(f'Backend/hashtag_database_json/{challenge.title}')
                print(f"Hashtag {challenge.title} has been created in the DB")
            
            for video in challenge.videos:
                
                #Store all the hashtags used in the video
                video_hashtags = []
                for i in video.challenges:
                    #Append all the challenges to the list
                    video_hashtags.append(i.title)
                    
                    
                data[video.id] = {
                    "likes":video.stats.digg_count,
                    "shares": video.stats.share_count,
                    "comments": video.stats.comment_count,
                    "play_count": video.stats.play_count,
                    "description": video.desc,
                    
                    "video_info": {
                        "height": video.video.height,
                        "width": video.video.width,
                        "duration": video.video.duration,
                    },
                    
                    "hashtags": video_hashtags,
                    
                    "music_info": {
                        "title": video.music.title,
                        "author_name": video.music.author_name,
                        "original_audio": video.music.original,
                    }
                }
                
                print(video_hashtags)
                
                
                with open(f"Backend/hashtag_database_json/{challenge.title}/{current_time}.json", 'w') as f:
                    json.dump(data, f, indent=4)
                    

[QUESTION] how to use LightVideo

Ask your question
hi,
please help to answer how to use LightVideo. In version 0.1.10, was using in this way

'' lights = user.videos._light_models
for v in lights:
json = v.json()
"
when I replace _light_models with LightVideo, it is not working, how to use LightVideo?thanks

Additional context
Add any other context or screenshots about the question here.

[Feature Request] Ability to Retrieve Following/Followed Users

[QUESTION] How to retrieve more than 30 videos from a user

Ask your question
I am pulling video data by passing through a username. The data that returns is what I want, however it only retrieves the user's 30 most recent videos. I see there is a video_limit parameter, but this seems to be invoked after the response is returned. I tried setting scrolling_down_time higher, but no matter how high, it still returns only 30 videos. The data response gets saved to a json how I want it to, so the only issue is this 30 video limit. Any help would be appreciated, thanks!

Here is my function

def get_user_video_data(username: str) -> None:
   
    filepath = 'some/file/path'
    with TikTokAPI(scroll_down_time=100,navigation_retries=5, navigation_timeout=0, 
                   data_dump_file=filepath) as api:
        user = api.user(username)

[BUG] Unable to collect comments.

There is a warning about being unable to collect comments, is there any way to solve it? Thanks!
Warning is given as follows:

D:\Python\envs\tiktok\lib\site-packages\tiktokapipy\api.py:478: UserWarning: Was unable to collect comments.
A second attempt might work.
warnings.warn(

Unable to fetch video comments

I want to get the comments of the video, but the data obtained by the following code is empty

from tiktokapipy.api import TikTokAPI
def do_something():
    with TikTokAPI(navigator_type = 'chromium') as api:
        video = api.video("https://www.tiktok.com/@sstowen/video/7232788728538844459")
        comments = video.comments
        comment_text = [comment.text for comment in comments]
        print("Comments: ", comment_text)

do_something()

console print result

A second attempt might work.
  video = api.video("https://www.tiktok.com/@sstowen/video/7232788728538844459")
Comments:  []

I checked the related issues, but I didn't find a suitable answer, I hope to get your help

first run error

got this error on first run
async_api.py:157: RuntimeWarning: coroutine 'BrowserContext.clear_cookies' was never awaited
2nd time it ran without error

Tiktok Is Blocking Downloads

Awesome library!

I wanted to ask if you were aware of Tiktok blocking downloads. I started getting the error as of Feb 10, 2023. It looks like Tiktok has added an extra layer of security for downloading videos. Have you been able reproduce this problem as well?

Error:

<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don\'t have permission to access "http&#58;&#47;&#47;v16&#45;webapp&#45;prime&#46;us&#46;tiktok&#46;com&#47;video&#47;tos&#47;useast5&#47;tos&#45;useast5&#45;ve&#45;0068c004&#45;tx&#47;9135085813b14270a32e64aecff14792&#47;&#63;" on this server.<P>\nReference&#32;&#35;18&#46;17f5df17&#46;1676300641&#46;4b789411\n</BODY>\n</HTML>\n

Code:

async def save_video(video):
    async with aiohttp.ClientSession() as session:
        async with session.get(video.video.download_addr) as resp:
            return io.BytesIO(await resp.read())

with TikTokAPI() as api:
	video = api.video(url)

bytes = asyncio.run(save_video(video))

How to save videos

Hey,

im new to webscraping so kind of a noob question i guess, but how to handle the saving of videos?
Tried several packages but they didnt work.

[Feature Request] add no watermark function to video

case:

video url:
https://www.tiktok.com/@skynews/video/7192867272225000710

watermark url:
https://v16m-default.akamaized.net/b7b2ea9125bb68666109897fa7215709/63d2c761/video/tos/useast2a/tos-useast2a-pve-0068/oUIAhfg2Q9LtDB6owxVHAqkSOjICUEgUzRUSho/?a=0&ch=0&cr=0&dr=0&lr=all&cd=0%7C0%7C0%7C0&cv=1&br=1260&bt=630&cs=0&ds=3&ft=XE5bCqT0mv_PD12~CrTq3wUjZ.9VMeF~OD&mime_type=video_mp4&qs=0&rc=NDk0OmY0PDg2Zzk7OTY8O0BpM2xsaTQ6Zjg2aTMzNzczM0A2NDU2Y14uNTUxYzQzXzQ1YSNpLV5ocjRvLjZgLS1kMTZzcw%3D%3D&l=20230126123208C632DE2B1997DC337123&btag=80000

no watermark url:
https://v16m-default.akamaized.net/492d9274c60b431520dcf7123a96fc58/63d2c761/video/tos/useast2a/tos-useast2a-ve-0068c001/oUgUoG6IoUuMgXxq9fRLC7hDSzBstWASIQAh3k/?a=0&ch=0&cr=0&dr=0&lr=all&cd=0%7C0%7C0%7C0&cv=1&br=1108&bt=554&cs=0&ds=6&ft=XE5bCqT0mv_PD12~CrTq3wUjZ.9VMeF~OD&mime_type=video_mp4&qs=0&rc=OWc6ZzY4OjQ0PDM6NmQ6OkBpM2xsaTQ6Zjg2aTMzNzczM0AuYjYuXjEwNV8xNS0zLTVjYSNpLV5ocjRvLjZgLS1kMTZzcw%3D%3D&l=20230126123208C632DE2B1997DC337123&btag=80000

functions reference:
https://github.com/Evil0ctal/Douyin_TikTok_Download_API
API try here:
https://api.tikhub.io/docs
https://api.tikhub.io/docs#/TikTok/tiktok_video_data_tiktok_video_data__get
the route : /tiktok_video_data
noted: api in https://api.tikhub.io/docs is v3.2.9 which is not opensource, the https://github.com/Evil0ctal/Douyin_TikTok_Download_API is opensource and the version is 3.1.2, this version has the no wartermark function.

How to serialise Pydantic data to JSON?

Hi, I'm trying this code but I'm getting an error:

from tiktokapipy.api import TikTokAPI

if __name__ == '__main__':
    with TikTokAPI(
            scroll_down_time=2.5, navigation_retries=2, navigation_timeout=10
    ) as api:        
        challenge = api.challenge("foo")
        for video in challenge.videos.sorted_by(lambda vid: vid.create_time):
            with open(f"videos/{video.id}.json", "w") as file:
                file.write(video.json())

File "pydantic/json.py", line 90, in pydantic.json.pydantic_encoder
TypeError: Object of type 'LightUserGetter' is not JSON serializable

Any suggestions would be helpful, thanks!