bellingcat / tiktok-hashtag-analysis Goto Github PK

View Code? Open in Web Editor NEW

293.0 11.0 38.0 205 KB

Provides tools to analyze hashtags within posts scraped from TikTok.

Home Page: https://pypi.org/project/tiktok-hashtag-analysis/

License: MIT License

Python 97.05% Dockerfile 0.88% Shell 2.08%

tiktok command-line python scraper social-network open-source-research

tiktok-hashtag-analysis's People

Contributors

Stargazers

Watchers

tiktok-hashtag-analysis's Issues

Started is not a json file issue

I have been having an issue after installing tiktok-scraper

if I enter:
python run_downloader.py -t london paris newyork -p

output:
Hashtags to scrape: ['london', 'paris', 'newyork']
Something's wrong with what is returned by tiktok-scraper for the hashtag london - Started is not a json file.

tiktok-scraper returned - TikTok Scraper Started

Something's wrong with what is returned by tiktok-scraper for the hashtag paris - Started is not a json file.

tiktok-scraper returned - TikTok Scraper Started

Something's wrong with what is returned by tiktok-scraper for the hashtag newyork - Started is not a json file.

tiktok-scraper returned - TikTok Scraper Started

when I installed tiktok scraper - there were no issues with its install. I was wondering what the potential issue could be?
Please note that I am using a windows pc
Node v16.17.1 is installed
Python 3.10.7 is also installed

Thanks for the help!

for the sake of clarity - here was my output when I installed tiktok-scraper

npm i -g tiktok-scraper

output:
npm WARN deprecated [email protected]: request-promise has been deprecated because it extends the now deprecated request package, see request/request#3142
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see request/request#3142

changed 270 packages, and audited 271 packages in 13s

15 packages are looking for funding
run npm fund for details

found 0 vulnerabilities

JSON data that does not match the column and there is data that exceeds the declared row.

I tried to retrieve data using a certain hastag on tiktok (in this case is "ngertikanrasanya"), after transforming the data retrieved (json to csv script in the image). Some data does not match the column name and there is data that exceeds the declared row as shown in the image.

CSV Data :

Script Transform JSON to CSV :

Thank you in advance.

Basically, cannot run the script on Windows 10

I installed everything by following the instructions (https://github.com/bellingcat/tiktok-hashtag-analysis#pre-requisites).
Tried different commands to launch but none of them work. Throws either "no such directory" error or KeyError(key)
KeyError: 'formatters'

I am using Python 3.10.5
Below I add what the console displays.

    (tiktok) PS C:\Users\Arseniy\AppData\Local\Programs\Python\Python310\Scripts> python -m tiktok_hashtag_analysis --help
    Traceback (most recent call last):
      File "C:\Users\Arseniy\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "C:\Users\Arseniy\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "D:\some python scripts\tiktok\lib\site-packages\tiktok_hashtag_analysis\__main__.py", line 2, in <module>
        from .file_methods import log_writer
      File "D:\some python scripts\tiktok\lib\site-packages\tiktok_hashtag_analysis\file_methods.py", line 14, in <module>
        logging.config.fileConfig(path.join(path.dirname(path.abspath(__file__)), 'logging.config'))
      File "C:\Users\Arseniy\AppData\Local\Programs\Python\Python310\lib\logging\config.py", line 72, in fileConfig
        formatters = _create_formatters(cp)
      File "C:\Users\Arseniy\AppData\Local\Programs\Python\Python310\lib\logging\config.py", line 105, in _create_formatters
        flist = cp["formatters"]["keys"]
      File "C:\Users\Arseniy\AppData\Local\Programs\Python\Python310\lib\configparser.py", line 964, in __getitem__
        raise KeyError(key)
    KeyError: 'formatters'

(Here I am trying to run the script directly from the place where Python is installed on the computer
C:\Users\Arseniy\AppData\Local\Programs\Python\Python310)

Maybe any ideas or recommendations why it's not working and how to get it to work?
Maybe it makes sense to try to start via WSL (Windows Subsystem for Linux)

Unhandled exception when retrying in headed mode

Reported through Discord

The user invoked tiktok-hashtag-analysis notcoin --number 20 --plot which should not obviously cause any issues.

Currently, if TikTok returns no results when retrying in headed it raises the `EmptyResponseException'. This exception is not handled and the user gets a large traceback.

It would be good if this error was handled, this is the relevant code block:

tiktok-hashtag-analysis/tiktok_hashtag_analysis/base.py

Lines 180 to 191 in 2a4af67

 # Attempt to be robust against TikTok's countermeasures for headless browsing 

 try: 

 fetched_data = asyncio.run( 

 _fetch_hashtag_data(hashtag=hashtag, limit=limit, headed=headed) 

 ) 

 except Exception as e: 

 logger.warning( 

 f"Encountered error {e} when fetching data, retrying in headed mode" 

 ) 

 fetched_data = asyncio.run( 

 _fetch_hashtag_data(hashtag=hashtag, limit=limit, headed=True) 

 )

Because these TikTok returned an empty response errors seem to be fairly frequent, I wonder if it would be good to rework the error message to give the user more context about why it happens.

TikTokApi response breaks tool

The TikTok API initially returns a Warning, and then Errors with an unexpected status code 10201, resulting in no posts found:

2024-02-19 11:22:02 WARNING | Encountered error None -> TikTok returned an empty response when fetching data, retrying in headed mode

2024-02-19 11:22:12,695 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '###'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}

2024-02-19 11:22:12,695 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '###'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}

2024-02-19 11:22:12 ERROR | Got an unexpected status code: {'log_pb': {'impr_id': '###'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}

2024-02-19 11:22:12 WARNING | No posts were found for the hashtag: winter

2024-02-19 11:22:12 INFO | Scraped 0 new posts containing the hashtag 'winter', with 0 posts previously scraped

The None -> TikTok returned an empty response error is discussed variously here: davidteather/TikTok-Api#1090 davidteather/TikTok-Api#1097 davidteather/TikTok-Api#1104

Error 10201 is discussed here and may be "invalid entity or invalid parameter". Another post suggests it is related to CAPTCHA.

This is probably the same error as davidteather/TikTok-Api#1069 but running in headed mode (which happens automatically in this tool) does not seem to solve the issue.

Limiting number of video downloads

It doesn't appear there is a built in function to limit the number of video downloads. I've been looking through the code. Where would be a good place to put a limiter or is it there and I missed it?

Never received >887 posts in initial run of analysis script

Searched 33 hashtags and the first time the analysis script ran, it never scraped more than 887 posts:

2024-04-29 16:03:49 INFO | Scraped 887 new posts containing the hashtag 'encampment', with 0 posts previously scraped

Attached is an abridged log file of the initial few runs of the script. Filing this ticket per request from @GalenReich.
first_tiktok_scrape.log

Can't Install on macOS Monterey, M1 chip, node v19

Seems there's a lot going on in this situation. Any chance to fix it and start using this scraper?

npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: request-promise has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: Use your platform's native performance.now() and performance.timeOrigin.
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
npm ERR! code 1
npm ERR! path /opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas
npm ERR! command failed
npm ERR! command sh -c node-pre-gyp install --fallback-to-build --update-binary
npm ERR! Failed to execute '/opt/homebrew/Cellar/node/19.4.0_1/bin/node /opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --update-binary --module=/opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas/build/Release/canvas.node --module_name=canvas --module_path=/opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas/build/Release --napi_version=8 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v111' (1)
npm ERR! node-pre-gyp info it worked if it ends with ok
npm ERR! node-pre-gyp info using [email protected]
npm ERR! node-pre-gyp info using [email protected] | darwin | arm64
npm ERR! node-pre-gyp http GET https://github.com/Automattic/node-canvas/releases/download/v2.11.0/canvas-v2.11.0-node-v111-darwin-unknown-arm64.tar.gz
npm ERR! node-pre-gyp ERR! install response status 404 Not Found on https://github.com/Automattic/node-canvas/releases/download/v2.11.0/canvas-v2.11.0-node-v111-darwin-unknown-arm64.tar.gz 
npm ERR! node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v111 ABI, unknown) (falling back to source compile with node-gyp) 
npm ERR! node-pre-gyp WARN Hit error response status 404 Not Found on https://github.com/Automattic/node-canvas/releases/download/v2.11.0/canvas-v2.11.0-node-v111-darwin-unknown-arm64.tar.gz 
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | darwin | arm64
npm ERR! gyp info ok 
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | darwin | arm64
npm ERR! gyp info find Python using Python version 3.9.7 found at "/Users/edwardpan/opt/anaconda3/bin/python3"
npm ERR! gyp info spawn /Users/edwardpan/opt/anaconda3/bin/python3
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   '/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
npm ERR! gyp info spawn args   'binding.gyp',
npm ERR! gyp info spawn args   '-f',
npm ERR! gyp info spawn args   'make',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas/build/config.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/Users/edwardpan/Library/Caches/node-gyp/19.4.0/include/node/common.gypi',
npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
npm ERR! gyp info spawn args   '-Dvisibility=default',
npm ERR! gyp info spawn args   '-Dnode_root_dir=/Users/edwardpan/Library/Caches/node-gyp/19.4.0',
npm ERR! gyp info spawn args   '-Dnode_gyp_dir=/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp',
npm ERR! gyp info spawn args   '-Dnode_lib_file=/Users/edwardpan/Library/Caches/node-gyp/19.4.0/<(target_arch)/node.lib',
npm ERR! gyp info spawn args   '-Dmodule_root_dir=/opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas',
npm ERR! gyp info spawn args   '-Dnode_engine=v8',
npm ERR! gyp info spawn args   '--depth=.',
npm ERR! gyp info spawn args   '--no-parallel',
npm ERR! gyp info spawn args   '--generator-output',
npm ERR! gyp info spawn args   'build',
npm ERR! gyp info spawn args   '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! /bin/sh: pkg-config: command not found
npm ERR! gyp: Call to 'pkg-config pixman-1 --libs' returned exit status 127 while in binding.gyp. while trying to load binding.gyp
npm ERR! gyp ERR! configure error 
npm ERR! gyp ERR! stack Error: `gyp` failed with exit code: 1
npm ERR! gyp ERR! stack     at ChildProcess.onCpExit (/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:325:16)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:513:28)
npm ERR! gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:293:12)
npm ERR! gyp ERR! System Darwin 21.6.0
npm ERR! gyp ERR! command "/opt/homebrew/Cellar/node/19.4.0_1/bin/node" "/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--update-binary" "--module=/opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas/build/Release/canvas.node" "--module_name=canvas" "--module_path=/opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas/build/Release" "--napi_version=8" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v111"
npm ERR! gyp ERR! cwd /opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas
npm ERR! gyp ERR! node -v v19.4.0
npm ERR! gyp ERR! node-gyp -v v9.3.0
npm ERR! gyp ERR! not ok 
npm ERR! node-pre-gyp ERR! build error 
npm ERR! node-pre-gyp ERR! stack Error: Failed to execute '/opt/homebrew/Cellar/node/19.4.0_1/bin/node /opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --update-binary --module=/opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas/build/Release/canvas.node --module_name=canvas --module_path=/opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas/build/Release --napi_version=8 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v111' (1)
npm ERR! node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/@mapbox/node-pre-gyp/lib/util/compile.js:89:23)
npm ERR! node-pre-gyp ERR! stack     at ChildProcess.emit (node:events:513:28)
npm ERR! node-pre-gyp ERR! stack     at maybeClose (node:internal/child_process:1098:16)
npm ERR! node-pre-gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:304:5)
npm ERR! node-pre-gyp ERR! System Darwin 21.6.0
npm ERR! node-pre-gyp ERR! command "/opt/homebrew/Cellar/node/19.4.0_1/bin/node" "/opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build" "--update-binary"
npm ERR! node-pre-gyp ERR! cwd /opt/homebrew/lib/node_modules/tiktok-scraper/node_modules/canvas
npm ERR! node-pre-gyp ERR! node -v v19.4.0
npm ERR! node-pre-gyp ERR! node-pre-gyp -v v1.0.10
npm ERR! node-pre-gyp ERR! not ok

/bin/sh: tiktok-scraper: command not found

After trying to execute a search

python3 run_downloader.py -t #hashtag -p

it returned

/bin/sh: tiktok-scraper: command not found.

Wondering how I can troubleshoot it.

Additional analyses

There's currently one analysis included in this tool: aggregating hashtags common to a specified hashtag. There are likely other useful analyses that researchers and journalists are interested in, including

Generating network graph of related hashtags (e.g. nodes are hashtags and edge weight between nodes is the number of posts that contain both hashtags)

If you encounter other interesting workflows involving TikTok, let us know in the comments of this issue.

Config file usage is unclear

does anyone have an example config file layout

File name of configuration file to store TikTok credentials to

How to save credentials

--config CONFIG File name of configuration file to store TikTok credentials to

How do TikTok credentials have to be formatted to be parsed correctly?

Enable setting a time/date range of interest

Currently there is no documented way to grab hashtags in a particular time period.

It would be good if there was a command line option or a config file option to set a time range of interest.

(From a mailbox request)

npm i -g tiktok-scraper not working/supported

I have been following the step-by-step instructions on the Bellingcat site. All the steps had worked so far except n 16, but I was still able to get the correct version number displayed through sudo n 16. However, when I enter npm i -g tiktok-scraper, it states that it cannot run the program because Request has been deprecated. When I tried following the directions on the GitHub page, I got this error: ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'.
Is there some way around this issue?

For context, I have a Mac book Air and had Python 3 installed before starting the tutorial. I have no real coding/computer science experience, so I don't really know how to troubleshoot. Thanks!

Example from README does not work: ERROR - Got an unexpected status code

When I run tiktok-hashtag-analysis london newyork paris, I get the following result:

2024-04-02 22:16:55 INFO | Hashtags to scrape: ['london', 'newyork', 'paris']
2024-04-02 22:16:55 INFO | Writing data to directory: /Users/emiel/tiktok_hashtag_data
2024-04-02 22:17:00 WARNING | Encountered error None -> TikTok returned an empty response when fetching data, retrying in headed mode
2024-04-02 22:17:05,955 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '20240402201707626915ACF1D3633DA83D'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:05,955 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '20240402201707626915ACF1D3633DA83D'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:05 ERROR | Got an unexpected status code: {'log_pb': {'impr_id': '20240402201707626915ACF1D3633DA83D'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:06 WARNING | No posts were found for the hashtag: london
2024-04-02 22:17:06 INFO | Scraped 0 new posts containing the hashtag 'london', with 0 posts previously scraped
2024-04-02 22:17:10 WARNING | Encountered error None -> TikTok returned an empty response when fetching data, retrying in headed mode
2024-04-02 22:17:15,627 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '202404022017169B1F5E152602A93EAFBC'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:15,627 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '202404022017169B1F5E152602A93EAFBC'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:15,627 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '202404022017169B1F5E152602A93EAFBC'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:15,627 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '202404022017169B1F5E152602A93EAFBC'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:15 ERROR | Got an unexpected status code: {'log_pb': {'impr_id': '202404022017169B1F5E152602A93EAFBC'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:15 WARNING | No posts were found for the hashtag: newyork
2024-04-02 22:17:15 INFO | Scraped 0 new posts containing the hashtag 'newyork', with 0 posts previously scraped
2024-04-02 22:17:20 WARNING | Encountered error None -> TikTok returned an empty response when fetching data, retrying in headed mode
2024-04-02 22:17:25,593 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '20240402201726506885F21A23AC3E505C'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:25,593 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '20240402201726506885F21A23AC3E505C'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:25,593 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '20240402201726506885F21A23AC3E505C'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:25,593 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '20240402201726506885F21A23AC3E505C'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:25,593 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '20240402201726506885F21A23AC3E505C'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:25,593 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '20240402201726506885F21A23AC3E505C'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:25 ERROR | Got an unexpected status code: {'log_pb': {'impr_id': '20240402201726506885F21A23AC3E505C'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
2024-04-02 22:17:25 WARNING | No posts were found for the hashtag: paris
2024-04-02 22:17:25 INFO | Scraped 0 new posts containing the hashtag 'paris', with 0 posts previously scraped

This is with the tool installed in a clean conda environment. How to solve this issue?

Emojis are not displayed in bar chart

For hashtags containing emojis (such as #ukraine🇺🇦), the plot generated by hashtag_frequencies.plot does not display the emoji, as shown in the figure below.

The problem seems to be that the default matplotlib font can't handle non-ASCII characters.
One option would be using a different font, but I doubt there's any single font installed by default on all operating systems (Windows, macOS, Linux) that we could specify.
One way around this is to specify a font for each operating system, e.g.

from sys import platform
...
if platform == 'win32':
    font_path = <WINDOWS SYSTEM FONT THAT SUPPORTS EMOJIS>.otf
elif platform == 'darwin':
    font_path = <MACOS SYSTEM FONT THAT SUPPORTS EMOJIS>.otf
elif platform in ('linux', 'linux2'):
    font_path = <LINUX/UBUNTU SYSTEM FONT THAT SUPPORTS EMOJIS>.otf
else:
    USE DEFAULT FONT
font_manager.fontManager.addfont(font_path)
prop = font_manager.FontProperties(fname=font_path)

plt.rcParams['font.family'] = 'sans-serif'
plt.rcParams['font.sans-serif'] = prop.get_name()

Another option could be using a package like mplcairo.

see the following links for reference:

Log file in root dir

Running an arbitrary command like tiktok-hashtag-analysis download -t london paris newyork -p, results in a log file trying to be saved in a root directory: PermissionError: [Errno 13] Permission denied: '/Users/logfile.log

OS: macOS 13.2.1
Python: 3.11.2

same as #11 tiktok-scraper: command not found

the program recognize the parameters sent hashtag london and paris, creates the data folder but returns tiktok scrape command not found
subprocess.CalledProcessError: Command 'tiktok-scraper hashtag london -t 'json' --filepath /var/folders/k0/mrsr8fr91856s9bpr07tk32w0000gn/T/tmpw5plbags' returned non-zero exit status 127.
python-BaseException

Investigate Python-only TikTok API wrappers

It's unfortunate that the current implementation relies on a non-Python TikTok API wrapper (https://github.com/drawrowfly/tiktok-scraper implemented in Node), which makes installation and dependency management more difficult. There may be native Python API wrappers (e.g. https://github.com/avilash/TikTokAPI-Python) that work as well.

Ideally any Python TikTok wrapper should be pretty actively maintained, since TikTok seems to change their API frequently.

ModuleNotFoundError: No module named 'playwright._impl._api_types'

playwright-python version:1.40.0
Execute command line:tiktok-hashtag-analysis -h

Traceback (most recent call last):
File "/usr/local/var/pyenv/versions/3.10.0/bin/tiktok-hashtag-analysis", line 5, in
from tiktok_hashtag_analysis.cli import main
File "/usr/local/var/pyenv/versions/3.10.0/lib/python3.10/site-packages/tiktok_hashtag_analysis/init.py", line 1, in
from .base import TikTokDownloader
File "/usr/local/var/pyenv/versions/3.10.0/lib/python3.10/site-packages/tiktok_hashtag_analysis/base.py", line 26, in
from playwright._impl._api_types import Error
ModuleNotFoundError: No module named 'playwright._impl._api_types'

Error: Non-zero exit status 1

Ran through all the steps of the scraper and when I got to this command (python3 run_downloader.py -t ukraine -p) I got a process error, non-zero exit status 1.

Does anyone know why this would be?

I had run a couple of other commands today with a different TikTok scraper which were successful.

Install fails on Windows with Python 3.12 due to msvc-runtime dependency

Currently, when installing on Windows (os_name=='nt'), dependency resolution fails for msvc-runtime because none of their distributions are tagged for CPython 3.12.

I haven't explored why msvc-runtime is required, but one possibility could be to factor out the dependency - which appears to be an unofficial release anyway. An alternative could be to set Python support to <=3.11

Minor code review suggestions

These are all relatively minor things, but they will make it easier for other users to understand and contribute to the project:

Include dependencies, via requirements.txt or (my personal preference) with pipenv, which manages dependencies and virtual environments in a user-friendly way.
Use argparse consistently rather than positional arguments as in extract_date.py
Add example output to README.md to make it more clear what the purpose of the tool is
Consider including a short comment at the beginning of each function that makes it clear what the purpose of the function is (i.e., if it is filtering something, what is it filtering and why?)
(Feature request) Write the graph images to a file rather than displaying in a window, as this can be easier to use. (Could be triggered by command line argument)

Very minor things (sorry for nitpicking):

Every other repository in our organization uses "kebab case" (concatenating-words-like-this) rather than underscores. It would be nice to be consistent.
"Occurrence" is spelled with an e, not an a

No posts were found for the hashtag

Running a command like tiktok-hashtag-analysis london results in the following message:

2023-09-11 20:57:59 INFO | Hashtags to scrape: ['london'] 2023-09-11 20:57:59 INFO | Loaded token from config file: /home/vision/.tiktok 2023-09-11 20:58:11 WARNING | No posts were found for the hashtag: london 2023-09-11 20:58:11 WARNING | No new posts were found for the hashtag: london 2023-09-11 20:58:11 INFO | Scraped 0 new posts containing the hashtag 'london', with 0 posts previously scraped

Installed via pip. Error occurs directly after installation and on the following systems:
Ubuntu 22.04 x64 / Python 3.10.12
Windows 11 / Python 3.11.2
MacOS 10.15.7 / Python 3.9.13

	# Attempt to be robust against TikTok's countermeasures for headless browsing
	try:
	fetched_data = asyncio.run(
	_fetch_hashtag_data(hashtag=hashtag, limit=limit, headed=headed)
	)
	except Exception as e:
	logger.warning(
	f"Encountered error {e} when fetching data, retrying in headed mode"
	)
	fetched_data = asyncio.run(
	_fetch_hashtag_data(hashtag=hashtag, limit=limit, headed=True)
	)

bellingcat / tiktok-hashtag-analysis Goto Github PK

tiktok-hashtag-analysis's People

Contributors

Stargazers

Watchers

Forkers

tiktok-hashtag-analysis's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs