scriptsmith / reaper

Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs

Reaper

Reaper is a PyQt5 GUI that scrapes Facebook, Twitter, Reddit, Youtube, Pinterest, and Tumblr APIs using socialreaper

Are you a developer? Try the Python package

Features

• Support for 6 social media platforms
• CSV output
• Instructions for getting API keys
• API key management
• Download queuing system
• Error management
• Disk caching for big data
• Ability to read a list of inputs from CSV and text files
• Ability to append to exsting data
• Dark theme
• UTF-8 and ASCII support

Download

To download the latest builds for your platform, check out the releases

Installers and standalone versions are available for Windows and macOS

Usage

Instructions for using Reaper are available on reaper.social

Run source

Reaper uses string formatting that was added in Python 3.6. You need to run Reaper with Python 3.6+ or download a pre-built version from the releases

Download

git clone https://github.com/ScriptSmith/reaper.git
cd reaper

Run

pip3 install -r requirements.txt
python3 reaper.py

reaper's Issues

Suggest how you collected exe

Hello, a little off topic, but tell me what you collected the sorts in exe?)

dont have access public repository

when i tried to install the package.. i got the error..

[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Command "git clone -q [email protected]:ScriptSmith/socialreaper.git /home/localadmin/reaper/src/socialreaper" failed with error code 128 in None

Make it easier to use

See these examples:
https://github.com/MalloyDelacroix/DownloaderForReddit

https://github.com/johanneszab/TumblThree

User Report

Submitter: Test User
This is a test of the gitreports system to assign issues to user's bugs without requiring them to sign up to github

SyntaxError

I Keep getting this error, how do i fix this error?

File "reaper.py", line 57
self.window.setWindowTitle(f"Reaper {self.version}")
^
SyntaxError: invalid syntax

No visual feedback for empty result

There is no visual feedback when a query returns an empty result except for the progress bar

Youtube video comments cause application close

Inputting a youtube video-id into youtube function-2 results in an application exit

Error on import qdarkstyle

After following the installation instructions I have the following error:
Traceback (most recent call last):
File "reaper.py", line 24, in
import qdarkstyle
ModuleNotFoundError: No module named 'qdarkstyle'

Reaper's download button becomes disabled after making an erroneous request

Reaper's download button becomes disabled after making an erroneous request

Not a real issue - Tumblr hashtags

Hi,

My name is Joe Hatfield, and I am a graduate student at the University of Colorado Boulder. First off, thank you so much for developing this tool; it has been a lifesaver for my research. Does the tool scrape for ALL hashtags included on individual Tumblr posts? In other words, is there a column(s) for additional hashtags included in a post other than the one used to scrape? My csv seems not to include that info, but I was wondering if perhaps I was missing it.

Many thanks,

Joe

Maximum recursion depth exceeded

The program quits on launching reaper.py

C:\Users\username\AppData\Local\Programs\Python\Python36-32\python.exe D:/reaper-master/reaper.py
Traceback (most recent call last):
File "D:\reaper-master\components\globals.py", line 17, in _calc_path
return _calc_path(head)
File "D:\reaper-master\components\globals.py", line 17, in _calc_path
return _calc_path(head)
File "D:\reaper-master\components\globals.py", line 17, in _calc_path
return _calc_path(head)
[Previous line repeated 993 more times]
File "D:\reaper-master\components\globals.py", line 13, in _calc_path
head, tail = os.path.split(path)
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\ntpath.py", line 206, in split
d, p = splitdrive(p)
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\ntpath.py", line 141, in splitdrive
if len(p) >= 2:
RecursionError: maximum recursion depth exceeded in comparison

404

I setup 3 keys following your guide; Twitter, FB and Pinterest.

When I try to scrape Pinterest or FB I get the following popup error.

"message": "404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.",
"type": "http"
I tried resetting the Pinterest API key as per your instructions, same thing.

Twitter key works fine.

Leading and trailing white-spaces cause errors that are hard for users to debug

Leading and trailing white-spaces in authentication keys cause errors in Reaper that don't make sense to users, and many incompetent users require manual help from instructors to strip white-space.