GithubHelp home page GithubHelp logo

thebarton / urs Goto Github PK

View Code? Open in Web Editor NEW

This project forked from josephlai241/urs

0.0 0.0 0.0 596.22 MB

Universal Reddit Scraper - A comprehensive Reddit scraping/archival command-line tool.

Home Page: https://josephlai241.github.io/URS/

License: MIT License

Python 97.55% Rust 2.45%

urs's Introduction

 __  __  _ __   ____
/\ \/\ \/\`'__\/',__\
\ \ \_\ \ \ \//\__, `\
 \ \____/\ \_\\/\____/
  \/___/  \/_/ \/___/

Universal Reddit Scraper - A comprehensive Reddit scraping command-line tool written in Python.

GitHub Workflow Status (Python) GitHub Workflow Status (Rust) Codecov GitHub release (latest by date) Total lines License

[-h]
[-e]
[-v]

[-t [<optional_date>]]
[--check]

[-r <subreddit> <(h|n|c|t|r|s)> <n_results_or_keywords> [<optional_time_filter>]]
    [-y]
    [--csv]
    [--rules]
[-u <redditor> <n_results>]
[-c <submission_url> <n_results>]
    [--raw]
[-b]
    [--csv]

[-lr <subreddit>]
[-lu <redditor>]

    [--nosave]
    [--stream-submissions]

[-f <file_path>]
    [--csv]
[-wc <file_path> [<optional_export_format>]]
    [--nosave]

Table of Contents

Contact

Whether you are using URS for enterprise or personal use, I am very interested in hearing about your use case and how it has helped you achieve a goal. Additionally, please send me an email if you would like to contribute, have questions, or want to share something you have built on top of it.

You can send me an email by clicking on the badge. I look forward to hearing from you!

ProtonMail

Introduction

This is a comprehensive Reddit scraping tool that integrates multiple features:

  • Scrape Reddit via PRAW (the official Python Reddit API Wrapper)
    • Scrape Subreddits
    • Scrape Redditors
    • Scrape submission comments
  • Livestream Reddit via PRAW
    • Livestream comments submitted within Subreddits or by Redditors
    • Livestream submissions submitted within Subreddits or by Redditors
  • Analytical tools for scraped data
    • Generate frequencies for words that are found in submission titles, bodies, and/or comments
    • Generate a wordcloud from scrape results

"Where’s the Manual?"

This README has become too long to comfortably contain all usage information for this tool. Consequently, the information that used to be in this file has been moved to a separate manual created with mdBook, a Rust command-line tool for creating books from Markdown files.

Note: You can also find the link in the About sidebar in this repository.

Demo GIFs

Here are all the demo GIFs recorded for URS.

Note: The nd command is nomad, a modern tree alternative I wrote in Rust.

subreddit demo

redditor demo

submission comments demo

livestream subreddit demo

frequencies demo

wordcloud demo

check praw rate limits demo

display directory tree demo

Sponsors

This is a shout-out section for my patrons - thank you so much for sponsoring this project!

urs's People

Contributors

josephlai241 avatar dependabot[bot] avatar iceberge421 avatar lukedschenk avatar theregoesmysanity avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.