nfflow / redditflow

Do everything from data collection from reddit to training a machine learning model in just two lines of python code!

Redditflow.

**Do everything from data collection from reddit to training a machine learning model in just two lines of python code! **

---

Website • Installation • Docs • Huggingface Hub • Blog •

Supports:

• Text Data
• Image Data

Execution is as simple as this:

• Make a config file with your required details of input.
• Run the API in a single line with the config passed as input.

Installation.

pip install redditflow

Latest installation from source.

pip install git+https://github.com/nfflow/redditflow

Examples

Text data collection and training a model in the end.

from redditflow import TextApi


config = {
        "sort_by": "top",
        "subreddit_text_limit": 50,
        "total_limit": 200,
        "start_time": "27.03.2021 11:38:42",
        "end_time": "27.03.2022 11:38:42",
        "subreddit_search_term": "healthcare",
        "subreddit_object_type": "comment",
        "ml_pipeline": {
            'model_name': 'distilbert-base-uncased',
            'model_output_path': 'healthcare_27.03.2021-27.03.2022_redditflow',
            'model_architecture': 'CT'
            }
    }


TextApi(config)

Image data collection

from redditflow import ImageApi


config = {
        "sort_by": "top",
        "subreddit_image_limit": 3,
        "total_limit": 10,
        "start_time": "13.11.2021 09:38:42",
        "end_time": "15.11.2021 11:38:42",
        "subreddit_search_term": "cats",
        "subreddit_object_type": "comment",
        "client_id": "$CLIENT_ID",  # get client id for praw
        "client_secret": '$CLIENT_SECRET',  # get client secret for praw
         }

ImageApi(config)

Since the image api requires praw api from python, a praw client_id and client_secret are required. Read here about how to get client id and client secret for praw.

Citation.

If you use our work, please cite the software in the url: https://github.com/nfflow/redditflow

Launching nfflow Rewards

Contributed to nfflow? Here is a big thank you from our community to you. Claim your badge and showcase them with pride. Let us inspire more folks !

Claim Now!

redditflow's Issues

Add more unsupervised training methods from sentence transformers

Currently, Redditflow suppors "CT" (Contrastive Tension) , an unsupervised training architecture from sentence_transformers. There are more unsupervised training architectures which has similar API under sentence_transformers, which we can integrate into redditflow pipeline.

Reference: https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning

Redditflow source code: https://github.com/nfflow/redditflow/tree/main/redditflow/text_search/trainer/SentenceEmbedder

User Agent Error While Running Config for Image API

While running `ImageAPI(config)`, the key error shown above is thrown.

Since the error is thrown because the user agent is not present in the config and the user agent is not a necessary attribute, a possible workaround would be specifying dummy strings if the user agent is NONE and passing through the headers

Support Distributed Machines

Scraping should be made available across distributed machines, in order to make it faster.
Few ideas to implement this:

• Split the config by period of time. Eg: 4 machines means the start and end time could be split into 4 and each period of time could be handled by each machine.
• Use docker images and pull it across multiple machines.

Separate the model and data APIs

Currently, the API works such that the data is scraped, and the model can run on top of that, all in two steps. We need to separate the model and data APIs, such that the model can start training from an already scraped data.

Dockerize Redditflow

Write dockerfile and update docker image to repository.

Add tests

Documentation

Support Cron job for scraping

The scraper works from a date to the present date, but it should be supporting cron job so that it can scrape until a time in the future.

Add a colab notebook with examples

Machine Learning pipelines

Optionally, users can plan to directly train the output data to an ML pipeline. This should fetch pre-existing scripts from sentence-transformers, or huggingface-transformers to train such models. Initially, unsupervised models would be easy to train.

Reference for training scripts
-: https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning

Add ML Intelligence to data fetching and scrapping ( example: OpenAI's CLIP )

• Download image data from different sources
• Use Colab to load data and train OpenAI's CLIP model to convert images into vector
• save the vectors into the user's gdrive
• Perform evaluation ( search query ) over downloaded data

References: https://github.com/monk1337/OpenAI-CLIP-Image-search

Few suggestions

The project is fantastic; here are a few suggestions :

1. It would be good if there were separate repo for redditflow data and reddit flow model APIs. Sometimes developers want to extract only data and use their model, and sometimes they want to use models but different data. Combining both things results in a bigger size of repo, and also, if I want to scrape only data, I need to install torch, sentence-transformer, sentencepiece etc. ( reference can be huggingface's dataset API and model API )

2. Update the doc for redditflow, including how to extract data based on a single keyword and extract all comments and posts from a single subreddit?

3. Organize the nfflow repo into some base functions which can utilize further for other platform APIs such as Twitter etc

4. Add ML Intelligence to data fetching and scrapping ( example: OpenAI's CLIP )

5. it can also include Elasticsearch to fetch data faster from the downloaded archive.

Here is a simple overview of integrating OpenAI's CLIP project into nfflow:

• Download image data from different sources
• Use Colab to load data and train OpenAI's CLIP model to convert images into vector
• save the vectors into the user's gdrive
• Perform evaluation ( search query ) over downloaded data

It can be automated end to end if training on colab and fetching vectors from the drive can be automated.