mlh-fellowship / jamspam

GitHub App to jam the spam PRs on your repo and keep maintainers stress-free (even in Hacktober 🎃)

JamSpam

A Machine Learning powered GitHub App built with Probot to jam the spam PRs on your repo and keep maintainers stress-free (even in Hacktober 🎃)

Summary

Building Dataset

• We listed links of PRs labelled as ⚠ SPAM or INVALID ⚠ on some popular repositories especially those that faced a pool of spam pull-requests during the recently concluded Hacktoberfest 🎃 in a .csv file.
• Similarly, we also listed links of ✅ MERGED PRs on the repositories in a separate .csv file for Ham (not Spam) features.
• We used Octokit, an API framework by GitHub to extract Pull Request Information from the PR links and save desired features locally to build our model.

Feature Extraction

We chose the standard PR attributes and some derived features to train our model

• Standard
  • Number of Commits
  • Number of Files Changed
  • Number of Changes (Additions + Deletions)
• Derived

  • Number of Files Changed of Documentation Type

    # File Extensions considered to be of Doc-Type 
    ['md', 'txt', 'rst', '']

  • Occurences of spam hit-words in text corpus of PR

    Text Corpus of a Pull Request includes the PR Title, Body, Commit Messages and Diffs.

    All text is pre-processed with regex to exclude any symbols.

Model Design

We are using Keras to build our baseline model. It is essentially a (5-16-16-1) Sequential Neural Network with first three layers being 'RELU' activated and the final output layer activated as a sigmoid function.

The model is run over 500 epochs with a unit batch size.

Transfer Model to Bot

The model is exported from Python using tensorflowjs that creates a model.json and a .bin file to store the model structure, variables and associated weights.

The model is imported seamlessly into Node.js using @tensorflow/tfjs-node for predictions to be made for incoming PRs

Getting Started

• For setup instructions to train and export the model, visit jam-spam-ml/README.md
• For setup instructions to build the bot and getting the GitHub App running, head to jam-spam-app/README.md

Contributing

If you have suggestions for how JamSpam could be improved, or want to report a bug, open an issue! We'd love all and any contributions.

For more, check out the Contributing Guide.

Screenshots

1. If you are a Collaborator, Contributor, Member, or Owner of the repository your pull request will never be flagged.

2. If you are a First Timer, Mannequin or First Time Contributor your pull requests will be checked.

If the pull request is legit, it is not flagged

If the pull request is suspected to be spam, it is marked as spam and closed.

License

MIT © 2020 MLH Fellowship

Made with ❤️ by Ajwad Shaikh & Vrushti Mody during Sprint 3 of the MLH Fellowship Explorer Batch, Fall 2020.

jamspam's Issues

Build Classifier Model

• Build Spam/Ham Classifier Model
• Decide on passing threshold for misclassification rate
• Test for Accuracy / Improve, if poor

Initialise the Probot Project

• Initialise app in jam-spam-app with Probot
• Set up GitHub App on MLH-Fellowship Organization
• Flag off 🏁

Add Check For Valid Documentation PRs

Keep pull request open irrespective of contents if all the below conditions are met (skip spam detection)

• PR should have a linked issue.
• Linked issue should have label documentation
• Labelled issue should be assigned to the same user who has sent the Pull Request

Build Training Dataset - SPAM

• Create a list of 50 Spam PRs (links) in the shared Google Sheet

• Add spam.csv to jam-spam-ml/data

NLP to detect hit keywords for SPAM/HAM dataset

Implement NLP to extract keywords from SPAM and HAM corpus.

A frequency vector of these keywords would be a great feature for our model. To make sure, we have keywords specific to SPAM and HAM characteristics of the PR, we decide to do the following.

N = complexity of the model (starting with 30, might change iteratively to achieve better results)

A = Top N keywords list from SPAM dataset
B = Top N keywords list from HAM dataset

SPAM_KEYWORDS = (A - B)
HAM_KEYWORDS = (B - A)

Suggest using multi-rake for rapid keyword extraction from corpus

Extract 'feature'ful attributes from fetched data

• PR Title
• PR Body
• Diffs (actual changes)
• Commit Messages
• Files Changes
• Documentation File Changes (.md, .rst, .txt)
• Number of commits
• Diffs number (additions + deletions)

(Those ticked have been worked on in #12 (WIP) or earlier)

Build Training Dataset - HAM

• Create a list of 50 Ham PRs (links) in the shared Google Sheet

• Add ham.csv to jam-spam-ml/data

Data Aggregation Script

Create a script to scrape or possibly use APIs to get attributes of spam and ham PRs in order to build a complete dataset

Attributes

• PR Title
• PR Description
• Diffs
• Commit Messages
• .. discussion open for more attributes