abdullah353 / cluster

Usage Instructions

Tested on Ubuntu 16

1. Install Spark

# Update package.
sudo apt -qy update

# Install JRE and pip
sudo apt -qy install default-jre python-pip

# Download from spark.
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz

# Extract it.
tar -xzf spark-2.2.0-bin-hadoop2.7.tgz

# Remove tar file
rm spark-2.2.0-bin-hadoop2.7.tgz

# Adjusting Environment variable PATH
echo 'export PATH=$PATH:~/spark-2.2.0-bin-hadoop2.7/sbin/' >> ~/.bash_aliases
echo 'export PATH=$PATH:~/spark-2.2.0-bin-hadoop2.7/bin/' >> ~/.bash_aliases
echo 'export SPARK_HOME=~/spark-2.2.0-bin-hadoop2.7/' >> ~/.bash_aliases

source ~/.bashrc

2. Install App and it's dependencies

git clone https://github.com/mabdullah353/cluster.git
cd cluster

# Install dependencies
sudo pip install -r requirements.txt
python -c 'import nltk
nltk.download("punkt")'

3. Run The App on spark

spark-submit app.py

This will generate index.p serialized python object which is going to be utillized by api endpoint in api.py

Launch The API

api.py holds is responsible for expoing API endpoint to launch it use the following method

$ python api.py --port 8083

To list full usage of above command use the following help command.

$ python api.py --help
usage: api.py [-h] [--host HOST] [--port PORT] [--debug DEBUG]

Expose K-Mean cluster

optional arguments:
  -h, --help     show this help message and exit
  --host HOST    Adjust the host of the server.
  --port PORT    Adjust the port of the server.
  --debug DEBUG  Enable app debugging.

Visit http://52.26.18.233:8083/cluster To see the response.

Note: Use the IP Address accordingly, in above url.

Understanding the Response:

{
  "algo_meta": {
    "wsss_error": "476.100523092",
    "k": 83,
    "iteration": 100
  },
  "clustere_news": [
    {
      "url": "https://www.cbsnews.com/news/billy-bush-donald-trump-voice-access-hollywood-tape/",
      "cluster_id": 46,
      "summary": "Billy Bush confirms it was Trump on infamous tapeBush writes of the tape validity amid reports tha"
    },
    {
      "url": "https://www.cbsnews.com/news/commentary-the-anti-trump-coalition-isnt-happening/",
      "cluster_id": 13,
      "summary": "Commentary: The anti-Trump coalition isn't happeningWhy a bipartisan "
    },
   .....
}

Key	Purpose
algo_meta.wsss_error	Within Set Sum of Squared Errors
algo_meta.k	K used to generate the data
algo_meta.iteration	Maximum Number of Iteration
clustere_news.cluster_id	Cluster Id in which news categorized
clustere_news.url	URL Of the original News
clustere_news.summary	Summary of the original New

cluster's Issues

Cluster News

Auto-group stories in to k clusters based on content similarity. A story about the
same company or person, for example, should not be in the same cluster unless the
substance of the material matches. For example, a story about the new iPhone
release should not be clustered with a story about Steve Job’s death.

Grab news content from available sources

1. Grab news from three major RSS feeds.
   a) http://rss.cnn.com/rss/cnn_topstories.rss
   b) http://feeds.foxnews.com/foxnews/latest
   c) https://www.cbsnews.com/latest/rss/main
2. Parse feed and utilize title, description, scraping of body text, and any other
   information you wish to leverage to solve the task.

API JSON

Expose API with JSON payload showing clustered content.