Generating Twitter Heatmaps using ELK Stack and NodeJS

This is a "How To" on generating a live heat-map of tweets sent within the U.S.A using the ELK Stack(Elasticsearch,Logstash,Kibana) and NodeJS. See below for the screenshot of the final product of this guide.

Directory Structure:

Getting Started:

Prerequisites

Installing

Deployment

Replace the ./filebeat->filebeat.yml with filebeat.yml
Update the server.js file with your Twitter API credentials:

  consumer_key:         '',
  consumer_secret:      '',
  access_token:         '',
  access_token_secret:  '',
  timeout_ms:           60*1000,  // optional HTTP request timeout to apply to all requests.
})

Open 4 Powershell terminals, run the following commands:
- C:\: .\Documents\elasticsearch-5.6.2\bin\elasticsearch
- C:\: .\Documents\kibana-5.6.3-windows-x86\bin\kibana.bat
- C:\: .\Documents\logstash-5.6.3\bin\logstash -f .\Documents\TwitterHeatmaps\twitter_heatmaps_logstash.conf
- C:\: .\Documents\filebeat-5.6.4-windows-x86_64\filebeat.exe -e -c .\Documents\filebeat-5.6.4-windows-x86_64\filebeat.yml
Configure the Index in Kibana:

Go to <localhost:5601> in your browser
Click "Management"
Enter 'twittermaps' as the index pattern
Click "Create"

Create the object in Elasticsearch:

Click "Dev Tools"
Paste the JSON object in the console,highlight the code,click "Play"

PUT twittermaps
{
  "mappings": {
    "locality": {
      "properties": {
          "location": {
          "type": "geo_point"
          }
      }
    }
  }
}

Open a new Terminal or Powershell Window, Go to the TwitterHeatmaps Directory and run: node server.js
In Kibana, click "Discover". You should be able to see the logs input.

Under "Available Fields" hover over "location" and click "Add"
Once "location" is added to "Selected Fields" click "Visualize" (We are almost there!!!)

Create the visualization:

In the "Options" select "Map Type" heatmap
Click "Play", you should see a heatmap appear
You can save this visualization and add it to a Dashboard.
Your visualization should look like the one below:

mkbru / twitterheatmaps Goto Github PK

twitterheatmaps's Introduction

Generating Twitter Heatmaps using ELK Stack and NodeJS

Directory Structure:

Getting Started:

Prerequisites

Installing

Deployment

Replace the ./filebeat->filebeat.yml with filebeat.yml

Update the server.js file with your Twitter API credentials:

Open 4 Powershell terminals, run the following commands:

Configure the Index in Kibana:

Create the object in Elasticsearch:

Open a new Terminal or Powershell Window, Go to the TwitterHeatmaps Directory and run: node server.js

In Kibana, click "Discover". You should be able to see the logs input.

Create the visualization:

The End.

twitterheatmaps's People

Contributors

Stargazers

Watchers

twitterheatmaps's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs

Open a new Terminal or Powershell Window, Go to the TwitterHeatmaps Directory and run: `node server.js`