stephenlee / hnpickup

This is an educational example of a data mining web application: when is good time to post on HN

How to build a data mining web app?

You will need some basic programming and statistical skills. Web Development, jQuery, Python, and Machine Learning skills are a plus. If you can look at new data and immediately see where data mining adds new value, then you are definitely overqualified to use this source code.

The first step is to get your own data. Is there any websites that you visit every day? I'm sure they produce fresh content every day: new articles, new stats, new numbers. How about you start collecting them? If you have your own data you decide what interesting question it may answer.

Next step. Does your favorite website have any trends? More articles are published in summer than in winter? More people are willing to "like" articles in spring than in autumn. Is it possible to predict which article will create more web traffic, thus, more revenue from advertisements?

Finally, once you mine the answer you should display it. Do it so that it's pleasurable for the eye. Colorful time series or multidimensional scaling should do the trick. Describe your graph so the people not familiar with your project can understand it and enjoy it.

Does it seem like a lot of work? Well, here is a source code that deploys your app with one command on Google App Engine. You just need to focus on where to get the data (ETL), what to do with it (DM), and how to display it (VISUALIZATION). The source code has example that you can swap with an idea of your own.

Little by little you will master how to add monetary value to your data, sell it, or build a business model.

START

Create an account and new app placeholder at:

http://appengine.google.com/

Please install Google App Engine SDK from:

http://code.google.com/appengine/downloads.html

Getting started with Python web app development is here:

http://code.google.com/appengine/docs/python/gettingstarted/

For lunching the app you can use this nice GUI:

http://code.google.com/p/google-appengine-mac-launcher/
http://code.google.com/p/google-appengine-wx-launcher/

It's also good to have Google Analytics account:

http://www.google.com/analytics/

You can also check webmaster tools to make sure your website is properly indexed by Google:

https://www.google.com/webmasters/tools

Don't forget to rename your app in the app.yaml file:

application: hnpickupdev -> application: yourapp

Deploy your frontend with one command:

appcfg.py update ./

Deploy backend with one command:

appcfg.py backends ./ update

Current source code requires at least six data points. That means you have to run "/etl_process" webpage at least six times and then "/dm_process" at least once before you see a graph.

INTRO

This is example of a simple data mining application. Here Hacker News aggregator is our source of data. The data mining objective is to figure out when is good time to post an article or a story on Hacker News website so other people will up-vote it and it will get to from the "newest" page to "news" page.

MODEL

This app can serve as a very simple business model where you claim is that your DATA MINING application brings better EXPERIENCE, OUTCOME, and VALUE to existing products. How come? If you start adding new knowledge to existing data you will see the pattern: large data can be abstracted to a small chunk of information that is more valuable than the large dataset. That's how you sell your service. Example? Every day you observe cars; that's a lot of data, however, you notice that around 8 am there are many more cars than at other hours; this is your small chunk of information. This small chunk will save you 30 min of stuck in traffic: better experience, outcome and value.

CODE

Most data mining application will have very similar information flow:

1.ETL -> 2.DM -> 3.VISUALIZATION

Which means:

1.GET THE DATA -> 2.ADD VALUE -> 3.USE IT

That's why the code is organized into three sections:

1. ETL = Extract, Transform, Load (GET THE DATA)
2. DM = Data Mining (ENRICH THE DATA, ADD VALUE)
3. VISUALISATION = Data presentation in a format that can support decision making process (USE IT, SHOW IT)

You can think of the code as a "Hello, World!" web data mining example. You shouldn't be surprised that most of the code went into visualization. That's how you get your customers to buy-in. Data for visualization is obtained using JSON serialization.

WARNING

This app shows some raw data. For more complicated projects it might not be good idea to show the raw data. Too much data on the user interface will clog the decision making process.

FUTURE

The hope is that early stage start-ups can use this code to quickly organize their thoughts and prototype their idea. Google App Engine can run this app for free, giving opportunity to demonstrate a working version of their idea.

OTHER

Similar data mining app:

http://stephenholiday.com/articles/2011/stack-overflow-by-day/

Remember, time series analysis is just a small portion of data mining.