tinyjumbo / stockweb

checkout www.zachqiu.com

stockweb

Description

Build stock web predict stock based on tweets

Webpage: www.zachqiu.com

Resource

1.[Flask ORG] http://flask.pocoo.org/

2.[Twitter API] https://dev.twitter.com/streaming/overview

Code style

Clean code.

Language & Framework

• Front-end: JavaScript
• Back-end: Python
• Database: MongoDB
• Framework: Flask

Owner

@team: 小飞象 Jumbo

stockweb's Issues

selected user comments about stock

1. Since we've already have bar chart showing user comments on several stocks.
   We can possibly add a separate div under the barchart showing some tweets about picked stocks.
   Criterion can be some hot tweets? maybe tweets with most comments?
   pick several and dynamically showing tweets.
2. Real time status instead of day change. Instead of bar chart. we can also have a cur showing how many tweets are there for each stock per 5 second? By saying 5 second I am tring to say.. got roll over window averaging tweets number.

generalize code block that reads db as a method

so that we can reuse it in the future. mentioned this #2

need to find a way to handle db connection error properly

#2 mentioned this.

should avoid duplicate db connection config code in __init__.py and views.py

#2 mentioned this.

design rate limiting for crawler (discussion)

@qiuzhihui
two use case:

1. for our crawling jobs.
2. prevent robot from crawling our web too often. I would assume this may happen later when there are some robots.. trying to visit the web thousands of times per second.. we can have some idea to block them.
   This just came out of my mind today, when I was writing code for rate limiting.
   Instead of using cronjob. we can possibly have a separate process for crawling using rate limiter?

I would assume doing crawling is sth we are constantly working, instead of just run it several time perday, since we are looking for real time data.

but the issue is, if we use same Id to craw too fast we may be block out? I don't know the limitation for crawling T and F.. but should be sth like.

def getLimit(max_frequency):
    last_time = [0.0]
    i = [0]
    def rateLimit(max_frequency):
        min_interval = 1.0 / max_frequency
        cur_time = time.clock()
        while min_interval > cur_time - last_time[0]:
            cur_time = time.clock()
        last_time[0] = cur_time
        print  i[0]
        i[0] += 1
    while True:
        rateLimit(max_frequency)
getLimit(2)

you can replace i[0] with a function.. so that we will not craw too frequently.
downside is this process will always take resources..
upside is you can use separate machine to do this job.
But I would say cron job may not be a correct solution for crawling, since it only handle periodic jobs.

Set stocks symbol as var

Now we only have hardcode stock symbol and show several stocks.
A feature we would possibly wanna do is to let user choose what stocks they wanna follow. and based on input showing stock charts.

Note that this is not just stock price char, it is also about barchart. as we may need to analyze more stocks in the future. This change seems much more important than most of other features.
@Jelly-Yu @qiuzhihui
Hardcoding is a bad habit. But a good start. yea.

separate db connection from views

currently we mix db configuration with views, according to @hyperchi 's suggestion, we should separate them.

Users management design.

@qiuzhihui @Jelly-Yu
This is sth I will do by the end of next month I mean March or early April. If you guys wanna do also fine.
We need a user management feature later on.

1. Set up user login feature
2. User management page to change each users' info.
3. User register, email notification.
   There would be a top bar in each page for users, we will need profile management.
   new pages
   login register reset_pwd user_management

Several tech needed. Ajax call. User pwd MD5 hash. Email part? have no idea right now.

table 'users' required fields.
I would personally maybe prefer all varchar except id date

1. id int(10)
2. username varchar(20)
3. password varchar(20)
4. email varchar(30)
5. birthday date default null
6. register_ts date default null
7. gender varchar(10)
8. score int(10)
   if we need more comment. make sure these are not sth we'r going to alter in the future. Alter field would be painful.

All work should better be under branch then merge.

@Jelly-Yu @qiuzhihui
Jelly keeps a great merging branch habit .
Seems like a better idea, since now we have a little more code to maintain.. avoid unnecessary push.
When git commit we can specify which file to commit.. instead of just commit the whole repo. Base on what I found on @qiuzhihui commit history, it seems like you are mostly committing the whole repo.
I will stick to this habit as well, if later on I have time to write code for it.
I'm not familiar with git. But should be sth like following.
@Jelly-Yu if not correct plz comment as well.

create xxx

git branch xxx

switch to xxx

git checkout xxx

commit to local

git commit -m ""

merge

git checkout master
git merge xxx