vedic-partap / event-driven-stock-prediction-using-deep-learning

A deep learning method for event driven stock market prediction. Deep learning is useful for event-driven stock price movement prediction by proposing a novel neural tensor network for learning event embedding, and using a deep convolutional neural network to model the combined influence of long-term events and short-term events on stock price movements

Event-Driven Stock Prediction using Deep-Learning

A deep learning method for event driven stock market prediction. Deep learning is useful for event-driven stock price movement prediction by proposing a novel neural tensor network for learning event embedding, and using a deep convolutional neural network to model the combined influence of long-term events and short-term events on stock price movements

Important concept need to be learnt :

1. NLP - http://web.stanford.edu/class/cs224n
2. GloVe - GloVe- https://nlp.stanford.edu/pubs/glove.pdf
3. NLT - www.nltk.org
4. Convolutional Neural Network - http://www.aclweb.org/anthology/D14-1181

Sentiment Analysis for Event-Driven Stock Prediction

Also try to visualize the data on smaller dimension using PCA to get the good idea of the problem

Use NLP to predict stock price movement based on news from Reuters, we need the following 5 steps:

1. Collection of Data ( essential and tricky task )

   1.1 get the whole ticker list

   1.2 crawl news

   1.3 crawl prices using urllib2 (Yahoo Finance API is outdated)

2. Train the GloVe in corpus in NLTK

   2.1 build the word-word co-occurrence matrix

   2.2 factorizing the weighted log of the co-occurrence matrix

3. Feature Engineering

   3.2 Unify word format: unify tense, singular & plural, remove punctuations & stop words

   3.2 Extract feature using feature hashing based on the trained word vector (step 2)

   3.3 Pad word senquence (essentially a matrix) to keep the same dimension

4. Trained a ConvNet to predict the stock price movement based on a reasonable parameter selection

5. The result shows a significant 1-2% improve on the test set

Use the following script to crawl it and format it to our local file Note : We can relate the news with company and date, this is more precise than Bloomberg News

1 Crawling of Data

python crawler_reuters.py 

Yahoo Finanace is also a great place to collect the data

2 Word Embedding

To use our customized word vector, apply GloVe to train word vector from Reuters corpus in NLTK

python embeddingWord.py

To use pre-trained GloVe word vectors

python embeddingWordPre.py

3. Feature Engineering

Projection of word to word vector Seperate test set away from training+validation test, otherwise we would get a too optimistic result.

python genFeatureMatrix.py

Here there is important point to note when we are separating the Cross Validation set and the Training Set. The shuffiling of data can create a very large mistake and untraceble. Consider we have news that are similar in contex but the language of news are slightly different and the got separated in training and the cross validation set. Then the error in cross validation set get biased as the model is already trained against that model so that example will be of no use and effective cross validation set reduces.

4. Training

To train a Stacked-Bidirectional GRU network to predict the stock price movement.

python model_sb_gru.py

To train a ConvoNet network to predict the stock price movement.

python model_cnn.py

5. Model

Model folder contains a trained model with 100K news on Stacked-Bidirectional GRU network. To use the model

cd model
python load.py

Accuracy of model is 96.79%. Furthur improvement like attention mechanism or cyclic learning rate is required.

Other Reading Materials

1. H Lee, etc, On the Importance of Text Analysis for Stock Price Prediction, LREC, 20145. Xiao Ding, Deep Learning for Event-Driven Stock Prediction, IJCAI2015
2. IMPLEMENTING A CNN FOR TEXT CLASSIFICATION IN TENSORFLOW
3. Keras predict sentiment-movie-reviews using deep learning
4. Keras sequence-classification-lstm-recurrent-neural-networks
5. tf-idf + t-sne
6. Implementation of CNN in sequence classification
7. Getting Started with Word2Vec and GloVe in Python

event-driven-stock-prediction-using-deep-learning's Issues

UnicodeDecodeError: 'gbk' codec can't decode byte 0x80 in position 0: illegal multibyte sequence

when run genfeatureMatrix.py,there is a mistake.
:UnicodeDecodeError: 'gbk' codec can't decode byte 0x80 in position 0: illegal multibyte sequence

can not craw reuters news

http://www.reuters.com/finance/stocks/company-news/BIDU.O?date=09262017
this link seems invalid now.

ImportError: No module named en

ub16hp@UB16HP:~/ub16_prj/Event-Driven-Stock-Prediction-using-Deep-Learning$ python embeddingWord.py
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
Traceback (most recent call last):
File "embeddingWord.py", line 8, in
import en
ImportError: No module named en

no news for any tickers

The following line 48 in crawler_reuters might need an update-
"has_Content = len(soup.find_all("div", {'class': ['topStory', 'feature']}))"

fields 'topstory' & 'feature' do not exist on the page. Might need replacement with "MARKET VIEWS"

Validation error very high

ileNotFoundError: [Errno 2] File b'../stockFeatures/train.csv' does not exist: b'../stockFeatures/train.csv'

where is the train.csv

ValueError: cannot reshape array of size 150000000 into shape (50000,20,50,1)

ub16hp@UB16HP:/ub16_prj/Event-Driven-Stock-Prediction-using-Deep-Learning$ python model_cnn.py
Using TensorFlow backend.
Traceback (most recent call last):
File "model_cnn.py", line 99, in
main()
File "model_cnn.py", line 94, in main
model_selection(clusters)
File "model_cnn.py", line 84, in model_selection
X_train, y_train, X_valid, y_valid, X_test, y_test = get_Feature_Label(clusters=clusters)
File "model_cnn.py", line 40, in get_Feature_Label
X = X.reshape(X.shape[0], 20, 50, 1).astype('float32')
ValueError: cannot reshape array of size 150000000 into shape (50000,20,50,1)
ub16hp@UB16HP:/ub16_prj/Event-Driven-Stock-Prediction-using-Deep-Learning$ vim model_cnn.py
ub16hp@UB16HP:~/ub16_prj/Event-Driven-Stock-Prediction-using-Deep-Learning$ ll
total 576
drwxrwxr-x 4 ub16hp ub16hp 4096 12月 25 16:04 ./