hfawaz / bigdata18

Transfer learning for time series classification

Transfer learning for time series classification

This is the companion repository for our paper titled "Transfer learning for time series classification" accepted as a regular paper at IEEE International Conference on Big Data 2018 also available on ArXiv.

Architecture

Source code

The software is developed using Python 3.5. We trained the models on a cluster of more than 60 GPUs. You will need the UCR archive to re-run the experiments of the paper.

If you encouter problems with cython, you can re-generate the "c" files using the build-cython.sh script.

To train the network from scratch launch: python3 main.py train_fcn_scratch

To apply the transfer learning between each pair of datasets launch: python3 main.py transfer_learning

To visualize the figures in the paper launch: python3 main.py visualize_transfer_learning

To generate the inter-datasets similariy matrix launch: python3 main.py compare_datasets

Pre-trained and fine-tuned models

You can download from the companion web page all pre-trained and fine-tuned models you would need to re-produce the experiments. Feel free to fine-tune on your own datasets !!!

Prerequisites

All python packages needed are listed in pip-requirements.txt file and can be installed simply using the pip command.

• numpy
• pandas
• sklearn
• scipy
• matplotlib
• tensorflow-gpu
• keras
• h5py
• cython

Results

You can download here the accuracy variation matrix which corresponds to the raw results of the transfer matrix in the paper.

You can download here the raw results for the accuracy matrix instead of the variation.

You can download here the result of the applying nearest neighbor algorithm on the inter-datasets similarity matrix. You will find for each dataset in the archive, the 84 most similar datasets. The steps for computing the similarity matrix are presented in Algorithm 1 in our paper.

Accuracy variation matrix

Generalization with and without the transfer learning

50words - FISH	FordA - wafer	Adiac - ShapesAll

Model's accuracy with respect to the source dataset's similarity

Herring	BeetleFly	WormsTwoClass

Reference

If you re-use this work, please cite:

@InProceedings{IsmailFawaz2018transfer,
  Title                    = {Transfer learning for time series classification},
  Author                   = {Ismail Fawaz, Hassan and Forestier, Germain and Weber, Jonathan and Idoumghar, Lhassane and Muller, Pierre-Alain},
  booktitle                = {IEEE International Conference on Big Data},
  pages                    = {1367-1376}, 
  Year                     = {2018}
}

Acknowledgement

The authors would like to thank NVIDIA Corporation for the GPU Grant and the Mésocentre of Strasbourg for providing access to the GPU cluster.

bigdata18's Issues

Accuracy: Train or Val?

Is the accuracy reported in the results on the paper, training or validation (test)? For e.g., Fig 9, Meat target dataset, the best accuracy without transfer is reported to be around 81%, however, I ran the train_fcn_scratch on Meat and the best val_acc I could get was 43%. This difference is stark!

I found the same for other datasets. Am I missing something?

What I did: just executed this command,

python3.py main train_fcn_scratch

Combined weights of all models

Is not is there any combined data set model. I am looking to classify EEG using transfer learning. But my query is that, all of your model are trained on seprate data set and have seprate weights.
Is not there any sort of cobined weight and architecture. Like we have in computer vision.

will it work for multivariate time series classification for example mixture of categorical and continues data?

will it work for  multivariate  time series classification for example mixture of categorical and continues data? 
for example at time t1 we have observation: red, 2.4 , 5, 12.456 and time t2: green, 3.5, 2, 45.78; time t3: black, 5.6, 7, 23.56; t4: red, 2.1, 5, 12.6 ?

Questions about 'distances.dtw.dtw'

Hi, I am using your previous solution to run the .sh file, but I get the following error. What can I do to fix it? Thank you for your concern.

$ D:/bigdata18-master/bigdata18-master/utils/build-cython.sh
rm: cannot remove '.so': No such file or directory
rm: cannot remove 'build': No such file or directory
D:/bigdata18-master/bigdata18-master/utils/build-cython.sh: line 13: cd: ../shapeDTWefficient: No such file or directory
rm: cannot remove 'shapeDTWefficient.c': No such file or directory
rm: cannot remove '.so': No such file or directory
rm: cannot remove 'build': No such file or directory
rm: cannot remove 'pycache': No such file or directory

transfer

Hello, I'm intested in your work. I want to know when you transfer the model which trained in source data to target data, you use all the target date to fine-tuning the model or just a part of it to fin-tuning it?
Look forward to your reply！
Thanks.

ValueError: Input 0 is incompatible with layer model

I'm attempting to use your work in my time series classsification pipeline. First of all: Thank you very much for providing the source code under an open source license!

I'm trying to train the model, save it to disk and load it again from another method. This is my code:

x_trainScaledNPArray = np.array(x_trainScaled)
x_testScaledNPArray = np.array(x_testScaled)
y_trainNPArray = np.array(y_train)
y_testNPArray = np.array(y_test)

batch_size = 16
nb_epochs = 1 # 2000
verbose = True
    
pathToModelCheckpoints = 'myPath/'
pathToModel = 'myPathToModel/'
reduce_lr = keras.callbacks.ReduceLROnPlateau(monitor='loss', factor=0.5, patience=50,min_lr=0.0001)
# model checkpoint
model_checkpoint = keras.callbacks.ModelCheckpoint(filepath=pathToModelCheckpoints, monitor='loss',save_best_only=True)
callbacks=[reduce_lr,model_checkpoint]
input_shape = x_trainScaledNPArray.shape
nb_classes = len(np.unique(y_train))
model = TransferLearningMain.build_model(input_shape, nb_classes, pre_model=None)
TransferLearningMain.train(x_trainScaledNPArray,y_trainNPArray,x_testScaledNPArray,y_testNPArray,batch_size,verbose,nb_epochs,callbacks,pathToModelCheckpoints,pre_model=None)
model.save(pathToModel)

print(x_trainScaledNPArray.shape)
print(x_testScaledNPArray.shape)

model = keras.models.load_model(model_path)
y_pred = model.predict(x_test,batch_size=4)

The output is:

(18287, 2048)
(347, 2048)

Traceback (most recent call last):
 [...]
    y_pred = model.predict(x_test,batch_size=4)
  File "C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\keras\engine\training.py", line 1727, in predict
    tmp_batch_outputs = self.predict_function(iterator)
  File "C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\def_function.py", line 933, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\def_function.py", line 764, in _initialize
    *args, **kwds))
  File "C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\function.py", line 3050, in _get_concrete_function_internal_garbage_collected
    graph_function, _ = self._maybe_define_function(args, kwargs)
  File "C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\function.py", line 3444, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\function.py", line 3289, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\framework\func_graph.py", line 999, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\eager\def_function.py", line 672, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\framework\func_graph.py", line 986, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

    C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\keras\engine\training.py:1569 predict_function  *
        return step_function(self, iterator)
    C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\keras\engine\training.py:1559 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\distribute\distribute_lib.py:1285 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\distribute\distribute_lib.py:2833 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\distribute\distribute_lib.py:3608 _call_for_each_replica
        return fn(*args, **kwargs)
    C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\keras\engine\training.py:1552 run_step  **
        outputs = model.predict_step(data)
    C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\keras\engine\training.py:1525 predict_step
        return self(x, training=False)
    C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\keras\engine\base_layer.py:1013 __call__
        input_spec.assert_input_compatibility(self.input_spec, inputs, self.name)
    C:\Users\myUser\AppData\Roaming\Python\Python37\site-packages\tensorflow\python\keras\engine\input_spec.py:270 assert_input_compatibility
        ', found shape=' + display_shape(x.shape))

    ValueError: Input 0 is incompatible with layer model: expected shape=(None, 18287, 2048), found shape=(None, 2048)

Any ideas to solve this issue are highly appreciated.

Question of dtw

I installed all dependencies and try to run this program but in util/distance.py,the import code:
from distances.dtw.dtw import dynamic_time_warping as dtw cant work well. It just can not find distances.dtw.dtw.So how to fix it？
I guess the distances/dtw/setup.py can help,but it cant run either:
dtw.c:623:31: Fatal Error：numpy/arrayobject.h：No such file or directory
#include "numpy/arrayobject.h"
^

I used python3.6.7 and CentOS 7.2, GCC 4.8.5
Thx a lot!

I can't download the pre-trained model

these websites are not working:
http://supplementarymaterial.xyz/bigdata2018/fine-tuned-models-part-1.tar.gz
http://supplementarymaterial.xyz/bigdata2018/pre-trained-models.tar.gz

this website is working:
https://germain-forestier.info/src/bigdata2018/ this website is working

Unable to download model

Greetings. I am curious to know that these time series pretrained model can diagnose normal vs disease patients using EEG. But i am unable to download the model. Link on https://germain-forestier.info/src/bigdata2018/ site is not working. Please check it

how can i get train dataset?

how can i get train dataset?

cd ../shapeDTWefficient

bash: cd: ../shapeDTWefficient: No such file or directory

Theere is no such file as it has mentioned in the bash file, hence the dtw load fails. Can you help me with this?

Get positivity of a number

I have a rule that throws numbers between -50 and 50 randomly, is there any way to predict the sign (positive or negative) of the next release with at least 90% accuracy based on a historical record?