karlheinzniebuhr / ancde

Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting

Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting
for Time-Series Classification and Forecasting
[arXiv]

The overall architecture of NCDE

Our proposed attentive NCDE In our method, the bottom NCDE produces attention values and the top NCDE produces the last hidden vector.

The detailed architecture of our proposed attentive NCDE. From the raw time-series data x(t), the continuous path X(t) is created by the natural cubic spline algorithm. The bottom NCDE reads X(t) to produce attention values. The path Y(t) is defined as the (element-wise) multiplication of X(t) and the attention value at time t. The top NCDE produces the last hidden vector with Y(t). Our framework supports both classification and regression.

The soft attention visualization in physioNet Sepsis. Blue means strong attention and white means weak attention. Note that the bottom NCDE gives strong attention when there are non-trivial changes on the curves.

The soft attention visualization in Google Stock. Blue means strong attention and white means weak attention. Note that the open price has weak attention always. This is because it shows high correlations to other values. Our attention model strategically ignores the open price.

create conda environments

conda env create --file ancde.yml

activate conda

conda activate neuralcde

UEA_CharacterTrajectories

python3 uea_attentive.py --seed 2021 --missing_rate 0.3 --model="ancde" --h_channel 40 --hh_channel 40 --layer 3 --lr 0.001 --soft 'True' --slope_check '' --timewise 'True' --attention_channel 20 --attention_attention_channel 10 --step_mode 'valloss'

PhysioNet Sepsis No OI

python3 sepsis_attentive.py --seed 2021 --intensity '' --model="ancde" --h_channel 49 --hh_channel 49 --layer 4 --lr 0.00001  --soft 'True' --slope_check '' --timewise 'True' --attention_channel 20 --attention_attention_channel 20 --step_mode 'valloss'

PhysioNet Sepsis OI

python3 sepsis_attentive.py --seed 2021 --intensity 'True' --model="ancde" --h_channel 49 --hh_channel 49 --layer 4 --lr 0.00001  --soft 'True' --slope_check '' --timewise 'True' --attention_channel 20 --attention_attention_channel 20 --step_mode 'valloss'

Google Stock(single-step forecasting)

python3 stock.py --seed 2021 --input_seq 24 --output_seq 1 --model="ancde_forecasting" --h_channel 12 --hh_channel 12 --layer 2 --lr 0.001 --soft '' --slope_check 'True' --timewise '' --attention_channel 4 --attention_attention_channel 8 --step_mode 'valloss'

Google Stock(multi-step forecasting)

python3 stock.py --seed 2021 --input_seq 20 --output_seq 5 --model="ancde_forecasting" --h_channel 15 --hh_channel 15 --layer 2 --lr 0.005 --soft '' --slope_check 'True' --timewise '' --attention_channel 4 --attention_attention_channel 8 --step_mode 'valloss'

Mujoco

python3 mujoco.py --seed 2021 --input_seq 20 --output_seq 5  --model="ancde_forecasting" --h_channel 12 --hh_channel 12 --layer 2 --lr 0.001 --soft '' --slope_check 'True' --timewise '' --attention_channel 4 --attention_attention_channel 8 --step_mode 'valloss'

Run shell file (ex PhysioNet Sepsis dataset)

sh sepsis.sh 

Run shell file, background (ex PhysioNet Sepsis dataset)

nohup sh sepsis.sh 2>sepsis.err 1>sepsis.log & 

Citation

@article{jhin2021attentive,
  title={Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting},
  author={Jhin, Sheo Yon and Shin, Heejoo and Hong, Seoyoung and Jo, Minju and Park, Solhee and Park, Noseong},
  journal={ICDM},
  year={2021}
}