zezhishao / basicts

A Standard and Fair Time Series Forecasting Benchmark and Toolkit.

A Standard and Fair Time Series Forecasting Benchmark and Toolkit.

---

$\text{BasicTS}^{+}$ (Basic Time Series Plus) is an enhanced benchmark and toolbox designed for time series forecasting. $\text{BasicTS}^{+}$ evolved from its predecessor, BasicTS, and now has robust support for spatial-temporal forecasting and long time-series forecasting as well as more general tasks, such as M4 competition. For brevity and consistency, we will interchangeably refer to this project as $\text{BasicTS}^{+}$ and $\text{BasicTS}$.

On the one hand, BasicTS utilizes a unified and standard pipeline to give a fair and exhaustive reproduction and comparison of popular deep learning-based models.

On the other hand, BasicTS provides users with easy-to-use and extensible interfaces to facilitate the quick design and evaluation of new models. At a minimum, users only need to define the model architecture.

We are collecting TODOs and HOWTOs, if you need more features (e.g. more datasets or baselines) or have any questions, please feel free to create an issue or leave a comment here.

If you find this repository useful for your work, please consider citing it as such.

✨ Highlighted Features

Fair Performance Review

Users can compare the performance of different models on arbitrary datasets fairly and exhaustively based on a unified and comprehensive pipeline.

Developing with BasicTS

Minimum Code

Users only need to implement key codes such as model architecture and data pre/post-processing to build their own deep learning projects.

Everything Based on Config

Users can control all the details of the pipeline through a config file, such as the hyperparameter of dataloaders, optimization, and other tricks (*e.g.*, curriculum learning).

Support All Devices

BasicTS supports CPU, GPU and GPU distributed training (both single node multiple GPUs and multiple nodes) thanks to using EasyTorch as the backend. Users can use it by setting parameters without modifying any code.

Save Training Log

Support `logging` log system and `Tensorboard`, and encapsulate it as a unified interface, users can save customized training logs by calling simple interfaces.

📦 Built-in Datasets and Baselines

Datasets

BasicTS support a variety of datasets, including spatial-temporal forecasting, long time-series forecasting, and large-scale datasets, e.g.,

• METR-LA, PEMS-BAY, PEMS03, PEMS04, PEMS07, PEMS08
• ETTh1, ETTh2, ETTm1, ETTm2, Electricity, Exchange Rate, Weather, Traffic, Illness, Beijing Air Quality
• SD, GLA, GBA, CA
• ...

Baselines

BasicTS implements a wealth of models, including classic models, spatial-temporal forecasting models, and long time-series forecasting model, e.g.,

• HI, DeepAR, LightGBM, ...
• DCRNN, Graph WaveNet, MTGNN, STID, D2STGNN, STEP, DGCRN, DGCRN, STNorm, AGCRN, GTS, StemGNN, MegaCRN, STGCN, STWave, STAEformer, GMSDR, ...
• Informer, Autoformer, FEDformer, Pyraformer, DLinear, NLinear, Triformer, Crossformer, ...

💿 Dependencies

Preliminaries

OS

We recommend using BasicTS on Linux systems (e.g. Ubuntu and CentOS). Other systems (e.g., Windows and macOS) have not been tested.

Python

Python >= 3.6 (recommended >= 3.9).

Miniconda or Anaconda are recommended to create a virtual python environment.

Other Dependencies

BasicTS is built based on PyTorch and EasyTorch. You can install PyTorch following the instruction in PyTorch. For example:

pip install torch==1.10.0+cu111 torchvision==0.11.0+cu111 torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html

After ensuring that PyTorch is installed correctly, you can install other dependencies via:

pip install -r requirements.txt

Warning

BasicTS is built on PyTorch 1.9.1 or 1.10.0, while other versions have not been tested.

🎯 Getting Started of Developing with BasicTS

Preparing Data

• Clone BasicTS

  cd /path/to/your/project
  git clone https://github.com/zezhishao/BasicTS.git

• Download Raw Data

  You can download all the raw datasets at Google Drive or Baidu Yun(password: 6v0a), and unzip them to datasets/raw_data/.

• Pre-process Data

  cd /path/to/your/project
  python scripts/data_preparation/${DATASET_NAME}/generate_training_data.py

  Replace ${DATASET_NAME} with one of METR-LA, PEMS-BAY, PEMS03, PEMS04, PEMS07, PEMS08, or any other supported dataset. The processed data will be placed in datasets/${DATASET_NAME}.

3 Steps to Evaluate Your Model

• Define Your Model Architecture

  The forward function needs to follow the conventions of BasicTS. You can find an example of the Multi-Layer Perceptron (MLP) model in baselines/MLP/mlp_arch.py

• Define Your Runner for Your Model (Optional)

  BasicTS provides a unified and standard pipeline in basicts.runner.BaseTimeSeriesForecastingRunner. Nevertheless, you still need to define the specific forward process (the forward function in the runner). Fortunately, BasicTS also provides such an implementation in basicts.runner.SimpleTimeSeriesForecastingRunner, which can cover most of the situations. The runner for the MLP model can also use this built-in runner. You can also find more runners in basicts.runners.runner_zoo to learn more about the runner design.

• Configure your Configuration File

  You can configure all the details of the pipeline and hyperparameters in a configuration file, i.e., everything is based on config. The configuration file is a .py file, in which you can import your model and runner and set all the options. BasicTS uses EasyDict to serve as a parameter container, which is extensible and flexible to use. An example of the configuration file for the MLP model on the METR-LA dataset can be found in baselines/MLP/MLP_METR-LA.py

Run It!

• Reproducing Built-in Models

  BasicTS provides a wealth of built-in models. You can reproduce these models by running the following command:

  python experiments/train.py -c baselines/${MODEL_NAME}/${DATASET_NAME}.py --gpus '0'

  Replace ${DATASET_NAME} and ${MODEL_NAME} with any supported models and datasets. For example, you can run Graph WaveNet on METR-LA dataset by:

  python experiments/train.py -c baselines/GWNet/METR-LA.py --gpus '0'

• Customized Your Own Model

  Example: Multi-Layer Perceptron (MLP)

Contributors ✨

Thanks goes to these wonderful people (emoji key):

S22 🚧 💻 🐛

LMissher 💻 🐛

Chengqing Yu 💻

CNStark 🚇

Azusa 🐛

Yannick Wölker 🐛

hlhang9527 🐛

This project follows the all-contributors specification. Contributions of any kind welcome!

📉 Main Results

See the paper Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis.

🔗 Acknowledgement

BasicTS is developed based on EasyTorch, an easy-to-use and powerful open-source neural network training framework.

basicts's Issues

DGCRN的结果复现有出入

您好！Zezhi Shao,很感谢您开源的代码，这对我很有帮助！
但是，我在PEMS数据集上复现DGCRN时，发现按照代码的基础参数，与您图表里的结果相差甚远。
想在此咨询下您这个问题，是不是DGCRN一些设置没在开源代码中更新。
这个是PEMS08的最终结果练日志：

PEMS04:

PEMS03:

期待您的回复！祝您万事顺遂！

Custom dataset

Hi!

Thank you for this awesome work!

Could you help me navigate on how I can use this on my custom dataset?

Thanks!

Add README for Traffic Runner.

请教下tests

在tests文件下进行测试，最后跑出来的结果只有test+time.log里面什么都没有，想问下是什么原因呢？

Rename all the config files

CL setting in calculating training loss in v0.2.0 is wrong

It used to be right. But after v0.2.0 change.
In base_tsf_runner.py #292 line the metric to calcuate loss should the forward_return[:2] which affected by CL setting.
But now it does not. And the model performance is worse than before because of this.
NEED TO FIX ASAP.

Unify data pre-processing of traffic datasets and LSTF datasets.

DCRNN seems to be having an GPU environment issue on multi-GPU machines.

怎么在DGCRN的消融实验中删除time of day 进行实验

单纯在generate_training_data.py中选择False会报错

switch this line

https://github.com/zezhishao/BasicTS/blob/5639ebbf3c1de171d3747654c02a34a50d65cbea/examples/MLP/MLP_METR-LA.py#L11

Error document.

https://github.com/zezhishao/BasicTS/blob/694e1c120a65abadf0cc58ff3368da5c13a7377c/basicts/archs/arch_zoo/pyraformer_arch/pyraformer_arch.py#L116

作者您好，请问各个模型的中注意力权重或者图矩阵权重在哪里保存？

我想对former模型，或者时空模型的图矩阵进行可视化，类似于热力图那种。请指点如何基于BasicTS实现?

Add GTS

Add GTS model.

MTGNN运行报错

终端指令：python examples/run.py -c examples/MTGNN/MTGNN_METR-LA.py

2023-08-20 11:17:39,599 - easytorch-training - INFO - Epoch 1 / 10
0%| | 0/192 [00:03<?, ?it/s]
2023-08-20 11:17:43,446 - easytorch-training - ERROR - Traceback (most recent call last):
File "D:\anaconda\envs\Basicts\lib\site-packages\easytorch\launcher\launcher.py", line 31, in training_func
File "D:\pycharmproject\BasicTS-master\basicts\runners\runner_zoo\mtgnn_runner.py", line 109, in train_iters
loss = super().train_iters(epoch, iter_index, data)
File "D:\pycharmproject\BasicTS-master\basicts\runners\base_tsf_runner.py", line 276, in train_iters
forward_return = list(self.forward(data=data, epoch=epoch, iter_num=iter_num, train=True))
loss = super().train_iters(epoch, iter_index, data)
File "D:\pycharmproject\BasicTS-master\basicts\runners\base_tsf_runner.py", line 276, in train_iters
forward_return = list(self.forward(data=data, epoch=epoch, iter_num=iter_num, train=True))
forward_return = list(self.forward(data=data, epoch=epoch, iter_num=iter_num, train=True))
prediction_data = self.model(
File "D:\anaconda\envs\Basicts\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
adp = self.gc(idx)
File "D:\anaconda\envs\Basicts\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\pycharmproject\BasicTS-master\basicts\archs\arch_zoo\mtgnn_arch\mtgnn_layers.py", line 192, in forward
s1, t1 = adj.topk(self.k, 1)
RuntimeError: selected index k out of range

这里k值报错，topk存在问题吗？没有研究明白，麻烦指点下，感谢！其他时空模型可以运行，唯独MTGNN报错。

请教一下关于loss部分的修改

我目前将其他模型移植到BasicTS,但是面临问题是移植模型的loss部分需要多个损失项的相加，loss = loss1(data1)+loss2(data2)+....，但BasicTS框架中losses.py由于自定义损失函数被包装了，传参只有input_data, target_data两个，是否有解决方案？

Can you add a prediction module?

We want to test with a trained model, but the code does not provide this functionality.

Learning rate

Hi, thanks for this repo! I am trying to run the MLP in example code, and in MLP_METR-LA.py the optimizer is adam. However when I run it the learning rate seems to be reduced to a half after every few epochs. Could you please help me with that? Where can I customize my learning rate? Thanks very much!

example_training_log.txt

Add Main Results.

How to implement single-step forecasting in DGCRN？

Why is my single-step prediction accuracy low?

关于WAPE评价指标

Zezhi Shao,

再次感谢您开源的代码！

您好，在此咨询您一些问题。

为什么额外选择WAPE作为评价指标？

WAPE和WMAPE公式分别如下：


从公式来看，WAPE相对于WMAPE少了权重项，是不是可以理解是WMAPE的特殊形式？

此外，在时间序列预测的背景下，WAPE作为评价指标又有哪些好处?
是不是可以降低对于预测数值的敏感性，尤其在多变量时间序列预测任务上，不同变量其数值上下限可能有很大差异，是不是用WAPE可以降低这些影响？

期待您的回复！
祝您万事顺遂！

the 'package' argument is required to perform a relative import for '.AGCRN_arch.AGCN'

I encountered this problem while running the code, and I don't know how to solve it. I look forward to your reply, thank you!

Add PEMS03, PEMS07 Datasets.

如何使用测试命令。

Add detailed documentation.

请教一下关于xlsx数据，date部分是如何处理的

air quaility数据中没有date这个字段了，想请教一下现在这个xlsx中每条数据和原始的时间戳之间的对应关系

Data visualization issues

Hello author, how can I export predicted values in BasicTS?I saw scripts/data_ Visualization/data_ Visualization.ipynb, but this program can only see real values, no predicted values。
作者你好，我如何导出预测值？在可视化程序中只能看到真实值的历史数据和未来数据，没有预测值的程序。感谢回答！

基于BasicTS，如何打造时间序列分类领域的开源库？

您好！看到BasicTS的架构和代码，感觉到非常规范、清晰和优雅，因此想要fork您的库做二次开发，以适用于时间序列分类及其相关应用领域。

经过阅读BasicTS主页的介绍和部分代码，我总结了以下需要修改的点，不知道理解的对不对，冒昧打扰，希望能和您请教：

1. 数据预处理

• 由于交通流预测的输入是[B, L, N, C]，而时间序列分类中的输入是[B, L, C]，因此在预处理时需要新增一个N维，且N=1。这样的话，是否可以不修改模型代码就适配已有的时序预测Baselines？

2. 训练过程

• 由于交通流预测属于regression任务，时间序列分类属于classification任务，可重写cross-entropy loss在losses.py中。
• 由于forward过程无需客制化，因此无需重写runner吗？

3. 可视化

• 参考这个

4. 自动化调参，也方便记录实验结果

• 想要使BasicTS支持wandb方便调参，但不知道实验结果记录在哪里（似乎是在base_tsf_runner.py这里么？）
• 如果需要记录实验结果的metrics以及不同CFG，是否要改写runner呢，方便问下如何改写吗？

感谢作者~ 辛苦了！

Out of memory when handling large scale datasets.

https://github.com/zezhishao/BasicTS/blob/f02703912318f334510bfe4d75464ad5c7617dec/basicts/runners/base_tsf_runner.py#L291

Add Long-Term Time Series Forecasting Datasets & Add Long-Term Time Series Forecasting Baseline.

Identify by easytorch version instead of throwing an exception.

https://github.com/zezhishao/BasicTS/blob/f8c7116f6d3739244b186c90d8e2b840f536a303/basicts/launcher.py#L21

use wandb

I would like to ask the author how to proceed if I want to use wandb to record losses in the BASICTS framework, thanks

🐛

https://github.com/zezhishao/BasicTS/blob/5639ebbf3c1de171d3747654c02a34a50d65cbea/basicts/runners/base_runner.py#L171

🐛：Start from 1.

https://github.com/zezhishao/BasicTS/blob/5639ebbf3c1de171d3747654c02a34a50d65cbea/examples/MLP/MLP_METR-LA.py#L100

Clear Metrics Description

With CFG.NULL_VAL = 0.0, three metrics (MAE, RMSE, MAPE) are defaulted as masked ones. Maybe you should use Masked XXX instead of XXX in experimental results?

Add more baselines.

General Time Series Forecasting:

• DeepAR
• LightGBM
• NBeats
• NHiTS ✅
• GRU

Spatial-Temporal Forecasting:

• STGODE ✅
• STSGCN❌, STFGCN❌ (They are built on MXNet)
• SCINet

Long Time Series Forecasting:

• TimesNet
• Vanilla Transformer
• GPTs for Time Series

Train problem,，停在这里没反应

2022-11-02 11:05:17,415 - easytorch - INFO - Building model.
C:\Users\luo\anaconda3\envs\basicts\lib\site-packages\torch\cuda_init_.py:143: UserWarning:
NVIDIA GeForce RTX 3050 Ti Laptop GPU with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the NVIDIA GeForce RTX 3050 Ti Laptop GPU GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
2022-11-02 11:05:17,481 - easytorch-training - INFO - Initializing training.
2022-11-02 11:05:17,482 - easytorch-training - INFO - Set clip grad, param: {'max_norm': 5.0}
2022-11-02 11:05:17,482 - easytorch-training - INFO - Building training data loader.
train len: 23974
2022-11-02 11:05:17,583 - easytorch-training - INFO - Set optim: Adam (
Parameter Group 0
amsgrad: False
betas: (0.9, 0.999)
eps: 1e-08
lr: 0.002
weight_decay: 1e-05
)
2022-11-02 11:05:17,584 - easytorch-training - INFO - Set lr_scheduler: <torch.optim.lr_scheduler.MultiStepLR object at 0x000001F12B3E4908>
2022-11-02 11:05:17,590 - easytorch-training - INFO - Initializing validation.
2022-11-02 11:05:17,590 - easytorch-training - INFO - Building val data loader.
val len: 3425
test len: 6850
2022-11-02 11:05:17,866 - easytorch-training - INFO - Epoch 1 / 100
0%| | 0/750 [00:00<?, ?it/s]

overall MAPE for STNorm on PEMS07

Thanks for sharing your excellent work!

When I estimate the performance of my work with BasicTS, I found the overall MAPE of STNorm on PEMS07 is extremely low to 8.00%, which is almost the MAPE of third horizon(7.98%). Also, I cannot reproduce the performance but get overall 8.85% for MAPE.

Is it possible that it happened to be a little digital mistake here?

Thanks again for your nice work!

关于easytorch的问题

你好，
我想问一下，我安装了easytorch，但是我运行的时候还是会显示：

from easytorch.easytorch import launch_training
ImportError: cannot import name 'launch_training' from 'easytorch.easytorch'

我尝试了好多，还是没有成功。想请教一下作者这到底是怎么回事。

请教下MAE问题

作者您好，BasicTS中时空预测12步的MAE，是每一步的MAE，还是多步MAE的平均值？比如说，第12步的MAE是，单纯12步的MAE，还是1步-12步MAE的平均值？

Question about multi-gpus.

Hi, I'm using BasicTS to run some models. However, when I use multiple GPUs, if there exists some unused parameters such as some constant variable, I will get the error like RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=Truetotorch.nn.parallel.DistributedDataParallel, and by making sure all forwardfunction outputs participate in calculating loss. If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module'sforwardfunction. Please include the loss function and the structure of the return value offorwardof your module when reporting this issue (e.g. list, dict, iterable). Parameter indices which did not receive grad for rank 1: 48 49 50 51 52 53 In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error
However, I can't find how to set the 'find_unused_parameters=True' in BasicTS. Could you do me a favor?
Thanks alot

How to save the prediction result on training set and test set?

Great job! But I didn't find how to save the specific prediction results in config.

No module named '.home'

I run this :python examples/run.py -c examples/GWNet/GWNet_METR-LA.py --gpus '0',
but it returned ' No module named '.home''

I want to ask you the reason that the acc of single-step prediction is not necessarily better than the acc of multi-step prediction at Horizon 1. And Why the acc of multi-step prediction at Horizon 1 does not decline with iteration.

          I want to ask you the reason that the acc of single-step prediction is not necessarily better than the acc of multi-step prediction at Horizon 1.  And Why the acc of multi-step prediction at Horizon 1 does not decline with iteration.

Originally posted by @zhishiouo in #61 (comment)

pycharm can't debug

你好，非常感谢你的工作，我在使用pycharm连接远程服务器debug代码的时候，会卡住，但是用vscode就没问题，想请教一下是什么原因。例如以下这种情况：

我在stid_arch.py文件下打了一个断点，然后运行example/run.py，得到上图的结果，但是我点击步过就卡住了，如下图所示


用vscode就不存在这个问题，这是怎么回事呢？

Feature Needed✨：Support Dataset Args.

对于实验结果，有几点疑问

1. 两个任务的结果都没有具体说明历史步数和预测步数各是多少，或是平均结果？
2. Long Time- Series Forecasting结果中MAE很大，而论文中MAE，MSE都是小于1的。

How to use bash code to tune hyperparameters?

Hello, congrats for the great work! I was trying to do hyperparameter tuning of my model, using a sh file to pass the arguments into the py file (using argparse).

However, if I have a script, say "run.sh", it will run the "example/run.py", and then "example/run.py" will try to get the hyperparameters in "examples/Triformer/Triformer_ETTh1.py".

Now I want to pass the hyperparameters from "run.sh" to "examples/Triformer/Triformer_ETTh1.py". But it seems I can only pass arguments to the "example/run.py". Since the parameters are stored in "examples/Triformer/Triformer_ETTh1.py", which is the file I want to modify, could you kindly suggest on that? Thanks very much for your time!

请教测试结果最佳模型指标问题

您好！十分感谢您的工作！关于模型测试结果有点疑问想请教下：

模型最终训练完成时如何显示最佳预测精度呢？即best model result的结果。目前BasicTS的log中是否是根据每个epoch找到val_mae最低的模型保存，然后用该checkpoint做多步预测计算平均test MAE呢？
如何在整个训练结束时打印出整个训练过程中val_mae最低模型的测试集预测精度指标呢？

例如下图所示，我发现在训练最后一轮时显示'Evaluate best model on test data'的结果并不是val_mae最低的模型的结果：
训练最后一轮时：

训练过程中间某一步：

请问在BasicTS中是如何确定当前的best model以及如何查看最终的评价结果呢？

期待您的回复！

Errors running STGCN, GMAN and STNorm

Thank you for providing an excellent easy-to-use library for spatial-temporal prediction.

But I found some running errors and now report them to you.

• STGCN
  KeyError: "No object named 'STGCN' found in 'Arch' registry!"

• GraphWaveNet
  RuntimeError: Expected 2D (unbatched) or 3D (batched) input to conv1d, but got input of size: [64, 32, 307, 13]

• STNorm
  RuntimeError: Expected 2D (unbatched) or 3D (batched) input to conv1d, but got input of size: [64, 32, 307, 13]

The above errors can be reproduced using the PEMS04 dataset.

Looking forward to your reply.

[TODO: 清理注释文档] 数据集问题

PEMS04 08 这类数据集原始数据是包括三个通道flow, speed, occupancy。在处理数据集的时候，只取了第一通道也就是flow，然后在脚本描述中出现了矛盾，所以我不是很清楚，在PEMS04 08这些数据集上，默认是预测速度还是流量

• Channels (features): three channels [traffic flow, time of day, day of week]
• Target: predict the traffic speed of the future 12 time steps.here