nixtla / neuralforecast

Scalable and user friendly neural :brain: forecasting algorithms.

Nixtla    

TimeGPT-1

The first foundation model for forecasting and anomaly detection

TimeGPT is a production ready, generative pretrained transformer for time series. It's capable of accurately predicting various domains such as retail, electricity, finance, and IoT with just a few lines of code 🚀.

📖 Table of Contents

• Quick Start
• Installation
• Forecasting with TimeGPT
• Anomaly Detection
• Zero-shot Results
• How to Cite
• Features and Mentions
• License
• Get in Touch

🚀 Quick Start

QuickstartTGPT.mp4

Install nixtla's SDK

pip install nixtla>=0.5.1

Import libraries and load data

import pandas as pd
from nixtla import NixtlaClient

Forecast using TimeGPT in 3 easy steps

# Get your API Key at dashboard.nixtla.io

# 1. Instantiate the NixtlaClient
nixtla_client = NixtlaClient(api_key = 'YOUR API KEY HERE')

# 2. Read historic electricity demand data 
df = pd.read_csv('https://raw.githubusercontent.com/Nixtla/transfer-learning-time-series/main/datasets/electricity-short.csv')

# 3. Forecast the next 24 hours
fcst_df = nixtla_client.forecast(df, h=24, level=[80, 90])

# 4. Plot your results (optional)
nixtla_client.plot(df, timegpt_fcst_df, time_col='timestamp', target_col='value', level=[80, 90])

Anomaly detection using TimeGPT in 3 easy steps

# Get your API Key at dashboard.nixtla.io

# 1. Instantiate the NixtlaClient
nixtla_client = NixtlaClient(api_key = 'YOUR API KEY HERE')

# 2. Read Data # Wikipedia visits of NFL Star (
df = pd.read_csv('https://raw.githubusercontent.com/Nixtla/transfer-learning-time-series/main/datasets/peyton_manning.csv')


# 3. Detect Anomalies 
anomalies_df = nixtla_client.detect_anomalies(df, time_col='timestamp', target_col='value', freq='D')

# 4. Plot your results (optional)
nixtla_client.plot(df, anomalies_df,time_col='timestamp', target_col='value')

🤓 API support for other languages

Explore our API Reference to discover how to leverage TimeGPT across various programming languages including JavaScript, Go, and more.

🔥 Features and Capabilities

• Zero-shot Inference: TimeGPT can generate forecasts and detect anomalies straight out of the box, requiring no prior training data. This allows for immediate deployment and quick insights from any time series data.

• Fine-tuning: Enhance TimeGPT's capabilities by fine-tuning the model on your specific datasets, enabling the model to adapt to the nuances of your unique time series data and improving performance on tailored tasks.

• API Access: Integrate TimeGPT seamlessly into your applications via our robust API. Upcoming support for Azure Studio will provide even more flexible integration options. Alternatively, deploy TimeGPT on your own infrastructure to maintain full control over your data and workflows.

• Add Exogenous Variables: Incorporate additional variables that might influence your predictions to enhance forecast accuracy. (E.g. Special Dates, events or prices)

• Multiple Series Forecasting: Simultaneously forecast multiple time series data, optimizing workflows and resources.

• Custom Loss Function: Tailor the fine-tuning process with a custom loss function to meet specific performance metrics.

• Cross Validation: Implement out of the box cross-validation techniques to ensure model robustness and generalizability.

• Prediction Intervals: Provide intervals in your predictions to quantify uncertainty effectively.

• Irregular Timestamps: Handle data with irregular timestamps, accommodating non-uniform interval series without preprocessing.

📚 Documentation with examples and use cases

Dive into our comprehensive documentation to discover examples and practical use cases for TimeGPT. Our documentation covers a wide range of topics, including:

• Getting Started: Begin with our user-friendly Quickstart Guide and learn how to set up your API key effortlessly.

• Advanced Techniques: Master advanced forecasting methods and learn how to enhance model accuracy with our tutorials on anomaly detection, fine-tuning models using specific loss functions, and scaling computations across distributed frameworks such as Spark, Dask, and Ray.

• Specialized Topics: Explore specialized topics like handling exogenous variables, model validation through cross-validation, and strategies for forecasting under uncertainty.

• Real-World Applications: Uncover how TimeGPT is applied in real-world scenarios through case studies on forecasting web traffic and predicting Bitcoin prices.

🗞️ TimeGPT-1: Revolutionizing Forecasting and Anomaly Detection

Time series data is pivotal across various sectors, including finance, healthcare, meteorology, and social sciences. Whether it's monitoring ocean tides or tracking the Dow Jones's daily closing values, time series data is crucial for forecasting and decision-making.

Traditional analysis methods such as ARIMA, ETS, MSTL, Theta, CES, machine learning models like XGBoost and LightGBM, and deep learning approaches have been standard tools for analysts. However, TimeGPT introduces a paradigm shift with its standout performance, efficiency, and simplicity. Thanks to its zero-shot inference capability, TimeGPT streamlines the analytical process, making it accessible even to users with minimal coding experience.

TimeGPT is user-friendly and low-code, enabling users to upload their time series data and either generate forecasts or detect anomalies with just a single line of code. As the only foundation model for time series analysis out of the box, TimeGPT can be integrated via our public APIs, through Azure Studio (coming soon), or deployed on your own infrastructure.

⚙️ TimeGPT's Architecture

Self-attention, the revolutionary concept introduced by the paper “Attention is all you need“, is the basis of the this foundational model. The TimeGPT model is not based on any existing large language model(LLMs). It is independently trained on vast timeseries dataset as a large transformer model and is designed so as to minimize the forecasting error.

The architecture consists of an encoder-decoder structure with multiple layers, each with residual connections and layer normalization. Finally, a linear layer maps the decoder’s output to the forecasting window dimension. The general intuition is that attentionbased mechanisms are able to capture the diversity of past events and correctly extrapolate potential future distributions.

TimeGPT was trained on, to our knowledge, the largest collection of publicly available time series, collectively encompassing over 100 billion data points. This training set incorporates time series from a broad array of domains, including finance, economics, demographics, healthcare, weather, IoT sensor data, energy, web traffic, sales, transport, and banking. Due to this diverse set of domains, the training dataset contains time series with a wide range of characteristics

For the Zero-shot Results section of your README, you can enhance the clarity and effectiveness by focusing on emphasizing the key findings and their implications, while also making the text more concise and digestible. Here's a refined version:

---

⚡️ Zero-shot Results

Accuracy:

TimeGPT has been tested for its zero-shot inference capabilities on more than 300K unique series, which involve using the model without additional fine-tuning on the test dataset. TimeGPT outperforms a comprehensive range of well-established statistical and cutting-edge deep learning models, consistently ranking among the top three performers across various frequencies.

Ease of use:

TimeGPT also excels by offering simple and rapid predictions using a pre-trained model. This stands in stark contrast to other models that typically require an extensive training and prediction pipeline.

Efficiency and Speed:

For zero-shot inference, our internal tests recorded an average GPU inference speed of 0.6 milliseconds per series for TimeGPT, which nearly mirrors that of the simple Seasonal Naive.

📝 How to cite?

If you find TimeGPT useful for your research, please consider citing the associated paper:

@misc{garza2023timegpt1,
      title={TimeGPT-1}, 
      author={Azul Garza and Max Mergenthaler-Canseco},
      year={2023},
      eprint={2310.03589},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

🎉 Features and Mentions

TimeGPT has been featured in many publications and has been recognized for its innovative approach to time series forecasting. Here are some of the features and mentions:

• TimeGPT Revolutionizing Time Series Forecasting
• TimeGPT: The First Foundation Model for Time Series Forecasting
• TimeGPT: Revolutionising Time Series Forecasting with Generative Models
• TimeGPT on Turing Post
• TimeGPT Presentation at AWS Events
• TimeGPT: Machine Learning for Time Series Made Accessible - Podcast
• TimeGPT on The Data Exchange
• How TimeGPT Transforms Predictive Analytics with AI
• TimeGPT: The First Foundation Model - AI Horizon Forecast

🔖 License

TimeGPT is closed source. However, this SDK is open source and available under the Apache 2.0 License. Feel free to contribute.

📞 Get in touch

For any questions or feedback, please feel free to reach out to us at ops [at] nixtla.io.

neuralforecast's Issues

NBEATS: Incompatibility of n_input < n_output with outsample_mask

When the n_input < n_output, the _NBEATS forward does incorrect wrangling of the outsample_mask, the dimensions of outsample_y and outsample_mask don't match.

forecast method: support for time series with different datestamps

Is your feature request related to a problem? Please describe.
Right now the forecast method assumes:

• All series end with the same timestamp.
• The datestamp column (ds) of the input dataframe (Y_df) is a well-formatted string.

To train a model these assumptions are not necessary. (A user can train using ints or floats as the ds column).

Describe the solution you'd like
Consider these scenarios in the forecast method.

Example notebooks

Is your feature request related to a problem? Please describe.
I see some cool work here, but it's hard to understand how to use the project without some examples.

Describe the solution you'd like
Add some example notebooks showing off how to use the different features in the project.

Describe alternatives you've considered
any other form of documentation

Additional context

Issue 1

Missing import pandas forecast method

Is your feature request related to a problem? Please describe.
The nhits' forecast method needs pandas to run properly. I think this is a silent bug because in the nbs , pandas is imported to test the method but it is not exported.

Describe the solution you'd like
Importing pandas from the beginning should solve the problem.

Anti-nan protection in MASS Loss is masking all-nan forecast tensors

There is a difference in the anti-nan protection for the MASE Loss function used by ElementAI and the one we are using:

ElementAI only uses the divide_no_nan function over the scale factor (https://github.com/ElementAI/N-BEATS/blob/04f56c4ca4c144071b94089f7195b1dd606072b0/common/torch/losses.py#L61):

masep = t.mean(t.abs(insample[:, freq:] - insample[:, :-freq]), dim=1)
masked_masep_inv = divide_no_nan(mask, masep[:, None]) <--- Anti-nan protection only used over the scale factor
return t.mean(t.abs(target - forecast) * masked_masep_inv) <--- No anti-nan protection for the forecast

Our MASELoss function is hiding the nans from the forecast (https://github.com/Nixtla/nixtlats/blob/a3c7442a4c16c255685e158c9347d045f87ffa3b/nixtlats/losses/pytorch.py#L160):

delta_y = t.abs(y - y_hat)
scale = t.mean(t.abs(y_insample[:, seasonality:] -
y_insample[:, :-seasonality]), axis=1)
mase = divide_no_nan(delta_y, scale[:, None]) <--- Anti-nan protection masks nans coming from the scale and the forecast
mase = mase * mask
mase = t.mean(mase)

This difference causes a silent bug by setting the loss to zero during training/validation when the forecasts are meaningless.

GPU Usage MQESRNN

Trying tu run mqesrnn using GPU the following error appears,

get_default_mask returns dataframe with different index

The get_default_mask function from the data.tsdataset module returns a dataframe with a different index than the original Y_df. I think the index should be the same.

GPU usage in inference/forecast phase

Is your feature request related to a problem? Please describe.
It would be a great feature to have gpu usage in inference/forecast phase to reduce processing time in production

Describe the solution you'd like
I think a possible solution could be:

@patch
def forecast(self: model, Y_df, X_df, S_df, batch_size, **trainer_kwargs):
       ...
       trainer = pl.Trainer(**trainer_kwargs)
       ....

So the user could simply use:

model.forecast(Y_df, X_df, S_df, batch_size, gpus=4)

Add logSparse as an attention option for Informer

Is your feature request related to a problem? Please describe.
add logSparse attention as an option for the informer model. I've experienced improved results over prob attention. It's a drop-in replacement

Describe the solution you'd like

class LogSparseAttention(nn.Module):
""" https://arxiv.org/abs/1907.00235
"""
    def __init__(self, mask_flag=True, factor=5, scale=None, attention_dropout=0.1, output_attention=False):
        super(LogSparceAttention, self).__init__()
        self.scale = scale
        self.mask_flag = mask_flag
        self.output_attention = output_attention
        self.dropout = nn.Dropout(attention_dropout)

    def log_mask(self, win_len, sub_len):
        mask = torch.zeros((win_len, win_len), dtype=torch.float)
        for i in range(win_len):
            mask[i] = self.row_mask(i, sub_len, win_len)
        return mask.view(1, 1, mask.size(0), mask.size(1))

    def row_mask(self, index, sub_len, win_len):
        log_l = math.ceil(np.log2(sub_len))

        mask = torch.zeros((win_len), dtype=torch.float)
        if((win_len // sub_len) * 2 * (log_l) > index):
            mask[:(index + 1)] = 1
        else:
            while(index >= 0):
                if((index - log_l + 1) < 0):
                    mask[:index] = 1
                    break
                mask[index - log_l + 1:(index + 1)] = 1  # Local attention
                for i in range(0, log_l):
                    new_index = index - log_l + 1 - 2**i
                    if((index - new_index) <= sub_len and new_index >= 0):
                        mask[new_index] = 1
                index -= sub_len
        return mask

    def forward(self, queries, keys, values, attn_mask):
        B, L, H, E = queries.shape
        _, S, _, D = values.shape
        scale = self.scale or 1./sqrt(E)

        scores = torch.einsum("blhe,bshe->bhls", queries, keys)
    
        mask = self.log_mask(L, S)
        mask_tri = mask[:, :, :scores.size(-2), :scores.size(-1)]
        scores = scores.to(queries.device)
        mask_tri = mask_tri.to(queries.device)
        scores = scores * mask_tri + -1e9 * (1 - mask_tri)

        A = self.dropout(torch.softmax(scale * scores, dim=-1))
        V = torch.einsum("bhls,bshd->blhd", A, values)

        if self.output_attention:
            return (V.contiguous(), A)
        else:
            return (V.contiguous(), None)

Describe alternatives you've considered

Additional context
https://arxiv.org/abs/1907.00235
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y. X., & Yan, X. (2019). Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. arXiv preprint arXiv:1907.00235.

add support for windows-latest CI

NBEATS: Zero inflated validation loss with shared Loss class.

The reason why we used to have the numpy validation loss was to deal with the zeros in the mask.
In the numpy loss zeros are not counted as zero they are just excluded, while the pytorch loss just masks the gradients by multiplying the entries.

Datasets: openpyxl >> xlrd xlrd cannot parse GEFCom

To avoid making a very complex environment, we could try to keep only the needed dependencies.
The openpyxl seems more complete to deal with excel sheets we could avoid xlrd.

DL: unbalanced panel and available_mask interactions

Partial work was done on unbalanced panels and protections of the dataloader with the available_mask
It would be beneficial to finish the experiment utility to balance the panel and make the needed tests of the available_mask use within the dataloader.

ESRNN: RNNs predict should filter predictions with sample_mask

ESRNN and RNNs, in general, need a warmup period before sample_mask==1.

The predictions should filter using the sample_mask ex-post. Dataloader should send all the windows so that all sampleable_mask gets predictions.

How to contribute instructions

I think it might be helpful to have a How to contribute document, particularly working with nbdev.

TCN model is missing from documentation

Is your feature request related to a problem? Please describe.
The TCN model should be listed under model the docs.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

ESRNN: Seasonality induced leakage + assert seasonality>1

The predict method uses seasonality from future observations if seasonality<n_out
For example: if predictions for 2 weeks, seasonality 7 will look at the Friday of last week (that comes from the future).

The solution is to add Naive2 values for seasonality in the prediction method.

Datasets: Tests for dataframe balance

From the experience of the other ESRNN repo, a test for the dataframe balance can save users a lot of debugging efforts.

Test for batch construction fails on Windows

When running test function test_batch_construction_windows in data__tsdataset.ipynb notebook folowing error appears.

I'm using python enviroment in anaconda3 on Windows.

Forecast method default X_df, S_df

Is your feature request related to a problem? Please describe.
Even if you don't use temporal exogenous variables and static variables you need to declare them as None in the model.forecast method.

Describe the solution you'd like
I think it would be good set the default to None.

Auto correlation test in statistical tests notebook?

Rather than having the autocorrelation test on the Naive2 model, it would be convenient to declare a test folder and specialized notebooks.

forecast method exogenous variables

what should be the format of the exogenous variables using the forecast method? does the method use the future?

TSDataset: Conditional on changing len_sample_chunk for window_size rethink 'simple/full' mode.

Mode will either return the full series, or single windows.
mode==single: returns single windows
mode==full: returns the whole series

The 'full' current mode is not intelligent, as it only breaks if window_size inputs are not satisfied.
NLP datasets have a long tradition of intelligently wrangling the time series.

Features to speed up training

Recently, we found a couple of tricks to speed up the training processes:

• Data Parallelism: torch.nn.DataParallel(module) (https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html). This function automatically splits the input in the forward pass among the available GPUs and automatically collects and adds up the gradients in the backward pass.
• Automatic Mixed Precision: torch.cuda.amp (https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html). This package provides functions to automatically cast the datatype of the tensors to optimize the threshold between precision and computing speed.

Datasets: Inherit sklearn.preprocessing scalers?

Should we check the preprocessing scalers to see if we can use the predefined sklearn class and inherit it?

data/ vs nbs/data/ folders

The notebooks generate heterogeneous testing data folders.
I suggest we unify them into the nbs/data folder by default.

The .gitignore file already contemplates the nbs/data folder.

EVALUATE: Weighted metrics have numerical error?

During hyperopt evaluation breaks with:
np.average weights sum to zero can't be normalized

The error does not seem to come from a mask full of zeros.

Missing assert statement: Nbeats 'n_hidden' argument

When instantiating a Nbeats model, the init() argument 'n_hidden' must be a list of lists, one per 'stack_type,' so each corresponding set of NBeatsBlock is initialized passing a list as the 'theta_n_hidden' argument.

An assert statement in Nbeats.init() would help the user identify if this argument is properly stated, particularly when running grid routines (hyper optimization or ensembling), as the 'n_hidden' item in the grid needs to be defined as a list of lists of lists.

lecun_normal vs uniform initializer NBEATS

Similar issue as the issue from the original NBEATSx repo:
cchallu/nbeatsx#1
The PyTorch default seems to be uniform and not LeCun normal.

Problem with downloading gefcom datasets on Windows

When running data_datasets__gefcom2012.ipynb and data_datasets__gefcom2014.ipynb following error appear.
When downloading gefcom2012 dataset this error occured.

When downloading gefcom2014 dataset this error occured.

I'm using python in anaconda enviroment on Windows.

Installation issues

Due PyTorch being a requirement, I had to downgrade out of python3.10 somehow, so I tried with conda

I tried this to no avail:

conda create -n nixtla python=3.7  # And also 3.6
conda activate nixtla
conda install -c nixtla neuralforecast

getting library errors such as:

This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

I finally was able to install it after using python=3.7.11 as shown in the documentation (not generic 3.7 as shown above)

Datasets: Homogeneize outputs of the load method

All datasets should give Y_df, X_df, S_df

Datasets: unrar dependency on M4 dataset

It seems that some Linux versions have problems with the unrar dependency.
A potential solution is to change the link to a previously unzipped dataset.

DataLoader: unstable batch_size

batch_size depends on the windows available at each time series, which makes it unstable:
e.g. if series per batch is 1 and one time series is of lenght 19 or 12 each batch size will have
19 or 12 gradient signals.

Datasets: evaluation benchmark example within the dataset

I think it would be extremely handy to have a simple evaluation example for the dataset tasks within the notebook.
Where we import the respective metric and try a simple benchmark, the measurement could be the one at the bottom of the evaluation table in each dataset.

Open AI gym library has a tiny wiki for each RL where they explicitly state what is to "solve" a task.
Example https://github.com/openai/gym/wiki/BipedalWalker-v2.

ERROR downloading M4 data

The logs printed by M4.load claim an error, but the downloaded data is correct.

Add forecast function for transformer-based models

Is your feature request related to a problem? Please describe.
Missing forecast function for transformer-based models.

Describe the solution you'd like
A forecast function for transformer-based models.

env.yml incompatibility: ‘HTMLExporter’ object has no attribute ‘template_path’

It seems that fastai has some dependency problems from the nbconvert library from jupyter
downgrading nbconvert solved the issue, seems to me yaml environment needs version protections.
https://forums.fast.ai/t/htmlexporter-object-has-no-attribute-template-path/80775/15

downgrading the nbconvert library solved the issue
conda install nbconvert=5.6.1

Merge nbeats and nhits PyTorch code

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

M5 Dataset: Missing raw data and references

Missing sample_submission.csv from raw data
Missing reference to original drive link and Kaggle competition.

Documentation: Logos of nixtlats and nixtlaml are not updated

The current logos are the default from FastAI.
Changes are needed in docs/images/company_logo_big.png and docs/images/company_logo.png

Datasets: Use sklearn LabelEncoder to declare S matrix

It seems convenient to use sklearn built-in functions to declare the static variables on all the datasets.
Time considerations and code convenience should be taken into account.

NBEATS forward is returning all-nan forecasts

The forward method from the NBEATS model (https://github.com/Nixtla/nixtlats/blob/a3c7442a4c16c255685e158c9347d045f87ffa3b/nixtlats/models/nbeats/nbeats.py#L402) is returning all-nan forecast tensors. The real issue comes from the forecast method (https://github.com/Nixtla/nixtlats/blob/a3c7442a4c16c255685e158c9347d045f87ffa3b/nixtlats/models/nbeats/nbeats.py#L431) when the argument return_decomposition is set to False. (https://github.com/Nixtla/nixtlats/blob/a3c7442a4c16c255685e158c9347d045f87ffa3b/nixtlats/models/nbeats/nbeats.py#L424)

This issue has only been detected when using MASE Loss as the training loss function.

.yml environment channels

Pytorch and pytorch lightning downloads take a lot of time depending on the download channel, and some HTTP issues arise:

1. Pytorch: CondaHTTPError: HTTP 000 CONNECTION FAILED for URL https://conda.anaconda.org/pytorch/linux-64/pytorch-1.10.2-py3.7_cuda11.3_cudnn8.2.0_0.tar.bz2
2. Pytorch lightning: socket.timeout: The read operation timed out

In the case of pytorch, specifying the channel (-pytorch::pytorch, -pytorch::torchvision, etc.) solved the problem; with pytorch lightning, setting the timeout to a large number (pip install -U --timeout 2000 pytorch-lightning) did the trick.

Missing imports forecast method

**Is your feature request related to a problem? Please describe.
The nhits' forecast method needs TimeSeriesLoader to run properly. I think this is a silent bug because in the nbs , TimeSeriesLoader is imported to test the method but it is not exported.

**Describe the solution you'd like
Importing needed imports from the beginning should solve the problem.

Related to #134.

Missing tests for get_filtered_ts_tensor.

The method get_filtered_ts_tensor from the class TimeSeriesDataset is not tested.

Datasets: Use feather for faster load times on M4

Feather is more than 10 times faster at loading than csvs.

upload demo preview

demo.mp4

TSDataset: Rethink len_sample_chunk

The len_sample_chunk and mode parameters are redundant with the complete_input condition + input_size parameter

The input_size parameter is confusing, as users may think it refers to the input_size of the model (it does not).

• We could rather define new window_size to guarantee that the rolled windows will be of window_size
• output_size will be only be needed for protection purposes and sampleable_conditions

The only thing that len_sample_chunk is doing is padding in a different way, (0s or input + output).