Series Stationarization implementation about itransformer

[lucidrains]

@ben10ben now you can run the experiment side by side with the change of a hyperparameter

share your results here! i'm curious if the gamma and beta are necessary

[lucidrains]

@ben10ben hey Benedikt, yeah that looks to be the same as RevIn except for the extra learned gamma and beta

perhaps we can just make it a RevIn setting to turn off the learned affines?

[lucidrains]

let me know if this works

[lucidrains]

@ben10ben on this subject, i still think the most correct way would be to gather up the statistics, project them into a conditioning vector, and use adaptive normalization throughout the transformer. the other way would be to create number tokens and offer them to the transformer

[ben10ben]

yes this works.

Still trying to wrap my head around it, but very much agree, that normalisation information needs to be propagated to the transformer. To my understanding, RevIn would be an alternative implementation to de-stationary attention + Stationarization?

This is my big question for the paper if the iTransformer solves this issue somehow. I wanted to try the Stationarization on its own. Maybe some theoretical understanding on my side is still missing.