feature map function φ (x) = elu(x) + 1, about fast-transformers

[angeloskath]

Hi,

Thanks for the good words!

The feature map needs to correspond to a non-negative similarity score. That means any <φ(x), φ(y)> >= 0 for all x, y would be usable. Section 3.2.1 in the paper has a few more details and discussion regarding that matter.

Cheers,
Angelos

[burcehan]

I use function elu(x)+1 in the model,I found that the convergence was very slow ,Whether choosing another mapping function can accelerate the convergence? Which mapping function is better?
Thanks for your help.

[angeloskath]

I would say this is an open research problem. In some cases we observed the same (see supplementary section B) in others not. Increasing the dimensions of the query and key (or the number of layers) would make the model slower but increase its representation capacity which might improve the convergence.

For now it depends on the problem, problems that require sparse attention patterns might be harder to learn using this feature map.

[angeloskath]

I am closing this issue since it is not a bug or a new feature.

Feel free to reopen it if you have more questions.