No option for rotary positional embeddings about hourglass-transformer-pytorch

[lucidrains]

oh no way, can you point to me where in his code that he does this?

[lucidrains]

would be happy to add it as an option if so

[vvvm23]

I didn't actually look at their original implementation, I just saw this detail in the paper.

My Jax knowledge isn't great, but I think they probably did it by changing the attention wrapper used when initialising the transformer here with something like this.

[vvvm23]

Relevant excerpt from paper:

We replace relative attention in vanilla layers by LSH attention (...), which allows us to handle 12288-long sequences. To achieve relative attention parametrization, the LSH attention is combined with rotary positional embeddings (...). In shortened layers, standard relative attention is used.

My pull request doesn't actually handle LSH attention though 🤔