Transformer-based architectures (Vaswani et al., 2017) have achieved state-of-the-art per- formance on several modern natural language processing(NLP) tasks. However, due to the quadratic space and time complexity of the at- tention mechanism, their use for large input se- quences remains limited. In recent years, many architectures have proposed approximations of the vanilla attention mechanism which scale linearly with respect to the input size. How- ever, as shown in (Tay et al., 2020a), these X- formers also introduce inductive biases, which prevent them from performing well on certain long-range NLP tasks, thus raising questions on generalizability. The Perceiver (Jaegle et al., 2021) formulates a transformer-based frame- work which has been empirically shown to con- tain minimal inductive biases for long-range vi- sion tasks, with limited assumptions about the input. However, it has not been tested on text data, especially for long-range tasks. We con- ducted the experiments on Long Range Arena proposed by (Tay et al., 2020a) and observed that Perceiver performs well for long context tasks.
In this project, we focus on the text-based Long-Range Arena tasks, namely Text Classification, Long ListOps and Byte-Level Document Retrieval. To generate the datasets for these tasks, go to /data
and follow the instructions in the README file to generate the .pickle
files.
To reproduce the results in our paper, go to /PIAYN_Pytorch/LRA
and follow the instructions in the README file to run the experiments for each of the models and tasks. To run on Greene, rename the PIAYN_Pytorch
folder to PIAYN
and place it in your /scratch
folder, and then follow the instructions present inside folder LRA.