How to avoid splitting [SEP] about exbert

[bhoov]

Hmm, interesting use case. I did not code in any functionality for users to input any special tokens beyond MASK. Everything typed into the input box is assumed to be tokenized as normal.

I'm not very aware of how QuestionAnswering works. When you insert a [SEP] token, do you want to treat that token as a special token that separates 2 inputs (perhaps question and answer), or do you want to just see what happens when [SEP] is inserted at different points in a sentence?

[danishpruthi]

Thanks for getting back. I want [SEP] to separate the question and passage.

For now, I added a small function that patches together the broken down [SEP] token. Further, I also disabled the meta_from_tokens which also breaks down such tokens. These fixed the problem so far.

I think I might have to change more code so that the token_ids that the model sees are of the form [0, 0,0, ... 0, 1, 1, 1, ... ,1] to denote two sentences.

I don't know about others but I would very much appreciate a version of this repository that only visualizes attention, and therefore directly operates using huggingface tokenizers rather than spacyface aligners.

[bhoov]

I hear you. A large part of the preprocessing for exBERT is tailored to extract linguistic features from each token in a self attention context (i.e., when there is only one input sequence), which does cause situations like the one you have mentioned to break.

Visualizing the attention for pairs of sequences is very possible, and if all you care about is the attention I would point you to https://github.com/jessevig/bertviz.

[danishpruthi]

Yeah, I actually tried BertViz, since I only needed attention visualization, however BertViz doesn't scale well (jessevig/bertviz#26), and is extremely slow (to the point that it doesn't work) for sequences of length 200+.