Will you share pre-training code of StructBERT about alicemind

[wangwei7175878]

We currently have no plan to open source pre-training code, please refer to the paper for more details.

[kaansonmezoz]

Thanks for the clarification @wangwei7175878 .

I've tried to implement my own version of Word Structural Objective (WSO).
There were some parts in the paper unclear to me, so I had to come-up with my own solutions.

Could you please explain more about the following questions ?

• Is it possible for a sentence to have more than one trigrams ?
• Since WSO is a single-sentence task, did you calculate WSO loss from a real single sentence or sequence of sentences ? (i.e Did you predict shuffled tokens from [CLS] SentA [SEP] or [CLS] SentA. SentB. SentC [SEP])
• Did you use same head for MLM and WSO (i.e Huggingface's BertOnlyMLMHead)? Since MLM and WSO are predicting tokens in the same way with different labels, MLM takes labels with masked tokens and WSO takes labels with shuffled tokens, same head can be used to predict token ids.

[wangwei7175878]

1. Yes, there can be more than one trigrams. Section 2.4: "5% of trigrams are selected for random shuffling"
2. We calculate WSO loss from sequence of sentences.
3. Yes, we use same head for MLM and WSO

[kaansonmezoz]

Thanks for the explanation @wangwei7175878

[luofuli]

Close this issue since it has been solved.