compatibility with lora about tokencompose

[zwcolin]

Hi Chengming,

Yes, we have attempted training with lora but encountered performance degradation with default ranks. Therefore, we did not end up having a lora finetuned version.

That said, it still might be possible to finetune with lora, although many hyperparameters (e.g., rank, types of attention weights) likely need to be carefully experimented specifically with our setup, which makes attention map itself an objective to optimize.

Best,
Zirui

[chmxu]

Thank you!