How about not sharing point-level queries in the one2many assigner about maptr

[LegendBC]

Hi @JingweiZhang12 , thanks for your interest!

1. We have not tried isolated point queries for one2many group. One2one and one2many are both at the instance level. Since we think that the point-level queries do not provide leaked information from auxiliary instances. All the point-level interactions are performed intra-instance. We only apply the self-attn mask on inter-instance self-attn to prevent data leakage.
2. In TABLE 5, the vanilla self-attention experiment uses the same setting as others.