Comparison to softmax with temperature about sparse-structured-attention

[vene]

Hi @AruniRC (replying here instead of to the email you sent me)

In short, we did not experiment with tuning the temperature. Below, I argue why, even with adjusting the temperature, softmax is qualitatively different from sparse mappings.

You're right that all generalized attention mappings (as defined in our paper), including softmax, indeed support a "temperature" parameter which can be seen as adjusting the strength of the regularizer Omega.

I disagree that T<1 results in a "sparser" softmax output. By sparse I mean with many values EXACTLY zero, and the output of softmax is never sparse! This is because e^t / Z is never zero no matter how low (but finite) you make t.

But in general it is true that decreasing the regularization strength (for any mapping: including softmax, sparsemax, fusedmax etc) makes it approach an argmax. With softmax, this happens only in the limit, but with sparse mappings (sparsemax, fusedmax etc) there is always a finite temperature for which this happens.

We studied the properties of such sparse mappings in more detail in this extended preprint: https://arxiv.org/abs/1901.02324

HTH

[AruniRC]

Thanks for the prompt response.