f-dangel / deepobs Goto Github PK

Make the quadratic testproblem compatible with BackPACK by using Linear and MSELoss.

Important This will change the loss for a single sample to 1 / dim (x−𝜃)ᵀ H (x − 𝜃) where dim denotes the dimensionality of the input (default: 100). This could affect the baselines.

Currently, the problem is given by 0.5 (x−𝜃)ᵀ H (x − 𝜃), minimizing over 𝜃. The Hessian is PSD and can be decomposed as H = RRᵀ. One can rewrite this problem as regression:

• Sample data points x from the quadratic problems and use 0 as label
• Use the neural network
  • nn.Linear: x → z = −I x + 𝜃
  • nn.Linear: z → f = Rᵀ z + 0
  • Only trainable parameter: 𝜃
• Loss function (dim denotes the dimensionality of f):
  • nn.MSELoss: f → ℓ = (f − 0)ᵀ (f − 0) = (x−𝜃)ᵀ H (x − 𝜃) / dim

Notes:

• Both linear layers can be combined into a single one, but then the gradient w.r.t its bias would not correspond to the gradient w.r.t 𝜃.
• MSELoss in torch does not use the factor 0.5. It would be discarded.
• MSELoss in torch averages over the inputs. Hence,there is an additional scale of 1 / dim