Neural Fit identity score low about vision

[qbilius]

I haven't tried such a test. With PLS regression, there is a dimensionality reduction, maybe that's causing it? Try using something simpler, like linear regression, to see if the issue persists. On the other hand, since you're training on some hvm images and testing on some other hvm images, why should it work anyways?

[mschrimpf]

The hvm here is actually the exact same dataset, i.e same images and same everything else.

Good point with the linear regression, I'll see how well that performs.

[qbilius]

Yes, but if you trained your regression on 5 images and tested on some other 5, you wouldn't expect te score to be at 1.0, right? (even though theoretically there is no problem, just use the identity function – the problem is how to do that well enough in general)

[mschrimpf]

Okay that's true, I still would have expected the score to be close to 1.0 especially since there is a trivial solution. I don't have a good intuition why regression doesn't find this trivial solution.

[qbilius]

First check if linear regression has the same problem, as I'm not sure at all that it will.

[mschrimpf]

Linear regression indeed works much better (100% score vs 78% with PLS).
@qbilius have you played around with linear regression for the model activations too by any chance? Or does that become too computationally intense?

[qbilius]

See here: https://paper.dropbox.com/doc/HvM-Neural-Fits-IkDSBHln2Qo2cx8onqqtu#:uid=017383260209745&h2=Comparison-of-regressions