LLaMA-65 outperforms Chinchilla-70B on all reported benchmarks but BoolQ about llama

[elephantpanda]

I don't think LLaMA is anything special. It's just something Facebook felt like they had to release at this point in order to keep the share price up.

We will see in the next few months that Large Language models are ten a penny

[gdmcdonald]

Looks like it's not inconsistent? It betters Chinchilla in every test but it is beaten by PaLM 540B on a couple.

Table 3: Zero shot performance on common-sense reasoning tasks.

Model	Parameters	BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA
GPT-3	175B	60.5	81.0	-	78.9	70.2	68.8	51.4	57.6
Gopher	280B	79.3	81.8	50.6	79.2	70.1	-	-	-
Chinchilla	70B	83.7	81.8	51.3	80.8	74.9	-	-	-
PaLM	62B	84.8	80.5	-	79.7	77.0	75.2	52.5	50.4
PaLM-cont	62B	83.9	81.4	-	80.6	77.0	-	-	-
PaLM	540B	88.0	82.3	-	83.4	81.1	76.6	53.0	53.4
LLaMA	7B	76.5	79.8	48.9	76.1	70.1	72.8	47.6	57.2
LLaMA	13B	78.1	80.1	50.4	79.2	73.0	74.8	52.7	56.4
LLaMA	33B	83.1	82.3	50.4	82.8	76.0	80.0	57.8	58.6
LLaMA	65B	85.3	82.8	52.3	84.2	77.0	78.9	56.0	60.2

[gyunggyung]

Thank you so much. I'm a big help. May I ask about the model structure, I'll open a separate issue. Let's create a new issue! @gdmcdonald

[jspisak]

Thanks for calling out the benchmark issue - we will make sure this is taken into account for future papers/models. thanks!