This is a notebook exploring the phenomenon of superposition in neural networks. We ask: how many concepts could we fit in a space the size of a typical transformer model's hidden dimension? And how much superposition interference should we expect for a given number of concepts in a given number of dimensions?

The notebook is exploration.ipynb