training data format about nnue-pytorch

[lucasart]

answering my own question: documentation here

sorry for the stupid question. closing.

[Sopel97]

I see you implemented the .bin format already. If you even want to tackle .binpack here's more info https://github.com/nodchip/Stockfish/blob/master/docs/binpack.md. You can also use the nnue_training_data_formats.h file directly (just note that bit from nodchip's code which may or may not be correctly attributed according to GPL), though it's C++ so may need a wrapper of some sort.

[lucasart]

I don't understand the purpose of binpack format, actually. If you just store all positions sequentially, indeed, you can do some compression trick there. But, don't you want to sparsely sample real games to produce the data, rather than take all positions that occur during games? This seems to be done ex-post in this repo. But, I think it's easier to do it ex-ante, and sample with the required frequency to produce a smaller .bin file. That way, you can remove some unnecessary logic on both side (producer -> no binpack stuff needed, consumer -> no sampling needed in training scripts).

For now, I just have a constant (user defined) sampling frequency. Perhaps something like freq*exp(-lambda*ply) would be better?

[Sopel97]

But, don't you want to sparsely sample real games to produce the data

If someone shows that this helps training we could consider it, but as far as I know it has no visible impact. We have an option in the pytorch trainer that skips entries with some probabilities, but also nothing particuarily interesting there, even with skipping like 10 on average. So we take 15x file size reduction instead.

edit. btw. .binpack files are always smaller than .bin even if all positions are unrelated

[glinscott]

I don't understand the purpose of binpack format, actually. If you just store all positions sequentially, indeed, you can do some compression trick there. But, don't you want to sparsely sample real games to produce the data, rather than take all positions that occur during games? This seems to be done ex-post in this repo. But, I think it's easier to do it ex-ante, and sample with the required frequency to produce a smaller .bin file. That way, you can remove some unnecessary logic on both side (producer -> no binpack stuff needed, consumer -> no sampling needed in training scripts).

When training, we do many passes through the dataset, and the sampling skips different positions on each pass through. So we end up seeing all positions in the dataset over a training run. So binpack compression is a big win in terms of data size, since we do want all these positions.