VarBuilder from custom VarMap causes OOM about candle

[LaurentMazare]

Could you provide details on the size of the weights that you're loading as well as the memory usage that you observe?

[EricLBuehler]

I ran the following command: cargo run --release --features cuda --example mistral -- --prompt "Write a 3 paragraph essay."

It exhausted the memory on a 24GB A10 after starting at a normal level in a few seconds.

[LaurentMazare]

I think that the issue is likely just that you use a varmap so the model will keep track of the gradient so that you can do a backprop. Maybe you just want to use a HashMap of tensors (and not vars), you can build var-builders out of this.

[EricLBuehler]

Wow, I can't believe I didn't see that. Thank you!