Model-level Parallelism about codegen

[rooa]

@VHellendoorn @boblee22 Changed (dec5101) for better readability, let me know if this clarifies the confusion.

[rooa]

Hi Vincent, the reason for 16B model not fitting into a GPU with sufficiently large RAM was due to not sampling under half precision. We made a change to the sampling code earlier and it got turned off by default. We just pushed a small change (139825f) that (1) turns the half precision on by default and (2) forces half precision on 16B models. Could you pull and try sampling again?
On my end, the model occupies about 33GB during sampling, which fits into a single NVIDIA A100 with 40GB RAM.

[VHellendoorn]

Thanks, that works! Here I was wondering why it needed more than 3 bytes per weight -- figures :) The memory footprint matches what you report now. Thanks again for freely sharing these models.

[boblee22]

Hi, could you please clarify what you mean by forcing half-precision on 16B models?
If I understand it correctly, when the model name starts with "codegen-16B", then no_fp16 = True and therefore fp16 will be disabled?

[rooa]

@boblee22 As you can see, --no_fp16 argument is set to store_false, which will set the value to False when the flag is specified. From the script user perspective this makes sense (setting "no" flag when they don't want to turn fp16 on), hence the name of the variable. However, this causes the confusion like your question, where semantics of the variable name is opposite of how it is used.

We could mitigate by setting the flag to store_true, then declare a new variable like use_fp16 = (not args.no_fp16). But it's as confusing as the current state of code in my opinion.

[VHellendoorn]

To add, I do agree that the current nomenclature is confusing. While the argument is called "no fp16", its boolean is passed (without inversion) to "fp16" here. So it already acts as "use fp16". Might be worth renaming to use_fp16 without changing anything else.