error converting a vcf about bio2zarr

[mufernando]

I confirmed removing the PL field it works! For future reference, you can remove it from the schema using jq:

vcf2zarr explode example.vcf.gz example.exploded
vcf2zarr mkschema example.exploded | jq 'del(.fields[] | select(.name == "call_PL"))' > example.json
vcf2zarr encode example.exploded example.zarr --schema example.json

[jeromekelleher]

Thanks for the bug report @mufernando!

I think the problem is with your call_PL field:

DEBUG Initialised <zarr.core.Array '/call_PL' (24739, 746, 66) int32>

Your PL field leads to an inner dimension of 66! 🫢

So, each variant chunk of this array is 10000 * 746 * 66 * 4 which is more than 2147483647. The immediate bug here then is that we're not giving a usable error message. More generally, PL fields are a major problem (#185 and linked discussion) but we know how they should be dealt with.

In the short term, I'd suggest creating a schema and dropping the call_PL field (assuming you're not using it):

vcf2zarr mkschema  chrY_mcc-746-kray-filtered.exploded > schema.json
... edit JSON to remove delete the call_PL field
vcf2zarr encode chrY_mcc-746-kray-filtered.exploded -s schema.json chrY_mcc-746-kray-filtered.zarr

[jeromekelleher]

I just merged a fix which should raise a more helpful error message @mufernando - would you mind trying it out on your data please? The message will need some links to as-yet unwritten docs about PL fields, but hopefully it's pointing people in the right direction.