ColumnEncodingUtility sort key using raw encoding about amazon-redshift-utils

[IanMeyers]

Hello,

Thanks for the note. There is in fact a reason to avoid doing this and this is a feature that we recommend for stability. In some cases, a sort key will be used which has a very low cardinality first column, and higher cardinality columns which follow. In these cases, it is possible that when the blocks for columns in the sort key are decompressed, the amount of temporary storage is exhausted, and you could see swapping and/or out of memory errors. By electing to not compress the first column, the cluster will allocate enough space for the subsequent encoded blocks to be decompressed, and we avoid these issues.

HTH,

Ian

[jrdi]

Hi Ian,

Thank you for your reply, it's a clarifying explanation. What do you think about adding an option to disable this feature?

[IanMeyers]

Yeah, open to that. If you had a sort key whose size in blocks was roughly the same for each column, then it should be fine.

[IanMeyers]

I'm going to close this now, but if you can send a pull request that's great.