Fail to parse compressed sas7bdat file containing uncompressed data about readstat

[vinhdizzo]

Are you sure the file is uncompressed? I explicitly set options compress=yes. Here's what the SAS log shows:

NOTE: The data set LIBFOO.COMBINED_ALL_SUB_00 has 1 observations and 2 variables.
NOTE: Compressing data set LIBFOO.COMBINED_ALL_SUB_00 increased size by 100.00 percent. 
      Compressed is 2 pages; un-compressed would require 1 pages.
NOTE: DATA statement used (Total process time):
      real time           0.03 seconds
      cpu time            0.01 seconds

[evanmiller]

You are right; it is compressed. Am I right in read the data as having the strings "XZXZ1234567"?

[vinhdizzo]

The data elements are "ZXZXZXZ1234567".

[evanmiller]

I've researched this issue a bit more -- it occurs when a nominally compressed file actually contains uncompressed data, usually because the data is highly variable and there would not be any compression savings. This paragraph has more technical information: https://github.com/BioStatMatt/sas7bdat/blob/master/vignettes/sas7bdat.rst#compressed-binary-data-subheader

I am not sure how to identify this kind of uncompressed data sitting in a compressed file. In any event, you are unlikely to see the issue "in the wild" -- I think it just cropped up from compressing a file with only 2 columns and no repeating bytes. (I conjecture if you change the data to "ZZZZZZZ1234567", it will parse OK.)

I'll leave this issue open for now, but it won't be a high priority until I hear that it's affecting real-world usage.

[vinhdizzo]

Sure. I'll try to use haven more often now and will let you know if this issue crops up in any of my data sets. Again, I don't use compress=yes that often so most likely not. Thanks so much for your responsiveness in getting these bugs fixed.

Is it safe to assume that the bug specific to AIX will have to wait for when Hadley has time?

[evanmiller]

I might be able to help with that bug -- have a couple ideas. Let's continue the discussion over there.

[evanmiller]

Fixed in c92f697