Hangs for large sas7bdat about readstat

[evanmiller]

A sample file would be great.

[BioStatMatt]

Swapping?

On Sat, May 9, 2015 at 2:11 AM, Evan Miller [email protected]
wrote:

A sample file would be great.

—
Reply to this email directly or view it on GitHub
#26 (comment).

[quantbabies]

Shared a dropbox via email.

[evanmiller]

Got it. This should speed things up:

2b8c161

[saurfang]

Just a quick thought: because sas7bdat is fixed length page based, the reading is easily parallelizable. I was successful implementing this using parso on Hadoop but I recognize it might be an overkill for an import task (presumably you don't import very often).

[evanmiller]

@saurfang I like the idea, but a parallel implementation probably isn't a great fit for this library, since the callbacks would have to worry about execution order and thread safety, and for large files you'd have to worry about disk contention.

[evanmiller]

@quantbabies Closing the issue -- please re-open if you still experience hangs.