resource hog -> .asfreq(DateOffset(seconds=1),method='pad') about pandas

[wesm]

You identified a weakness (which I was aware of) with the way that date ranges are handled-- not a problem for lower frequency data but here it's a problem. I think it's inherently fixable but I need to think about it. I've been thinking recently about using something like Francesc Alted's carray package to implement an efficient (compressed) tickdata data structure-- since tickdata is fairly compressible I expect that it would work quite well.

[wesm]

This ought to be remedied by the datetime64 time in NumPy...once it's arrived. I have some ideas to try to do something about it in the meantime

[wesm]

Not sure exactly how but I appear to have fixed this problem in recent work on DateRange. Note that there's a Second offset in datetools with significantly better performance:

In [5]: s
Out[5]: 
2000-01-03 00:00:00    -1.4477377311
2000-01-04 00:00:00    -0.0657188405266
2000-01-05 00:00:00    -0.240092884503
2000-01-06 00:00:00    0.377959314609

In [6]: %time s.asfreq(DateOffset(seconds=1), method='pad')
CPU times: user 4.68 s, sys: 0.08 s, total: 4.76 s
Wall time: 4.78 s

In [7]: %time s.asfreq(datetools.Second(), method='pad')
CPU times: user 0.33 s, sys: 0.01 s, total: 0.34 s
Wall time: 0.35 s