Pandas Exclusive?

#1 · open · 2 comments

View on GitHub ↗

hydrosquall

Thanks for writing this library! When writing scrapers, sometimes a URL in the middle fails because the website needs to have a captcha or something answered before I can continue scraping. The state machine works beautifully, and saved me the trouble of cluttering up my code with defensive measures as I have had to do the past. I was wondering about the decision to write the safe methods specifically for Series and DataFrame object. It's relatively straightforward to wrap some other iterable (i.e. list, set, etc) in a pd.Series(). Would it be possible to extend the standard python `map` function to have this safe feature, in the event that pandas wasn't available on the computer the script was being run on?

Comments

ResidentMario

Yes, it should be possible; after all a `pandas` `Series` is just a fancy iterable. Would I implement that? Well...I use this little shim every so often in my own work; you're the first *other* person I'm aware of using it. For my own purposes I've never needed to checkpoint a pure iterable, so I never implemented it. A maybe better direction to go in would be to integrate this tool into a more popular one (see [engarde#33](https://github.com/TomAugspurger/engarde/issues/33)). But, we'll see. I'll keep this open in the meantime.

ResidentMario

Also, this is a small design consideration but you should almost never overwrite an internal Python name like `map`, and certainly not here. This should probably be an importable `safe_map` function or something like that instead.