gurgeous / httpdisk Goto Github PK

➜  mm-sales-db git:(master) ✗ → gem install httpdisk
➜  mm-sales-db git:(master) ✗ → httpdisk https://s3.amazonaws.com/irs-form-990/201932849349300103_public.xml
httpdisk: uninitialized constant HTTPDisk::Cache::Tempfile

This also occurs when using the library directly. It's easy enough in my own project to require 'tempfile', but it seems like httpdisk should require it itself.

Thanks!

Thanks for this gem. It's been super helpful in a scraping project I've been fiddling with this weekend.

An Exception will get raised when charset is nil, even when utf8: true option is enabled. For example when Content-Type: text/xml as in https://s3.amazonaws.com/irs-form-990/201932849349300103_public.xml

There are several ways to handle this, I'm not sure your preference, but likely changing

I'm happy to do a PR, but maybe it's simple enough for your to take a quick peak.

Thanks!

I was reading about Redis Streams the other day and it got me thinking about the power of Redis in general. In particular, I thought about an army of sidekiq workers crawling some sires for weeks or months, sharing caches via Redis.

Would you be interested in an extracted interface for cache backends? Similar (or the same as) https://api.rubyonrails.org/classes/ActiveSupport/Cache/Store.html?

I was using httpdisk-grep to clean up some failures and ran into this issue.

httpdisk-grep -h "HTTPDISK 403"
httpdisk-grep: not in gzip format (.DS_Store) # << this is unexpected
rm -rf ./**/.DS_Store

httpdisk-grep -h "HTTPDISK 403"
httpdisk-grep: 809: unexpected token at ...

Are you OK with it changing to warn on non-gzip formats as opposed to exiting?

When working on my project, there were a few API calls that would be in an error state, but the API creator responded with an HTTP 200 instead of a 4xx or 5xx.

It would be nice if we could provide custom callbacks before persisting or before using a cache to transform the response -- in my case either changing the status code/being able to tell httpdisk that it should be an error code based on the contents of the message.

Similarly, when popping a value out of the cache, it would be nice to be able to mutate the response or tell httpdisk that it's a :miss on custom logic.

What do you think?