Please use a custom User Agent header about pitchfork

[michalczaplinski]

Im guessing you mean modifying the default user agent string that is sent by urllib2 "Python-urllib/2.6" to something else? Excuse my ignorance, but what is a purpose of it other than allowing the server admins to see what app reads their page? Is there a commonly accepted practice on what should go in there, like author+app name?

[mattdennewitz]

So that we can see your scraper is exactly why I'm asking. I'll defer to you for what the string could be, but something as simple as michalczaplinski/pitchfork-1.0 would be helpful.

[michalczaplinski]

fixed! thanks for the feedback.