Guzzle Config Options not honored fully about crawler

[freekmurze]

For now, I'd like a PR that just fixes the problems with the config options.

[freekmurze]

You're right that the passing of options seems to been have coded up a bit sloppy. Sorry for that.

Both issues should be addressed. I welcome a PR that fixes this.

[mallardduck]

Awesome, just wanted to make sure I was on the right track with my assumptions!
I can definitely submit a PR to address all that.

To be honest, I've kinda ended up chopping this up for my own needs. In my use case I'd like to track the actual DOM element the links come from, then when results are presented I can present both the link that redirected, the HTML/DOM element that caused it, and an updated element to prevent redirects.

It's still pretty rough, but the current results look something like:

SEO Warning: Redirect found 
This HTML link: <a href="/cloud.php">test</a> redirects to: https://www.liquidweb.com/cloudsites/
The final page/URL is: https://www.liquidweb.com/cloudsites/. The corrected link is:<a href="https://www.liquidweb.com/cloudsites/">test</a>

In order to accomplish that I had to diverge quite a bit from how things are currently done. Keeping track of links/URLs based on a new class called HtmlLink that tracks the URL and the HtmlNode.

Would that be something you'd be interested in merging as I polish it up and refactor a bit? I know this may only seem like a benefit in specific cases; with an seo-tool in mind though I think this could be really helpful! Let me know your thoughts on that?

I'm really excited and curious to see where we can go with these tools!

[freekmurze]

Your fix is included in the newly tagged version 2.0.7. Thanks again for your work on this.