nadar / crawler Goto Github PK

The $links array should only return absolute paths if possible, otherwise further processing of the link won't work (for example linkcheck in luya crawler)

• A new "stack" interface in order to check wheter this url has been crawled or not and to add new urls to the stack (default ArrayStack)
• Runner Interface (https://github.com/nadar/php-page-crawler/blob/master/src/Crawler.php#L85) - It would be nice to run those jobs async, for example in queue. How to pass back the information about "its finished"?

It might be helpfull to skip a certain file size internally, therefore us content length header:

https://stackoverflow.com/a/52479875/4611030

In order to display the link text in the luya module crawler the title is required.

Ignore links which contains "no-follow" param. in my case this is used when using cookie widgets which just reloads the page with ?acceptCookes=1 therefore this should not be followed

When there is its nice!<br/>See you it will be its nice!See you in search results .. maybe replace newlines with spaces and trim them. So Its nice! See you is expected even thought it changes its original content.

Make some tests first

When a link contains ?id=y but the current url context is foo.com/bar it should be foo.com/bar?id=y as absolute link, instead it seems o take foo.com?id=y.