mmerian / phpcrawl Goto Github PK

...and when autoloading the classes the wrong one is loaded (the one missing some of the methods used by the script). PHPCrawlerUtils appears in libs/Utils (the right one) but also in libs/.

The following regexes in the prepareHTMLChunk function leads to a complete empty html source for many pages:

$html_source = preg_replace("#^(?:(?!<script).)*<\/script># Uis", "", $html_source);

$html_source = preg_replace("#<\!--.*(?:-->|$)# Uis", "", $html_source);

$html_source = preg_replace("#^(?:(?!<\!--).)*-->#Uis", "", $html_source);

My regex skills are not good enough to debug it.

Framework: Laravel
PHPCrawl version: 0.83

Issue:
I'm trying to set the obeyRobotsTxt but it uses the wrong PHPCrawlerUtils. obeyRobotsTxt calls PHPCrawlerRobotsTxtParser::parseRobotsTxt which in turn calls PHPCrawlerUtils::getURIContent but it doesn't find it, reason why is because it uses this Class:

vendor/mmerian/phpcrawl/libs/PHPCrawlerUtils.class.php //Doesn't contain getURIContent

Instead of this one, which it should use.

vendor/mmerian/phpcrawl/libs/Utils/PHPCrawlerUtils.class.php ////Does contain getURIContent

error:

Call to undefined method PHPCrawlerUtils::getURIContent()

autoload warning:

Warning: Ambiguous class resolution, "PHPCrawlerUtils" was found in both "/Users/macmini2/securityscan/vendor/mmerian/phpcrawl/libs/PHPCrawlerUtils.class.php" and "/Users/macmini2/securityscan/vendor/mmerian/phpcrawl/libs/Utils/PHPCrawlerUtils.class.php", the first will be used.

Hello

I'm using the following code

class MyCrawler extends PHPCrawler {
        function handleDocumentInfo(PHPCrawlerDocumentInfo $PageInfo)   {
            // Your code comes here!
            // Do something with the $PageInfo-object that
            // contains all information about the currently 
            // received document.

            // As example we just print out the URL of the document
                //weekly pode ser fornecido como paramentro
            //return -1;
        }
    } 

    $crawler = new MyCrawler();
    $crawler->setURL('http://example.com'); 
    $crawler->setWorkingDirectory("/dev/shm/");
    $crawler->setUrlCacheType(PHPCrawlerUrlCacheTypes::URLCACHE_SQLITE);
    $crawler->excludeLinkSearchDocumentSections(PHPCrawlerLinkSearchDocumentSections::ALL_SPECIAL_SECTIONS); 
    $crawler->addContentTypeReceiveRule("#text/html#");
        $crawler->go();  

How can I access the PHPCrawlerDocumentInfo->links_found after the crawl is complete?
Thanks in advance.

Hello

Is phpcrawl being maintained by the author?
I have tried to reach him by e-mail regarding the class but no success...

Does it work with html5?