sjdirect
I am trying to crawl this website with isRespectRobotsDotTextEnabled set to true: http://artofprogress.com/ This is triggering the PageCrawlDisallowed event with "[Disallowed by robots.txt file]" as the DisallowedReason. As far as I can tell, the robots.txt file doesn't prevent any page from being crawled. Here is complete text of the robots.txt file: User-agent: * Disallow: This is happening on other websites (incidentally, all WordPress sites) as well