kixell-nicolasjardillier / robots-txt Goto Github PK
View Code? Open in Web Editor NEWThis project forked from spatie/robots-txt
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
Home Page: https://spatie.be/en/opensource/php
License: MIT License