Slash in robot txt is misinterpreted.

#10 · open · 1 comments

View on GitHub ↗

sjdirect

The two lines below are included in robots.txt: Disallow: /om/work-at-abc/lediga-jobb/ Disallow: /om/work-at-abc/lediga-jobb? Consequently, this page is incorrectly disallowed: https://www.abc.se/om/work-at-abc/lediga-jobb Log: [https://www.abc.se/om/work-at-abc/lediga-jobb] not crawled, [Disallowed by robots.txt file] Kind Regards, Ola

Comments

sjdirect

Moved from https://github.com/sjdirect/abot/issues/186