should add option to scan robots.txt about gobuster

[OJ]

Hi @iddoeldor91, not sure I agree with this. This is a brute forcing tool. If you want to hit robots.txt then surely wget and curl have already got that covered?

What exactly are you looking to see happen? Bear in mind that gobuster isn't intended to be a generic web scanner, but a brute force tool. Thanks!

[mzpqnxow]

@iddoeldor I agree with @OJ, a feature like this would make gobuster behave more like a wordlist generation tool. Consider how gobuster works, it takes a single subdirectory (/ by default) and then throws the wordlist at it. How would you expect it to behave if it parsed robots.txt? Would it create new wordlists in memory, and then run the scan multiple times using each subdirectory found in robots.txt? That's a lot of effort and complexity relative to just using a short command line wrapper. Even something as rudimentary as this would get you most of the way there:

for elements in $(seq 1 3); do
  for subdir in $(curl <site>/robots.txt | cut -d '/' -f 1); do
    gobuster dir -u <site/$subdir -w wordlist.txt -e
  done
done

A better idea would be to do this in a dozen lines of python, grab robots.txt, use your wordlist as input, and spit out a new wordlist. Then invoke gobuster with the new wordlist.

It's a perfectly good idea and most people do make use of robots.txt, it's just not suitable for inclusion in gobuster in my opinion. The real value in gobuster is the performance of the HTTP requests and the flexibility it provides to build the various elements of those requests.

[iddoeldor]

Hi @OJ ,
I understand it's a brute force tool and not a generic web scanner,
I cannot convince you how important is to add it even as a bonus feature.

I am working on smart dictionary using github API & popular sitemaps,
it will reduce the amount of time & provide couple more accurate links,
also will load the dictionary file as Trie and iterate it.