Add support for dir "wildcard" detection

[gehaxelt]

Hi,

I noticed that a few times webservers are configured to return 200s or 301s instead of 404s. That means that gobuster will happily "find" the whole words.txt.

Idea: Count the "valid" hits in a row and if it exceeds 5% or 10% of the whole words.txt, then abort the scan (and/or add an "force" switch).

Kind regards,
gehaxelt

[OJ]

Thanks for the suggestion @gehaxelt. This is a quirky edge case. Detection based on the percentage of results is hard given that we don't know how big the world list is up front and I'm deliberately not pre-parsing the file to reduce memory impact.

I'll bounce some ideas around for this, coz it's definitely an issue. I think in the short term the best option is to get the grep functionality going as per #5, so that we can say "If you find this text it's actually classed as as404" or something like that.

Another option is to look at a baseline page size. If everything comes back as 200 with the content length the same, we could add a command line option to say "if you find a page of size X then mark it as 404".

Watch this space, we'll come up with something!

[gehaxelt]

Using grep is a good idea, but not really useful for my use case, because I usually run gobuster on a batch of different sites with a for-loop. If I need to adjust the 404-grep-filter for every page then there is no difference if I hit ctrl+c manually to abort the scan.

Basline page size might work, but could break on 200 OK-404-error pages which contain the full url.

I'd try to argue for my hit-count idea, but with a fixed "hits-in-a-row" parameter instead of a percentage.

[OJ]

Could we possibly take the approach that we took with wildcard DNS detection? Generate a GUID and do a single request for that, if it comes back with a status code match, then we can assume wildcard?

[gehaxelt]

That's probably the best idea and it should work.

[OJ]

Thanks again for all your efforts so far mate!

[gehaxelt]

Thank you for this awesome tool! :)

[gehaxelt]

bump
Just wanted to let you know that I've implemented the uuid-approach and that I'll open a PR tomorrow.

[OJ]

Awesome, thanks mate!

[Shaked]

I know this thread is old, but what was the reason that you guys didn't go with the grep feature? It seems very useful

[sergevanhaag2]

My two cents
The grep feature would be amazing. Just like hydra (brute-force credentials) implements a feature that checks the response body, brute-forcing URIs this way would be amazing too!

Maybe this helps someone in the future
Right now, if I notice that a website works with a wildcard that I cannot pass simply based on response codes, I employ the powers of ffuf. At least, that way I'm able to filter on word length or response lines. This tool is written in GO too.