Katana VS BBOT Web Spidering

[Sh4d0wHunt3rX]

I'm comparing these two tools for web spidering.

I used these two commands:

katana -u https://www.myaccounting.it/ -o katana.txt and pass results from httpx -l katana.txt -o httpx.txt -mc 200

bbot -t https://www.myaccounting.it/ -m httpx -om asset_inventory -c web_spider_distance=4 web_spider_depth=4 web_spider_links_per_page=1000 omit_event_types=["DNS_NAME_UNRESOLVED","URL_UNVERIFIED"] url_extension_httpx_only=[]

I only considered links from https://www.myaccounting.it/ . There are many links exist in httpx result, but not in bbot, A lot from links cointain /wp-json/ and embed. I got diff and attached the links here. I also removed blacklist extensions such as .css .

Is this because of my settings for depth?

httpx.txt
bbot.txt
diff.txt

[TheTechromancer]

Yes, you will not see links like https://www.myaccounting.it/wp-json/wp/v2/categories/291 because the depth of that URL is 5, but you have your max depth set to 4.

[TheTechromancer]

Thanks for testing this btw. Our goal is to be the best web spidering tool, but most likely there are still a few improvements we need to make before it's fully up to speed with katana. Most of these will involve better URL extraction logic.

Something to note is that right now BBOT strips off the query string (everything after the ?). But @liquidsec is experimenting with making that option configurable.

[Sh4d0wHunt3rX]

Wow, that's awesome, thanks for this ❤️

[Sh4d0wHunt3rX]

Hey @TheTechromancer , is there any possibility to set timeout for individual modules? For example same as what you gave me, "send kill nuclei after 10 seconds" ? Because I don't want to finish the whole scan in some cases, crawling and new hostnames is important for me, but for examplee vhost is slow and I don't want it to run more than 1 hour. 🙏

[TheTechromancer]

That's a good idea. We've actually already implemented that for gowitness. It wouldn't be hard to do for nuclei. You wanna make an issue?

[TheTechromancer]

Also if vhost is misbehaving, it might be a bug. Can you include any information about the vhost issue?

[Sh4d0wHunt3rX]

No, I just said as an example, sorry 😁