BBOT Server

[TheTechromancer]

I've started work on a backend for ingesting BBOT scans and tracking them over time.

This "BBOT server" will include both a python and REST API useful for ASM, pentest OSINT, bug bounties, and threat intel, that can be easily spun up with a single command, but also scaled to billions of events when needed.

The goal is to keep a history of changes to each asset, like new open ports, vulnerabilities, technologies, etc., and allow easy querying + alerting. Originally I had used SQLite as the default database, with a Postgres option for bigger setups. But this setup is brittle and icky to maintain.

I'm beginning to sense that something like Elastic might be better. Specifically, I like the idea of having a central store of events that can be easily browsed by power users, separate from BBOT server.

@colin-stubbs @nicpenning @CarsonHrusovsky Do you want to show off some of the work you've done with Elastic? I'm curious to see if it would be a good fit as a backend for BBOT server.

[nicpenning]

Hello, Techro!

The dashboards you see in the integration is a good example of what use today. We have customized it a little bit because we do more enrichments to the data with organizational related data (What is known vs unknown assets).
https://github.com/elastic/integrations/blob/main/packages/bbot/img/BBOT-Dashboard.png

Asset management can be tricky so the only way we can have the "current" state is by filtering down to a specific entity like evilcorp.com for example then apply a filter for the last scan date / scan name.

Since Elastic is a document store and not a relational database, we have to get creative with queries to present the information we seek. However, this is more of a limitation of using Kibana (The GUI) for Elasticsearch which is what that dashboard above is from. If you were to index all the data in Elasticsearch, you could use the API calls to develop your own stats/metrics and visualizations (I just don't know how that works).

External tooling would be needed to do diffs with current/previous scans to show a "Latest info" on an asset, which we don't do today, but would be worthwhile.

We can use a "New Terms" alert type rule with Kibana to show data in fields that haven't been seen in 30 days for example. This would allow us to alert/create events on any new findings over time via "Alerts".

Are you thinking about having Elasticsearch included in this BBOT build that would store these many events over time that users of BBOT could query specifically for, or maybe perhaps show in a new BBOT Web App the results of scans or something similar?

FYI - Elasticsearch is going open source again, so that may or may not help your design to embed that type of product into BBOT (if that is what you are looking to do).

[TheTechromancer]

Thanks @nicpenning.

Are you thinking about having Elasticsearch included in this BBOT build that would store these many events over time that users of BBOT could query specifically for, or maybe perhaps show in a new BBOT Web App the results of scans or something similar?

Yes, exactly. The idea would be to make full use of Elastic's features, while adding on a layer of custom functionality that makes it easy to track assets over time, diff between scans, etc.

Basically, I loathe SQL and want a schemaless backend. I'm not sure yet if Elastic is the best option, but I want to try it out and see.

It's been a real challenge to find a backend that makes full use of BBOT's data. In a way, a BBOT scan is NoSQL JSON documents. It's also time-series data, like logs. But then there are complex relationships between the documents. These relationships are why I think a graph database would be best.

However, we've already tried Neo4j, and while it was pretty neat, it was very slow, and its UI was incapable of rendering >1000 nodes. Not production ready.

Basically, I am exploring these types of databases:

• Search Engine (e.g. Elasticsearch)
• Time-series DB (e.g. Prometheus)
• NoSQL (e.g. MongoDB)
• Graph DB (e.g. Dgraph)

Probably none of them will be perfect, but my hope is to find the best possible one, and really commit to it, and use it as a foundation to build a really nice Python/REST API. This will most likely be open-source, and have a CLI to go with it. There won't be an official open-source GUI (because we are selling one, and we have to make money somehow 🫠), but I'm sure the community will make their own GUIs, similar to how @Sh4d0wHunt3rX has done.

[nicpenning]

Okay, thanks for the context. Elasticsearch is not a graph database for sure. The Graph feature was more for Kibana and I don't think that will scale well for visualization anyways. There is a new query language called ES|QL that is a bit more complex but unsure how it fits the use cases. There will still be much work to have the diff capability between BBOT executions since there isn't anything native in the platform.

[TheTechromancer]

Here I'll keep a running tally of the pros and cons of each database, as I explore them. Anyone is free to weigh in.

SQL

✅ PROS:

• Stable
• Allows relationships between tables
• Mature library support (SQLAlchemy)
• Can be extremely lightweight (SQLite)
• Integrates well with FastAPI (SQLModel)
• Plenty of query tools / visualizers for power users

❌ CONS

• Strict schema
  • makes it hard to evolve codebase over time
  • makes maintenance a pain (db migrations 🤮)
  • makes it difficult to add custom fields (i.e. custom server modules)
• Poor indexing of JSON data (i.e. nested fields like tags)

Elasticsearch

✅ PROS:

• Open source, large community
• Flexible Schema
• Full text indexing, powerful search
• Nice query / visualization tools for power users (Kibana)

❌ CONS

• Resource-intensive (Java)
  • Indexes everything by default, maybe overkill
• No builtin graph functionality (except this?)

MongoDB (NoSQL)

✅ PROS:

• Open source, stable
• Schema-less
• Extremely fast and efficient (C++)
• Simple and easy-to-use Python API
• Scalable

❌ CONS

• No query / visualization tools for power users
• No concept of relationships (except this?)

DGraph

✅ PROS:

• Schema-less
• Fast and efficient (Golang)
• Python API
• Native GraphQL
  • Schema can be generated from Pydantic models!
• Evolvable schema without migrations (theoretically)
• Can add/remove indexes at will
• Graph database
• Simple query tool / visualizer for power users (Ratel)

❌ CONS

• TODO

[nicpenning]

@TheTechromancer, if you would like some more information about using Elasticsearch within this project, you can reach out to my trusted contact from Elastic who can provide some guidance on this topic: [email protected] - you can give them my name.