Setting: Block AI bots

[jeffpaul]

Is your enhancement related to a problem? Please describe.

In conversations that I've had with ClassifAI users (actual and prospective) one item that's come up a couple times (more regularly with medium-to-large publishers) is the ability to block AI bots from scraping content from a site (so that a publishers content isn't used to train an AI model without their consent). Some related content for anyone that chooses to research this further:

• How to Block AI Chatbots From Scraping Your Website’s Content
• WP VIP: Block by partial User-Agent match for AI crawlers
• WP VIP: Disallow AI crawlers
• GPTBot - OpenAI API
• Websites That Have Blocked OpenAI’s GPTBot CCBot Anthropic Google Extended - 1000 Website Study
• How to block GenAI crawlers such as Google's Bard or OpenAI's ChatGPT from your website

Designs

The one place where this could be integrated is the basic ClassifAI Registration step in the Set Up flow that currently only includes the registration email and key fields. We could possibly add a disabled-by-default setting there akin to Block AI bots from crawling my site. Alternatively, it could be a setting like Block the following bots from crawling my site that then has disabled options for the current list of known AI bots (CCBot, ChatGPT-User, GPTBot, Google-Extended, anthropic-ai, claude-web, cohere-ai).

Describe alternatives you've considered

While this isn't necessarily a direct relationship to ClassifAI's focus, and more akin to a robots.txt management plugin, its certainly something that could be added (should more confirmed interest warrant it).

Code of Conduct

• I agree to follow this project's Code of Conduct