Cloudflare introduces an artificial intelligence audit tool to block access to websites by data scraping bots

Cloudflare has introduced AI Audit, a new suite of tools designed to give website owners and bloggers more control over their content and how it is used by automated systems. The security tool will allow users to block or allow artificial intelligence (AI) bots that frequently crawl websites to scrape data. The company offers this tool to all its existing customers at no additional cost. Cloudflare is also adding a feature that allows website owners to see detailed analytics about which bots are visiting the platform and their behavior patterns.

With the growing popularity of generative AI technology, there has been a rush to train large language models (LLMs) with human-generated data. This is important not only for the creation of the basic model, but also for its progression and improvement. However, the main problem here is access to publicly available data sources. Most AI companies have already gone through these data sets, and now they need more data to train their AI models.

AI bots are designed to help developers gather more information to train AI models. They can be understood as system programs that imitate a real user that can access web pages and copy text, image and video data. These AI robots can search large amounts of data in a short time and feed it to the AI ​​model. Recently, several media companies and major websites have filed lawsuits against AI companies alleging plagiarism and illegal use of data to feed the LLM.

Cloudflare’s AI Audit tool comes as a protective layer that can block such bots from accessing websites. The company revealed in a press release that it has also improved the tool to give users more control over which bots will be restricted and which will be allowed access. This is useful in cases where the platform has struck a deal with an AI company and doesn’t mind its bots taking data. Alternatively, a website owner may want to give access to specific AI models that attribute data source to achieve better reach.

Cloudflare noted that it is also building a workflow where website owners can set a fair price for their content. AI bot owners, on the other hand, will be able to transact with this firewall and after paying the amount, they will get the right to scan the content. The company points out that its marketplace-like tool can be useful for users who don’t have the bandwidth or resources to negotiate and close deals with every AI company that hits their website.