AI Identification
51Degrees helps publishers and other content rights holders to accurately identify AI bots for licensing or analytics.
Read on to learn how to gate access to content. Or follow the button to analytics.
AI for analyticsStop thief!
Rights holders need to decide if and how to license their copyrighted material.
Some AI companies are prepared to identify themselves and then offer content owners monetization options. But what about the ones that either don't or are offering rubbish terms?
51Degrees gives you the data you need to stop the bad AIs, and be rewarded by the good.
Complete Toolkit
All web requests contain passive information that can be used to identify device, platform, app or browser, and human or bot.
- IP address can be used in aggregate to determine the likelihood of a web session being from a human or not.
- User-Agent identifies AIs that want to be identified, and can be used with IP address to single out suspicious activity.
13%
of website sessions (not total traffic) came from AI bots in May 2026, up by more than 60% in a year, according to 51Degrees' analysis.
Three AIs
The good, the bad, and the ugly
Our products are designed to help deal with the good, the bad, and the ugly of AI harvesting.
Some AIs play nicely with publishers, others just want something for nothing, and then there's the ones that keep tipping the value exchange in their favor.
Use our IsArtificialIntelligence property to identify the AIs that want to be identified.
Then use CrawlerName to find the common name of the AI and decide how you want to treat it.
If you've a payment arrangement with the AI operator then let them in. If not then send them to a web page so they can find out how to set a payment relationship.
Here's the page we send AIs that crawl location and device information to.
No more using 51Degrees in Retrieval-Augmented Generation (RAG) without permission!
Too many AIs are not identifying themselves and pretending to be humans.
We've got you covered here with our HumanProbability property. A number between 0 and 10 indicates how likely the request is to be from a human.
Armed with this information you can decide when to challenge for proof of humanity.
Here's the page we send suspicious activity to.
If you're not a B2B then a captcha can be used to separate silicon from carbon based life.
Some crawlers use the same request to obtain data for both search results and AI overviews.
These can't be blocked via technical means without suffering in search. No one wants that.
Our friends at Movement for an Open Web (MOW) can help. Cases are being run against big tech companies to limit their use of AI overviews in search results.
Contact MOWOur customers
Publishers, website owners, streaming companies, and all content rights holders are under siege from AI bots looking to steal their content.
51Degrees helps companies to recognize AI bots, whether they identify themselves or not, and to then support licensing agreements to drive revenues from harvesting.
3 ways to handle AI crawlers
Set a fee for AI use of your articles
Set the terms. Four lines in robots.txt point crawlers at the contract before they read a page. Block the AI crawlers you never want, or let them in at the contract's price. Our free robots.txt service writes the rest of the file.
User-Agent: * # Terms https://m4ow.uk/socw/2.txt TDL: https://m4ow.uk/socw/2.txt Allow: /
Who it serves. Publishers whose articles are used to train AI models or answer questions.
Pairs well with Evidence for the invoice, to show who took what once the price is set.
Integration time. 10 minutes with the free robots.txt generator.
Ingredients, 3
- PlainText
- AnnotatedText
- CrawlerUsage
Hero ingredient. CrawlerUsage Use the crawler's purpose to allow free search and charge for other AI uses.
- Allow the uses the contract permits for free, such as search
- Decide whether each AI use is blocked or let in at the contract's price
- Keep the Search Only Contract in, or add a link to your own terms
- Publish the file at the root of your site as robots.txt
Chef's tip. Put the contract link in robots.txt, where crawlers read your terms before visiting a page.
Tell each crawler which uses of your pages you allow
Crawlers collect pages for different uses. Some build a search index that sends you visitors, while others train AI models that answer questions using your content. robots.txt tells named crawlers what you allow, but new crawlers appear every month.
Choose the uses you allow and our robots.txt generator writes the file for you. It emails a new version when crawlers or their uses change.
Who it serves. Publishers, retailers, and any website whose pages crawlers collect.
Pairs well with Suspicious traffic management, for investigating visits from crawlers that ignore your file.
Integration time. 10 minutes, with no code.
Ingredients, 4
- IsCrawler
- CrawlerName
- CrawlerUsage
- CrawlerProductTokens
Hero ingredient. CrawlerUsage Choose whether crawlers can use your pages for search indexing or AI training.
- Choose the uses you allow on the robots.txt generator
- Publish the file at the root of your website
- Replace it when we email you an update
- Compare CrawlerUsage on each request with the uses allowed in your file
Chef's tip. Keep the email address current, because the generator sends a new file when crawlers or their uses change.
Check for bot traffic in your analytics and spending
Use the crawler properties to identify visitors that give their name and purpose, then handle them according to your robots.txt. For visits that aren't named crawlers, use Human Confidence to decide what they can see and how to count them in analytics. The network properties let you review traffic from a data center, VPN, or proxy.
Who it serves. Teams checking bot traffic in analytics and ad spending.
Pairs well with Spot the fake, for a closer look at the device in a suspicious request.
Integration time. 30 minutes in any Pipeline API integration.
Ingredients, 10
- IsCrawler
- CrawlerName
- CrawlerUsage
- CrawlerProductTokens
- IsArtificialIntelligence
- HumanConfidence
- IsVPN
- IsProxy
- IsHosted
- HumanProbability
Hero ingredient. HumanConfidence Use the rating to decide whether a visit should see your prices.
- Check IsCrawler on every request
- Filter named crawlers by name and usage
- Use IsArtificialIntelligence to identify AI traffic
- Use Human Confidence to review visits that aren't named crawlers
Chef's tip. Crawler detection is free on every plan. Use it to handle the visitors that identify themselves before reviewing the rest.
Start with the AI problem you want to solve
Each recipe explains the business use, followed by the properties and steps your developers need. It also includes a link to create a Resource Key.