Perplexity is an AI-powered search engine that provides users with a list of results and an insight to their enquiries. It relies on large language models to interpret queries with live web retrieval.
Perplexity use their crawler to utilise the content that gets surfaced, cited, or linked in their AI answers. Perplexity’s crawler follows robots.txt directives, so allowing the Perplexity User-Agent makes the content discoverable through Perplexity responses.
Perplexity has its own help center, where it documents information about its crawler and describes its intended use. These documentations are used to identify its purposes. Below is a breakdown of the primary user-agents and how they function.
Bot and Non-Bot User-Agents
Perplexity delivers its search results by collecting data, which operates in two different ways, either automatically or in response to the users’ requests.
Crawler: PerplexityBot
This PerplexityBot is an automated crawler used for general web crawling. It is not used for crawl content to train AI foundation models. It has been identified that this crawler is used for "Search" and "AI Input" purposes.
The Full user-agent string is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
This crawler is associated with AI, therefore to indicate this, the IsArtificalIntelligence property is set as “True”.
NotCrawler: Perplexity-User
This Perplexity-User is triggered directly by users’ requests to support user actions and is tied to real user activity. It does not behave like a traditional crawler and ignores the robots.txt rules. Additionally, this user-agent is not used for web crawling. Thus, it has been identified as a "NotCrawler" using the IsCrawler property, which in this case returns False.
The Full user-agent string is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Robots.txt Generator
Use 51Degrees to work with crawler usages rather than tracking individual crawlers and AIs. Checkout the free Robots.txt generator today.
Try our Robots.txt Generator