Perplexity is an AI-powered search engine that provides users with a list of results and an insight to their enquiries. It relies on large language models to interpret queries with live web retrieval.
Perplexity use their crawlers to utilise the content that gets surfaced, cited, or linked in their AI answers.
Perplexity has its own help center, where it documents information about its crawlers and describes their intended uses. These documentations are used to identify their purposes. Below is a breakdown of the primary user-agents and how they function.
User-Agents
Perplexity delivers its search results by collecting data, which operates in two different ways, either automatically or in response to the users' requests.
Crawler: PerplexityBot
This PerplexityBot is an automated crawler used for general web crawling. It is not used to crawl content to train AI foundation models. It has been identified that this crawler is used for "Search" and "Input" purposes.
PerplexityBot follows robots.txt directives, so allowing the PerplexityBot User-Agent makes the content discoverable through Perplexity responses.
The Full user-agent string is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
This crawler is associated with AI, therefore to indicate this, the IsArtificalIntelligence property is set as “True”.
Crawler: Perplexity-User
This Perplexity-User crawler is triggered when users ask Perplexity a question. This crawler visits web pages to help provide an accurate answer. The data is not used to provide a direct excerpt from the original web page, it is used to shape and produce generative AI responses, therefore it falls under the “Input” crawler usage category. As a result, the IsArtificialIntelligence property returns “True”.
The Full user-agent string is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Robots.txt Generator
Use 51Degrees to work with crawler usages rather than tracking individual crawlers and AIs. Checkout the free Robots.txt generator today.
Try our Robots.txt Generator