crawlers
Tuesday, June 16, 2026 1:30:00 PM
DuckDuckGo's Robots and Crawlers
Research into DuckDuckGo's approach to scraping web content
This blog explains how DuckDuckGo’s web crawlers, including DuckDuckBot and DuckAssistBot, interact with website content. It outlines their different roles, how they follow robots.txt directives, and the management of crawler access and content visibility.
Monday, March 30, 2026 5:30:00 PM
Apple Web Crawler
How Applebot and Apple crawlers shape visibility across Apple’s ecosystem
This blog explains how Apple’s web crawlers, including Applebot and Applebot-Extended, work across its ecosystem to surface content in Siri, Spotlight, and Safari. It outlines the different crawlers, their purposes—such as search indexing, feed updates, and AI training—and how enabling them can improve content visibility, traffic, and control over how content is used, particularly in generative AI.
Tuesday, March 3, 2026 5:30:00 PM
Perplexity's robots and crawlers
Research into Perplexity's approach to scraping web content
Find out how we classify Perplexity's AI web crawlers.
Friday, February 13, 2026 5:30:00 PM
Meta's robots and crawlers
Research into Meta's approach to scraping web content
Find out how we classify Meta's crawlers and AI content scrapers.