crawlers
Tuesday, July 28, 2026 1:30:00 PM
Naver's robots and crawlers
Research into Naver's approach to scraping web content
This blog explains how Naver’s web crawlers, including Yeti, Ads-Naver, and Blueno, interact with website content. It outlines their different roles, the crawler usage values assigned to each, and how they affect content visibility across Naver’s search and services.
Tuesday, June 16, 2026 1:30:00 PM
DuckDuckGo's Robots and Crawlers
Research into DuckDuckGo's approach to scraping web content
This blog explains how DuckDuckGo’s web crawlers, including DuckDuckBot and DuckAssistBot, interact with website content. It outlines their different roles, how they follow robots.txt directives, and the management of crawler access and content visibility.
Monday, March 30, 2026 5:30:00 PM
Apple Web Crawler
How Applebot and Apple crawlers shape visibility across Apple’s ecosystem
This blog explains how Apple’s web crawlers, including Applebot and Applebot-Extended, work across its ecosystem to surface content in Siri, Spotlight, and Safari. It outlines the different crawlers, their purposes—such as search indexing, feed updates, and AI training—and how enabling them can improve content visibility, traffic, and control over how content is used, particularly in generative AI.
Tuesday, March 3, 2026 5:30:00 PM
Perplexity's robots and crawlers
Research into Perplexity's approach to scraping web content
Find out how we classify Perplexity's AI web crawlers.
Friday, February 13, 2026 5:30:00 PM
Meta's robots and crawlers
Research into Meta's approach to scraping web content
Find out how we classify Meta's crawlers and AI content scrapers.