Apple operates its own web crawler called Applebot, which surfaces web content throughout its ecosystem, including Spotlight, Siri, and Safari, allowing users to discover information directly across their Apple devices globally.
Applebot enables a site’s content to be accessible. Apple also offers controls over how content is used, particularly in relation to generative AI. Enabling Applebot in robots.txt ensures that web content can appear in Spotlight, Siri results, and Safari suggestions. This enhances visibility and reach, making it easier for users to discover sites and driving potential traffic and engagement.
Apple provides detailed information about Applebot on its support page, outlining its User-Agent identifiers and usage purposes. This information helps users understand how Applebot crawls content across the web.
Exploring Apple’s crawlers and their usage
Applebot
For web crawling and rendering, Apple’s primary crawler contains the word “Applebot” along with other general information, such as browser version. It has been identified that this crawler is mainly used for “Search” purposes.
The full User-Agent string is:
Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)
This crawler is not associated with AI; therefore, the IsArtificialIntelligence property is set to “False”.
Applebot-Extended
Apple’s secondary crawler, Applebot-Extended, specifically assists in gathering data to train and improve Apple’s generative AI models. However, it is not used for indexing or crawling websites for search results.
This provides web publishers with additional control, particularly regarding how their content is utilised in training Apple’s generative AI models. Allowing this crawler aids in improving Apple’s AI models. It has been identified that this crawler is used only for “AI Train” purposes and returns the IsArtificialIntelligence property as “True”.
The User-Agent includes “Applebot-Extended”.
Apple Podcasts
The iTMS User-Agent for Apple Podcasts is a crawler that retrieves and updates content from registered podcast URLs. It focuses on maintaining and syncing podcast feeds and associated media. Therefore, this crawler is identified as being used for “Feed” purposes only.
It is identified by the User-Agent “iTMS”.
AppleNewsBot
AppleNewsBot is utilised to fetch content for Apple News articles. It has been identified that this crawler is used for both “Search” and “Feed” purposes. This crawler accesses remote resources included in Apple News articles and helps update content for the Apple News platform, ensuring articles remain discoverable.
It is identified by the User-Agent “AppleNewsBot”.
References
About Applebot