Google Introduces Two New Googlebot Crawlers: Google Other-Image and Google Other-Video

Google has expanded its family of web crawlers by introducing two new members: Google Other-Image and Google Other-Video.

two new GoogleOther crawlers, optimized for fetching images and videos — Two new Google Other crawlers, optimized for fetching images and videos

These new crawlers are designed to enhance the process of crawling binary data, specifically for research and development purposes.

The New Crawlers:
Google Other-Image: Optimized for fetching publicly accessible image URLs, this crawler is a specialized version of Google Other.

Its user agent token is `Google Other-Image`, and the full user agent string is `Google Other-Image/1.0`.

Google Other-Video: This crawler is fine-tuned for fetching publicly accessible video URLs.

It operates under the user agent token `Google Other-Video`, with the full user agent string `Google Other-Video/1.0`.

Both of these crawlers are extensions of Google Other, which was introduced in April 2023.

Google Other was initially developed to handle non-essential crawling tasks, thereby freeing up resources for the primary Googlebot crawlers, which are dedicated to core search functions.

Why New Crawlers?
The addition of these new crawlers is aimed at better supporting the crawling of binary data such as images and videos, which are vital for various Google services and internal research.

Types of Googlebot Crawlers:
Google uses different crawlers to manage various tasks:
Googlebot: The primary crawler responsible for indexing web content for Google Search.

This bot strictly adheres to `robots.txt` rules.
Special-case Crawlers: These perform specialized functions, like the AdsBot, and may or may not follow `robots.txt` directives depending on the task.
User-triggered Fetchers: These are tools that act based on a user’s request. For example, the Google Site Verifier or some functions in Google Search Console trigger crawls when users perform specific actions.

IP Address Ranges and DNS Details:
Google also provides information about IP address ranges and reverse DNS masks for each type of crawler:
Googlebot: Uses IP addresses listed in `Googlebot.json` and is identifiable by reverse DNS patterns such as `crawl-.googlebot.com` or `geo-crawl-.geo.googlebot.com`.
Special-case Crawlers: Use IP addresses from `special-Crawlers.json` and have reverse DNS patterns like `rate-limited-proxy-.google.com`.
User-triggered Fetchers: Referenced in `user-triggered-Fetchers.json` and use reverse DNS patterns such as `-.gae.googleusercontent.com`.

These updates reflect Google’s ongoing efforts to optimize web crawling, especially for multimedia content that plays a significant role in enhancing user experience and research outcomes.