Googlebot crawlers:
Google adds two new Googlebot crawlers:
GoogleOther-Image and GoogleOther-Video
Google has added two new crawlers to the Googlebot family of web crawlers:
Google’s new crawlers are to “better support crawling of binary data that may be used for research and development.
GoogleOther-Image
GoogleOther-Video
Google’s new crawlers are versions of GoogleOther optimized for fetching image and video bytes.
GoogleOther was added in April 2023 to be used internally by Google teams to crawl the public web and free up some resources for the main Googlebot crawlers.
GoogleOther-Image is the version of GoogleOther optimized for fetching publicly accessible image URLs.
It will go under the user agent tokens of GoogleOther-Image and GoogleOther and the full user agent string will be GoogleOther-Image/1.0.
GoogleOther-Video are the version of GoogleOther optimized for fetching publicly accessible video URLs.
It will go under the user agent tokens of GoogleOther-Video and GoogleOther and the full user agent string will be GoogleOther-Video/1.0.
Google new crawlers were launched to better support crawling of binary data that may be used for research and development.
The types of Googlebot crawlers include:
Googlebot – The main crawler for Google’s search products.
Google says this crawler always respects robots.txt rules.
Special-case crawlers – Crawlers that perform specific functions (such as Ads Bot), which may or may not respect robots.txt rules.
User-triggered fetchers – Tools and product functions where the end-user triggers a fetch.
For example, Google Site Verifier acts on the request of a user or some Google Search Console tools will send Google to fetch the page based on an action a user takes.
GoogleOther-Image and GoogleOther-Video
Google also has listed IP address ranges and reverse DNS mask for each type:
Googlebot Googlebot.json (crawl–.googlebot.com or geo-crawl-–.geo.googlebot.com)
Special-case crawlers special-Crawlers.Json (rate-limited-proxy-–.google.com)
User-triggered fetchers user-triggered-Fetchers.json (–.gae.googleusercontent.com)