Google Updates Googlebot verification documentation: Google has updated the official Googlebot documentation, outlining its approach towards visits originating from IPs linked with GoogleUserContent.com.
Google has updated new information in their Search Central Documentation concerning verifying Googlebot and user-initiated bot visits. This documentation was absent from earlier Googlebot documentation, resulting in confusion among publishers who had been blocking legitimate visits from the IP ranges.
Google has updated its Search Central Documentation to clarify its approach towards visits from IPs associated with GoogleUserContent.com. In the newly updated documentation, Google has introduced three categories of Google bots: Googlebot, special-case crawlers, and user-triggered fetchers (GoogleUserContent). Previously, confusion among publishers who blocked legitimate visits from IP ranges caused difficulties, as there was no documentation for GoogleUserContent. The updated documentation also states that user-triggered fetchers ignore robots.txt rules and that IP addresses associated with GoogleUserContent can be activated by the Google Site Verifier tool. Publishers blocking IP addresses related to GoogleUserContent should now unblock them.
Newly Updated Bot Documentation: Googlebot Verification Documentation
Google added a new Googlebot verification documentation that categorizes the three different kinds of bots that publishers should expect.
These are the three categories of Google Bots:
- Googlebot – Search crawler
- Special-case crawlers
- User-triggered fetchers (GoogleUserContent)
That last one, GoogleUserContent is one that’s confused publishers for a long time because Google didn’t have any documentation about it.
This is what Google says about GoogleUserContent:
“User-triggered fetchers
Tools and product functions where the end user triggers a fetch.
For example, Google Site Verifier acts on the request of a user.
Because the fetch was requested by a user, these fetchers ignore robots.txt rules.”
The documentation states that the reverse DNS mask will show the following domain:
“***-***-***-***.gae.googleusercontent.com”
Previously, some members of the SEO community suggested that bot activity from IP addresses linked with GoogleUserContent.com was initiated when users employed a translate function that used to exist in Google’s search results. This feature is no longer present in Google’s SERPs, and it’s unclear if the claim was accurate. Nevertheless, it was widely believed that such visits were attributable to Google.
According to the new documentation, Google states that bot activity from IP addresses associated with GoogleUserContent.com may be activated by the Google Site Verifier tool. However, Google hasn’t specified any other potential triggers for bots from the GoogleUserContent.com IP addresses.
Another modification to the documentation concerns the use of googleusercontent.com regarding IP addresses allocated to the domain name GoogleUserContent.com.
This is the new text:
“Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.”
Another new addition is the following text which was expanded from the old page:
“Alternatively, you can identify Googlebot by IP address by matching the crawler’s IP address to the lists of Google crawlers’ and fetchers’ IP ranges:
Googlebot
Special crawlers like AdsBot
User triggered fetches”
Google Bot Identification Documentation
Googlebot verification documentation: The new documentation finally has something about bots that use IP addresses that are associated with GoogleUserContent.
Search marketers were confused by those IP addresses and presumed that those bots were spam-related. A 2020 discussion on Google Search Console Help revealed how confused individuals were about the activity connected to GoogleUserContent.
Numerous participants in that discussion correctly deduced that it wasn’t Googlebot. However, some wrongly concluded that it was a phony bot impersonating Google.
A user posted:
“The behaviour I see coming from these addresses is very close (if not identical) to legitimate Googlebot behaviour, and it hits multiple sites of ours.
…If it isn’t – then this seems to indicate there is widespread malicious bot activity by someone trying quite hard to look like Google on our sites which is concerning.”
The initiator of the discussion ultimately deduced that the GoogleUserContent activity was spam after receiving multiple responses.
They wrote:
“…The Googlebots in question do mimic the official User-Agents, but as it stands the evidence seems to point to them being fake.
I’ll block them for now.”
Now we know that bot activity from IPs associated with GoogleUserContent are not spam or hacker bots.
They really are from Google. Publishers who are currently blocking IP addresses associated with GoogleUserContent should probably unblock them.
The current list of User Triggered Fetcher IP addresses is available here.
Read Google’s updated documentation:
Verifying Googlebot and other Google crawlers
