Google updated their Googlebot and crawler documentation to add a range of IPs for bots triggered by users of Google products. The names of the feeds switched which is important for publishers who are whitelisting Google controlled IP addresses. The change will be useful for publishers who want to block scrapers who are using Google’s … Read more
Google recently updated the documentation of its Google-Extended web crawler user agent, reflecting changes in product naming and clarifying the impact on search, which may be a concern for those who choose to block the crawler. The updated documentation offers clearer guidance on controlling content access for use in AI model training. Google-Extended User Agent … Read more
Google updated the list of their official crawlers by adding the name and information for a relatively unknown crawler that publishers have been seeing now and then but no documentation for it previously existed. Although Google added official documentation for this crawler the information provided seems to encourage more clarification. Special Case Crawlers Google has … Read more
Google has fixed a typo in their crawler documentation that inadvertently misidentified one of their crawlers. In general, this is a minor issue but it’s a major issue for SEOs and publishers who depend on the documentation to set firewall rules. Failure to notate the correct data could cause a website to inadvertently block a … Read more
Google has recently introduced a new web crawler called “GoogleOther,” designed to alleviate strain on Googlebot, its primary search index crawler. The addition of this new crawler will ultimately help Google optimize and streamline its crawling operations. Web crawlers, also known as robots or spiders, automatically discover and scan websites. Googlebot is responsible for building … Read more
