Google announced an update to their crawler documentation, adding more information about caching which should help better understand how to optimize for Google’s crawler. By following the new guidelines on implementing proper HTTP caching headers, SEOs and publishers can improve crawling efficiency and optimize server resources. Updated Crawler Documentation The crawler documentation now has a … Read more
Google has launched a major revamp of its Crawler documentation, shrinking the main overview page and splitting content into three new, more focused pages. Although the changelog downplays the changes there is an entirely new section and basically a rewrite of the entire crawler overview page. The additional pages allows Google to increase the information … Read more
Google quietly added a new bot to their crawler documentation that crawls on behalf of commercial clients of their Vertex AI product. It appears that the new crawler may only crawl sites controlled by the site owners, but the documentation isn’t entirely clear on that point. Vertex AI Agents Google-CloudVertexBot, the new crawler, ingests website … Read more
In a recent episode of the Search Off The Record podcast, it was revealed that Google’s rendering system now pretends to be “idle” to trigger certain JavaScript events and improve webpage rendering. The podcast features Zoe Clifford from Google’s rendering team, who discussed how the company’s web crawlers deal with JavaScript-based sites. This revelation is … Read more
Google updated their Googlebot and crawler documentation to add a range of IPs for bots triggered by users of Google products. The names of the feeds switched which is important for publishers who are whitelisting Google controlled IP addresses. The change will be useful for publishers who want to block scrapers who are using Google’s … Read more
Google recently updated the documentation of its Google-Extended web crawler user agent, reflecting changes in product naming and clarifying the impact on search, which may be a concern for those who choose to block the crawler. The updated documentation offers clearer guidance on controlling content access for use in AI model training. Google-Extended User Agent … Read more
Google updated the list of their official crawlers by adding the name and information for a relatively unknown crawler that publishers have been seeing now and then but no documentation for it previously existed. Although Google added official documentation for this crawler the information provided seems to encourage more clarification. Special Case Crawlers Google has … Read more
Google has fixed a typo in their crawler documentation that inadvertently misidentified one of their crawlers. In general, this is a minor issue but it’s a major issue for SEOs and publishers who depend on the documentation to set firewall rules. Failure to notate the correct data could cause a website to inadvertently block a … Read more
Google has recently introduced a new web crawler called “GoogleOther,” designed to alleviate strain on Googlebot, its primary search index crawler. The addition of this new crawler will ultimately help Google optimize and streamline its crawling operations. Web crawlers, also known as robots or spiders, automatically discover and scan websites. Googlebot is responsible for building … Read more
