Indexation Researcher Discovers Google InspectionTool Crawlers Have Gone Rogue
American Way Media, a forensic SEO consulting agency, has discovered that the Google InspectionTool crawlers, used to render new content for indexing, are currently making excessive crawls on sites and this is likely what contributed to the recent spate of rejected search console messages earlier in the month and their removal from the crawling system in the past week.
Using the Google Mobile Friendly testing tool and then surveying server log records as part of her research on indexation, Carolyn Holzman, saw that a request to test one page for mobile friendliness resulted in hundreds of InspectionTool crawlers coming to the site and pulling multiple render crawls of the tested page.
According to Holzman, normal rendering crawls for similarly constructed content are more in the range of 12 - 15 requests. Seeing crawl requests as high as 192 caught her attention. A chart on her site on a page dedicated to Google’s Indexing Issues - she details the crawl issues in a chart highlighting 10 out of the 30 site, she examined a page and reviewed the number of crawls initiated by the InspectionTool crawlers.
“I believe this extreme crawl behavior is what was behind last week’s rash of search console error messages when site owners across the globe were reporting that they were receiving these errors. But other than the error message, they could find no reason based on their manual requests when even launching new sites on dedicated servers where they had earned that error,” she observed.
“I’m no stranger to the crawling behavior of Googlebots because shortly after I started testing the indexation system in August 2021, I started noticing how the chrome builds update and how older crawlers with older chrome builds get retired. And from there I was able to see what types of content they tend to crawl and render.”
“When the InspectionTool crawlers were announced and introduced in May 2023, they were sped through all the earlier chrome versions to catch up to, at the time, the current chrome build of the Googlebots by July 1st,” Holzman replied.
Holzman reports her findings on both her company site, AmericanWayMedia.com and on the Youtube Channel - Crawl or No Crawl.
Starting in September she had started to test the differences in indexation rates between requests made through search console and those made via the Google Indexing API.
On October 25, after reviewing multiple days of server logs from the tests, she remarked on the absence of the GoogleInspection crawlers in the testing data on Crawl or No Crawl
She continued, “While this unusual crawler behavior is not currently impacting sites through the search console, the underlying source is still running wild.”
Her discovery today aligns with her assertions that what caused the Hostload Exceeded rejection errors which on Monday appeared in some search console page inspections as “Temporary Processing Error” is ongoing. Her podcast episode launching on October 27th shares her observations and conclusions during this recent crawling issue.
According to Google’s website, the Mobile Friendly testing tool is slated to be retired by November 30, 2023.
About American Way Media:
https://www.youtube.com/watch?v=8Ah0flsPuss
American Way Media is a boutique onpage SEO consulting firm specializing in Forensic SEO for sites that either have on page deficits or been impacted by Google updates.. A practicing SEO since 2008, she has been a frequent speaker at SEO Conferences. Most recently, she shared her SEO tests results at the YACSS SEO conference in Aalborg, Denmark as well as SEO Rockstars and SEO Spring Training. Holzman is the host of Confessions of an SEO podcast now in Season Three. Her Forensic SEO Training course is a live training course over 4 weeks.
###
For more information about American Way Media, contact the company here:
American Way Media
Alison Kane
512-853-9484
akane@americanwaymedia.com
2309 Broughton Ct Suite B
Austin, TX 78727
Press Advantage
