.Google.com's John Mueller answered an inquiry regarding why Google.com indexes webpages that are actually disallowed from creeping through robots.txt and also why the it's risk-free to overlook the related Browse Console documents concerning those creeps.Bot Traffic To Query Specification URLs.The person inquiring the concern documented that bots were making web links to non-existent question criterion URLs (? q= xyz) to webpages with noindex meta tags that are also blocked out in robots.txt. What motivated the question is actually that Google.com is actually crawling the hyperlinks to those pages, receiving obstructed through robots.txt (without watching a noindex robots meta tag) then getting shown up in Google Explore Console as "Indexed, though blocked by robots.txt.".The person inquired the adhering to inquiry:." However listed below is actually the large question: why will Google mark web pages when they can not also observe the content? What is actually the advantage because?".Google's John Mueller affirmed that if they can't creep the page they can't see the noindex meta tag. He also creates a fascinating mention of the website: hunt operator, encouraging to dismiss the results considering that the "average" customers won't see those end results.He created:." Yes, you are actually appropriate: if our experts can't crawl the page, our team can't view the noindex. That stated, if our company can not creep the webpages, then there's certainly not a great deal for our company to mark. So while you could observe several of those webpages along with a targeted website:- inquiry, the ordinary user will not observe all of them, so I wouldn't bother it. Noindex is actually also great (without robots.txt disallow), it merely suggests the Links will definitely find yourself being crawled (as well as find yourself in the Search Console report for crawled/not recorded-- neither of these standings trigger issues to the rest of the website). The integral part is that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's response validates the restrictions in operation the Site: hunt evolved search driver for diagnostic reasons. Some of those main reasons is actually due to the fact that it's certainly not connected to the regular search mark, it is actually a distinct trait completely.Google's John Mueller commented on the website hunt driver in 2021:." The quick answer is that a web site: query is actually not indicated to become complete, nor made use of for diagnostics purposes.An internet site query is actually a particular kind of search that confines the end results to a particular site. It is actually essentially simply words website, a digestive tract, and after that the site's domain.This question confines the outcomes to a certain site. It is actually not indicated to become a thorough collection of all the webpages coming from that site.".2. Noindex tag without utilizing a robots.txt is actually fine for these sort of conditions where a crawler is actually connecting to non-existent web pages that are actually obtaining found by Googlebot.3. URLs along with the noindex tag will certainly create a "crawled/not listed" entry in Explore Console and that those will not possess a negative effect on the rest of the web site.Check out the concern and also respond to on LinkedIn:.Why would Google.com mark webpages when they can't also view the material?Featured Graphic through Shutterstock/Krakenimages. com.