.Google's John Mueller responded to an inquiry concerning why Google.com marks pages that are forbidden coming from creeping by robots.txt and also why the it is actually safe to neglect the associated Look Console reports about those crawls.Crawler Website Traffic To Inquiry Parameter URLs.The individual asking the inquiry recorded that crawlers were developing links to non-existent concern parameter URLs (? q= xyz) to webpages along with noindex meta tags that are additionally shut out in robots.txt. What cued the inquiry is that Google.com is creeping the web links to those web pages, receiving blocked out through robots.txt (without seeing a noindex robotics meta tag) then receiving shown up in Google Browse Console as "Indexed, though shut out by robots.txt.".The person talked to the observing concern:." Yet listed below is actually the huge inquiry: why would certainly Google index webpages when they can't even observe the content? What's the perk because?".Google.com's John Mueller confirmed that if they can't creep the page they can't see the noindex meta tag. He likewise makes an interesting mention of the site: search operator, advising to overlook the end results due to the fact that the "normal" users will not find those results.He wrote:." Yes, you are actually right: if our company can't crawl the webpage, our team can not view the noindex. That mentioned, if our team can not crawl the webpages, at that point there's not a whole lot for us to mark. Thus while you could view some of those webpages along with a targeted internet site:- inquiry, the ordinary individual won't view all of them, so I definitely would not bother it. Noindex is additionally alright (without robots.txt disallow), it merely suggests the Links will wind up being crept (and end up in the Look Console record for crawled/not listed-- neither of these conditions result in problems to the rest of the web site). The fundamental part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's solution validates the limitations in operation the Web site: hunt accelerated hunt driver for analysis factors. Among those causes is actually considering that it is actually not hooked up to the regular search index, it is actually a distinct thing completely.Google.com's John Mueller talked about the web site hunt driver in 2021:." The quick answer is that a web site: question is actually certainly not implied to be comprehensive, nor used for diagnostics functions.A website query is actually a particular type of search that confines the results to a particular website. It's basically only words internet site, a bowel, and then the web site's domain.This question confines the results to a details web site. It is actually certainly not suggested to become a detailed compilation of all the webpages coming from that internet site.".2. Noindex tag without using a robots.txt is actually alright for these sort of scenarios where a bot is connecting to non-existent web pages that are actually receiving found out by Googlebot.3. URLs along with the noindex tag are going to generate a "crawled/not catalogued" entry in Browse Console and that those won't possess a bad impact on the rest of the website.Read the inquiry as well as answer on LinkedIn:.Why will Google index web pages when they can't also observe the content?Featured Graphic through Shutterstock/Krakenimages. com.