.Google's John Mueller answered an inquiry concerning why Google marks webpages that are actually prohibited coming from crawling through robots.txt and why the it is actually safe to ignore the related Explore Console documents about those creeps.Crawler Website Traffic To Question Parameter URLs.The person talking to the concern documented that bots were generating hyperlinks to non-existent concern guideline Links (? q= xyz) to web pages with noindex meta tags that are actually likewise obstructed in robots.txt. What prompted the question is actually that Google is actually creeping the links to those pages, acquiring blocked by robots.txt (without envisioning a noindex robots meta tag) at that point obtaining turned up in Google.com Browse Console as "Indexed, though blocked out by robots.txt.".The person inquired the following inquiry:." But below's the significant question: why would certainly Google index pages when they can not even see the information? What's the advantage in that?".Google.com's John Mueller confirmed that if they can't creep the webpage they can not view the noindex meta tag. He additionally makes an interesting mention of the internet site: search driver, urging to disregard the end results due to the fact that the "common" consumers won't see those end results.He created:." Yes, you're correct: if we can't crawl the page, our team can not see the noindex. That claimed, if our experts can not creep the pages, at that point there is actually certainly not a lot for us to mark. Thus while you may view a few of those web pages along with a targeted web site:- query, the common consumer will not see them, so I definitely would not fuss over it. Noindex is likewise alright (without robots.txt disallow), it just indicates the URLs will certainly wind up being actually crept (and also end up in the Look Console record for crawled/not listed-- neither of these standings result in problems to the remainder of the web site). The fundamental part is that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's solution confirms the restrictions in operation the Website: search progressed hunt driver for analysis reasons. Among those main reasons is given that it's certainly not attached to the regular search index, it's a separate trait entirely.Google.com's John Mueller commented on the site hunt operator in 2021:." The quick solution is that an internet site: inquiry is certainly not meant to become total, nor used for diagnostics functions.A site question is a certain sort of search that limits the outcomes to a specific website. It is actually generally merely words website, a bowel, and then the internet site's domain name.This question limits the end results to a specific website. It is actually certainly not indicated to be a detailed compilation of all the web pages coming from that internet site.".2. Noindex tag without utilizing a robots.txt is actually alright for these kinds of conditions where a crawler is linking to non-existent web pages that are obtaining discovered through Googlebot.3. Links with the noindex tag are going to create a "crawled/not catalogued" entry in Search Console which those will not have a negative result on the rest of the website.Review the concern and also address on LinkedIn:.Why would Google.com mark webpages when they can not even view the information?Included Image through Shutterstock/Krakenimages. com.