Whether it’s a few pages you’re trying to hide or an entire staging site – almost everyone eventually has stuff showing up in Google that shouldn’t be there.
So let’s say you added a noindex meta tag (<meta name=”robots” content=”noindex”> in the <head>) to the relevant pages and you’re still seeing them in Google (usually via a site: search). Now what?
Why hasnt google removed my pages?
Almost always, the issue is one of two things.
- Google can’t crawl the content
- Google hasn’t crawled the content
goOgle can’t crawl the content
This is because you either blocked crawling via robots.txt or put the pages behind a login. Both decent ways to prevent google from seeing pages in the first place, but not helpful after it’s already indexed pages.
Google hasnt crawled the content
Common for dev sites or pages that once we’re linked on your site but have been removed.
How you can fix this
In both cases mentioned above, the issue is Google doesn’t know you’ve added a noindex meta tag. Google doesn’t visit everything in it’s index every day or even every week, so we need to draw it’s attention.
- Open up crawling – take away logins or robots.txt crawl restrictions
- For more urgent issues, use the URL Removal Tool in search console (note – this is temporary!)
- Inspect the URL using the search bar at the top of every page of search console.
- Click Request Indexing
You read that right. You’re not really telling Google to index the url, you’re telling it to crawl. This will get Google to notice your noindex tag.
Once your page is removed from Google you can go back to putting the content behind a login or blocking via robots.txt.
but i have so mANY urls
Got a ton of pages? Maybe an entire site? No need to worry.
The easiest way to do this is creating an XML sitemap of your noindex pages and submitting that via search console.
If you have a list of URLs, there are sitemap generators out there. You can also use Screaming Frog to generate the file for you (and verify your noindex tags are all present at the same time).
This accomplishes the same thing (get Google to crawl your pages and see the noindex tags) at scale. You can also monitor the sitemap in search console to see the number of indexed pages drop to zero.
Simple as that! Go forth and make your content disappear.
0 Comments