Website Audit Series: Sitemaps, Noindex, and Robots.txt
If you manage a website, it is crucial to understand how search engines “see” your site. One simple (and free) tool that does this is Google Search Console. Once you set up your account and verify your website ownership, there are a variety of metrics and reports about your site available. These include redirect issues and crawl errors (which we covered in this post), as well as your sitemap, noindex issues, and your robots.txt file, which we will cover here!
Sitemaps
A sitemap is important because it allows search engines to crawl your site quickly to see updates since the sitemap should be updated regularly. This captures changes to pages, new pages added, new blog posts, updated content, etc. If you are using WordPress, a plugin can easily create a sitemap in the correct format (XML). This may need to be done manually if you have a custom website or a different CMS. Once you have an XML sitemap created, you can submit it in the “Sitemaps” section of Google Search Console. The results will tell you if Google could read the sitemap, how many URLs it found, and if there are any issues with these pages.
Noindex Errors
In the “Coverage” section of Google Search Console, you may see an error that reads “Submitted URL marked ‘noindex’ – for example:
You might decide a page should have the noindex tag for various reasons. If this page was not indexed intentionally, it isn’t really an error. But you might find a page that should be indexed has this tag. If you click on the error, it will tell you exactly what pages are not being indexed, and you can decide if they are supposed to be that way or not.
Robots.txt
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. Although it is not necessary to have one, it can help Google avoid crawling unimportant or similar pages on your website. To test your current robots.txt file once you are logged into Search Console, you can use this Google tool to test your robots.txt file. If you think your site or certain pages are not being indexed correctly, this is a great place to start!
Looking for More?
Here are a few more posts to help you audit your site, plus a list of our favorite tools: