How to update robots.txt in Shopify

A robots.txt file is a crucial but sometimes overlooked component in optimizing a website’s technical SEO. In this blog, we’ll be diving into what a robots.txt file is and how to update it in Shopify.

What is a robots.txt file?

Search engines, such as Google, frequently crawl the internet to find new data as a source for their search results. A robots.txt file allows a website owner to inform search engine crawlers which URLs can be accessed on a website. A website’s robots.txt can be viewed by adding “/robots.txt” at the end of the URL for a primary domain name. If you fancy seeing something creative, check out Nike’s robots.txt: https://www.nike.com/robots.txt (trust me, just do it…)

The primary use is to prevent a website from being overloaded with requests, allowing the crawl budget to be preserved. It’s worth noting that it is not a way of keeping a web page out of Google or other search engines. If you need to keep a page out of search engines, you need to prevent indexing by giving it a ‘noindex’ attribute.

For a breakdown of how robots.txt affects different file types, check out this guide Google put together.

Shopify’s default robots.txt functionality

All Shopify stores have a robots.txt file by default. The default configuration works for most Shopify websites, but edits can be made through the robots.txt.liquid theme template. With this, the following edits can be made:

• You can allow or disallow certain URLs from being crawled
• You can add crawl-delay rules for specific crawlers
• You can add extra sitemap URLs
• And you can block specified crawlers

Steps to update robots.txt in Shopify

Updating the robots.txt.liquid can become a little technical. If you have web development experience or technical SEO knowledge, you shouldn’t run into any issues. However, if you’re unsure what you might need to include, allow, or disallow, we’d strongly recommend consulting an expert Shopify SEO agency. Robot.txt mistakes can be costly. If you accidentally disallow pages that actually need to be indexed by search engines, you could end up missing out on some very crucial organic traffic.

Before you edit your robots.txt.liquid file, it’s a good idea to remove previous customizations or workaround. These are typically third-party services, like Cloudfare, that can cause some conflicts. Then, complete the following steps:

1. From the Shopify admin screen, select Settings > Apps and sales channels
2. Select Online store
3. Select Themes
4. Select Actions, then click Edit code
5. Select Add a new template and then select robots
6. Select Create template
7. Make your desired changes to the default template
8. Save any changes to the robots.txt.liquid in your published theme.

And that’s it — your robots.txt file has been updated!

How to test your robots.txt changes

The changes to your robots.txt will happen immediately, but they might not get noticed by search engine crawlers right away. Google’s robots.txt Tester is a great, free tool to see whether your robots.txt file blocks Google’s web crawlers from specific URLs on your website.

However, this will require you to have Google Search Console set up. If you haven’t already set this up, it is a very powerful and free tool allowing for detailed insight into your site’s organic traffic and health.

Thank you for reading our blog post on how to update robots.txt in Shopify! We hope you found the tips and strategies we shared to be helpful and informative. If you’re looking for more guidance or support with optimizing your eCommerce website for search engines, as an eCommerce SEO Agency, we offer a range of services from eCommerce SEO to WooCommerce SEO and Shopify SEO at Tillison consulting. Our team of experts can help you create and implement a customized SEO strategy that fits your business needs and goals. To learn more about our services or to schedule a consultation, please contact us today. We look forward to working with you