Shopify sites have been a popular choice for online store owners for years, but the recent addition of customisable Shopify robots.txt file options has made a huge difference.
But what does this actually mean, and how does it benefit Shopify store owners?
What is Robots.txt?
Shopify robots.txt is a file altering how search engine crawlers can access Shopify sites and it can create additional rules for specific pages.
This gives users crawl control over any important pages, offering greater technical SEO flexibility.
Located at [yourwebsite.com]/robots.txt, the Shopify robots.txt file default configuration is applied to sites automatically.
Now, being able to add custom commands means that users can start blocking pages from search engine results with only a minor tweak to the default file.
What does txt file do?
When sites are crawled under default rules, each crawled web page becomes available in search results.
For example, when Google crawls a web page, that web page can appear in relevant Google search results.
However, not all Shopify store owners want all pages to appear there. Some Shopify stores have a Shopify admin page, unfinished online store sections, or other areas that could still appear in search engines.
The Shopify robots.txt file allows users to exclude certain pages from crawling. By editing robots, the Shopify site can get custom rules that adjust how bots work.
This means that those Shopify store pages no longer appear in search engines and can even be blocked from internal site search results.
How To Edit Shopify Robots.txt
Editing the Shopify robots.txt file is relatively easy:
- First, go to your Shopify dashboard, then to Online Store > Themes
- Go to the Live Theme section, then Actions > Edit Code
- Under the Templates section, choose Add New Template
- Change the “Create a new template for” option to the Shopify robots.txt file
- Finally, click Create Template
Is There an Easier Way to Edit Code in the Shopify robots.txt file?
If you prefer, you can always download or copy the Shopify robots.txt file and edit it in a text editor.
This may be useful if you, as a Shopify admin, want another person to edit code in the Shopify robots.txt file without using your account.
Customising Robots.txt.liquid
There are three main ways to modify the Shopify robots.txt file, letting you edit Shopify robots in different ways.
- Add a rule to an existing group
- Remove a rule from an existing group
- Add a custom rule
In this context, a “group” means specific crawlers. The Shopify robots.txt file will usually cover most crawlers by default.
There are also “user agents”. User agent refers to any software that presents or collects web content.
Any crawlers that visit an ecommerce platform are user agent, and blocking each user agent via the Robots.txt file can prevent certain pages from appearing in relevant search engines.
Add a new rule to an existing group
Adding a new rule can be complex at first but is relatively easy once you understand how it works.
The usual method to block search engines involves a system like:
` {%- if group.user_agent.value == ‘*’ -%}`
`{{ ‘Disallow: /collections/all*’ }}`
The /collections/all section is what matters since this can be replaced with a range of different blocking types, including (but not limited to):
- /collections/*?*filter* – common filter parameters
- /collections/*?*view* – how any products are displayed
This will make no sense if you have not explored this part of your Shopify site yet. However, editing the Shopify robots.txt file is relatively simple after some brief practice.
Remove a default rule from an existing group
Removing rules from the robots.txt file can be done by using specific kinds of code to hide certain rules.
While there is not usually a reason for most sites to do this, knowing how can still be important for many Shopify sites that want to temporarily change how pages are blocked off.
Add custom rules
Custom rules can be used to block things like the WayBackMachine.
Shopify stores start with no custom rules, but users may decide to block a user agent like the WayBackMachine archiver for personal reasons.
Why customise Robots.txt?
It might seem like most sites have no reason to alter the robots.txt file, and many Shopify sites stick with the default rules and robots.txt file, updated automatically as needed.
However, the robots.txt file is a vital part of optimising your site for search engines. The default Shopify robots.txt file does not just present product pages to search engines but almost all pages.
These preconfigured rules make it hard to manage two core parts of running ecommerce sites: crawl budget and thin content.
Crawl Budget
Search engines cannot crawl every page of every site. The more sites offer, the less of that site (proportionally) they will crawl.
Shopify’s default rules mean that most pages are visible, and that includes low-quality pages.
Due to this, a search engine might run out of resources while crawling your site and simply skip parts of your ecommerce site.
If the crawler only happened to monitor middling or low-quality parts of your site’s SEO, then your entire Shopify site could suffer a rankings slip due to pure chance.
Editing the robots.txt liquid file lets you block these pages from search results. By not allowing Google to see these pages, you can block Google from crawling these non-optimized pages.
Crawl Budget Issues
Keep in mind that crawling is unpredictable.
A crawler might choose an account page, or a shopping cart page/checkout page, or a template page, or some other page that is not meant to appear on search results anyway.
Using the file’s custom rules to prevent this from happening ensures that only desirable pages can appear in search results.
Thin Content
Some pages add no content to search engines. As mentioned directly above, a shopping cart page or internal search URLs are not meant to appear in searches – but crawlers do not know that.
Thin content is any page that adds no content to search engine users. These are often part of the site’s internal search and sitemap links, meant for existing site users, or even part of the site’s overall structure that users are not really supposed to visit.
It also includes pages that hold no marketing value of their own or cannot be accessed straight away, like checkout pages or account history pages.
Category pages can be a mixed bag. Some category pages are meant for customers to view, while others may be internal or meant purely as a backup navigation option.
Value-less pages like a search template or technical sources like faceted navigation URLs from a faceted navigation sitemap are important to consider too.
Thin Content on Search Engines
Platforms like Google Search Console do not know what each page is for.
For example, the URLs that internal search results render as part of the site’s standard internal search function could be picked up.
This also includes any kind of search template, page template, or any other technical sections of the site that are mostly to keep it all functioning well.
If a new user is directed straight to an empty template or an unfinished product page, they are far more likely to give up on the site entirely. Avoiding this can be incredibly important.
Should you remove thin content from your Shopify store?
In general, there is no need to remove this thin content – you just need to block it off from being seen.
Among the wider SEO community, there are a lot of proposed ways to separate customer-facing and site-functional pages, with crawler rules being the most dominant.
Using the custom rules method preserves Shopify’s ability to keep the site functioning as normal and allows you to continue using the same site structure that you always have.
Anything that messes around with the root directory and structure of your site can be a serious risk. The less you change about your site’s general functions on that level, the better.
Conclusion
The robots.txt file is a vital part of managing any Shopify site, and having access to it now makes a huge difference for many site owners.
Not only can Shopify users now steer crawler bots away from unwanted pages, but they can also set up custom rules to suit their specific needs.
While learning how to edit code in this file can be daunting at first, there are a lot of resources out there to make things much easier.
Of course, this file is not the only thing you need to consider when SEO is involved. SEO as a whole covers a wide range of different tasks and techniques, all of which have their own intended goals.
If you are a current Shopify owner or intend to become one in the near future, then take some time to learn how this new feature works.
Remember to always save a backup copy of your robots.txt file, just in case. Being able to revert to an old set of rules quickly can make a huge difference, especially if you are making changes rapidly or want to block specific pages only temporarily.
Whatever you do with this new feature, it is important to take your time and learn how it all works. While robots.txt can seem complicated when you first open it, learning to adjust its rules can be incredibly important.
