[wd_asp id=1]

How to update robots.txt in WordPress

Computer screen of a simplified robots.txt file

This post is part of The Complete Guide to WordPress Technical SEO. Also in this series:

Knowing how to update robots.txt is crucial for anybody who owns or maintains a website.  Robots.txt is a file that is used to communicate with search engine bots and can be used to block certain pages or sections of your website from being indexed. In this quick guide post, we will provide a step-by-step guide on how to update the robots.txt file in WordPress.

Creating a robots.txt file in Yoast SEO

Many site owners will choose to manage their robots.txt through third-party plugins to make it a lot easier. A very popular method is to use Yoast SEO – a great SEO tool that allows a decent amount of technical SEO customisation. In this guide, we’ll be covering how you can create a robots.txt file and manage it using Yoast as it is free to do so and incredibly easy to set up.

Once you have Yoast SEO installed, follow the below steps:

Step 1: Log in to your WordPress dashboard

The first step is to log in to your WordPress dashboard. This can be done by visiting your website’s URL and adding “/wp-admin” to the end of it. For example, if your website’s URL is “www.example.com”, you would visit “www.example.com/wp-admin” to log in.

Step 2: Click on ‘Yoast SEO’ in the admin menu.

The Yoast SEO menu interface

Step 3: Select “Tools”

The Yoast SEO interface with an arrowing pointing to "tools"

Step 4: Select ‘File editor

NOTE: Please note that this menu may not appear if your WordPress install has disabled file editing. It’s best to enable file editing if you can, or you can edit the file through FTP if needed.

Step 5: Select the ‘Create robots.txt file’ button

If you already have a robots.txt file created, you’ll instead be directed straight to the interface in Step 6.

Step 6: View or edit your robots.txt file generated by Yoast

Here, you’ll see your robots.txt file. By default, Yoast includes basic instructions by default such as a disallow line. This is where you’ll be specify which pages, or types of pages, are to be blocked from crawlers.

And that’s it! You now know how to create your robots.txt with Yoast.

How to block crawlers without plugins

If you want to be able to disallow crawlers from reaching certain pages without third-party plugins, or robots.txt customisation, you can do so by follwing the below steps:

  1. Log in to your WordPress dashboard
  2. Go to the “Settings” menu
  3. Click on “Reading”
  4. Scroll down to the “Search Engine Visibility” section
  5. Check the box labelled “Discourage search engines from indexing this site”
  6. Save your changes

By blocking certain pages or sections of your website from being indexed, you can ensure that only the most relevant content is shown in search engine results.

Leave a Reply

Your email address will not be published. Required fields are marked *

Share this article

Understanding Google Discover Google Discover is a feature by Google that provides users with a personalised feed of content based...
Understanding Your Target Audience Before diving into any marketing initiatives, it’s crucial to understand your target audience inside and out....
In the realm of e-commerce, a vital aspect of success lies in optimising your presence on platforms like Google Shopping....