Where Is Robots.txt In WordPress? How To Edit Or Modify Robots.txt In WordPress?

If you are a blogging enthusiast or a beginner, you might know that robots.txt plays a critical role in indexing or de-indexing the site’s every post and page. But are you ever wondered where is robots.txt in WordPress? Why is it necessary, and how to use it?

Robots.txt is an integral part of any website irrespective of the hosting platform. You may hear that visitors come to any website through the browser’s search results, but how a webpage appears there to the crawlers? How can one optimize the robots.txt for desired results and where to find robots.txt in WordPress? You will know all these in further section, read it completely.

What is Robots.txt in WordPress? How does robots.txt work?

Robots.txt is a set of instructions or commands for web crawling bots (or robots) for a website that crawlers are crawling on the internet.

The robots.txt page is the first page that bots consider before moving to other available pages of any site. So, you must know how to optimize the robots.txt file.

The bots or crawlers look for the robots.txt file to confirm if there are any special instructions for a particular webpage, like either indexing a page or blocking any page or URL. It means you can allow or disallow any specific page or URL of your website that you want to display or hide from the search engines.

After bots analyze the particular webpage or site, webmasters like Google Webmaster decide what to index or what to not. Accordingly, a webpage or website appears on search results for specific queries on the browser. If you instruct crawlers or bots not to index any page through the robots.txt file, it will not appear on search engines.

So now, you may get a basic idea of what robots.txt is in WordPress and how robots.txt works. I highly recommend that new bloggers not do unnecessary meddling in robots.txt files until it is too necessary. However, hiding your webpage or website is not like blocking a website from Google search results.

Why is the Robots.txt file important? Why use Robots.txt?

As I said, robots.txt files play a crucial role in how your website or webpages appear on search engines. You can instruct or command search crawlers or bots to crawl a specific directory, page, or area of your website. For instance, if a particular area of your website goes through a maintenance mode, then you can command the bots not to index that area.

On the other hand, if you want all of the pages or posts you publish to be indexed instantly and appear on google search results, then you instruct the bots for the same.

However, indexing or de-indexing is not only restricted to what you command to bots for a specific page, but a webpage can still appear on search results if it is linked with any other already indexed page.

Where is Robots.txt in WordPress? How to find robots.txt?

Once you acquaintance about robots.txt file and understand why robots.txt file necessary, now you may ask where is robots.txt in WordPress? Where can you locate this file?

The answer is: Your robots.txt file is located in your WordPress site root directory. To view your file, you need to connect your site using your cPanel’s file manager.

To view whether your website has a robots.txt file or not, just search on your browser (as shown above); yoursite.com/robots.txt, For example – basicwebguide.com/robots.txt.

I recommend using and modifying robots.txt files with SEO plugins rather than manually because plugins make it pretty easy to modify or create new robots.txt files. I am mentioning two popular plugins (Rank Math SEO & Yoast SEO).

Access robots.txt file using Rank Math SEO

• Login to your WordPress admin dashboard and head to the left side menu bar.
• Navigate to the Rank Math option(activate if not yet).
• Inside Rank Math, look out for General Settings and click on it.

• You can see various settings, click on Edit robots.txt.
• It is the default robots.txt file; you can edit it, update it if required, and click save changes.

Access robots.txt file using Yoast SEO

• Locate the SEO plugin as mentioned above steps.
• Under the SEO section, look for Tools that leads you to the new page where you can get your robots.txt file.

• Click on File Editor, and your robots.txt file will be displayed on the screen. You can either edit, modify, or create a new one, then save it finally.

These are the two popular plugins to find robots.txt files and answers to where is robots.txt in WordPress. However, you can also use dedicated plugins to locate the robots.txt file like Virtual Robots.txt, Better Robots.txt.

What does a robots.txt file look like?

The structure of robots.txt depends on what you are allowed to crawl and what you restrict. But a typical robots txt file looks like the following:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://basicwebguide.com/sitemap_index.xml

Here, User-agent is a bot or crawler, and (*) means you instruct all the bots to crawl your website or webpages.

Allow, disallow means what you allowed to crawl or what to disallowed to crawl, and in the above case, the WordPress admin area is restricted to crawl.

How to submit the Robots.txt file to Google?

Now you know where is robots.txt in WordPress and how to find this file using different plugins. But if you don’t see your robots.txt file, then you can submit it to Google using Search Engine Console. First, you need to create your WordPress robots.txt file as given above instructions or generate it from popular robots.txt generators.

To ensure you created or submitted the correct file, I recommend testing your file using Google’s robots.txt tester. It displayed your recently uploaded file; if you find any error or a different version, re-upload it. Uploading a correct file is a must to optimize robots.txt files.

Final Words

It is all about where is robots.txt in WordPress? Why is it necessary, and how to use it. I hope you also understand well what is robots.txt file is and how it works.

Robots.txt file is an inevitable part of any website, whether it is self-hosted or hosted by Google (blogger). However, creating a robots.txt file in WordPress is not mandatory because it is a default feature of WordPress.

I do not recommend making any changes in your robots.txt files until it is extremely required. Because if you have less knowledge of robots.txt file, you will destroy your site rather than improve.

The better way is to read this article again and again till you fully understand it, then go for any changes. If you encounter any issues, let me know through the comment section.