If you are a blogging enthusiast or a beginner, you might know that robots.txt plays a critical role in indexing or de-indexing the site’s every post and page. But are you ever wondered where is robots.txt in WordPress? Why is it necessary, and how to use it?
Robots.txt is an integral part of any website irrespective of the hosting platform. You may hear that visitors come to any website through the browser’s search results, but how a webpage appears there to the crawlers? How can one optimize the robots.txt for desired results and where to find robots.txt in WordPress? You will know all these in further section, read it completely.
Robots.txt is a set of instructions or commands for web crawling bots (or robots) for a website that crawlers are crawling on the internet.
The robots.txt page is the first page that bots consider before moving to other available pages of any site. So, you must know how to optimize the robots.txt file.
The bots or crawlers look for the robots.txt file to confirm if there are any special instructions for a particular webpage, like either indexing a page or blocking any page or URL. It means you can allow or disallow any specific page or URL of your website that you want to display or hide from the search engines.
After bots analyze the particular webpage or site, webmasters like Google Webmaster decide what to index or what to not. Accordingly, a webpage or website appears on search results for specific queries on the browser. If you instruct crawlers or bots not to index any page through the robots.txt file, it will not appear on search engines.
So now, you may get a basic idea of what robots.txt is in WordPress and how robots.txt works. I highly recommend that new bloggers not do unnecessary meddling in robots.txt files until it is too necessary. However, hiding your webpage or website is not like blocking a website from Google search results.
As I said, robots.txt files play a crucial role in how your website or webpages appear on search engines. You can instruct or command search crawlers or bots to crawl a specific directory, page, or area of your website. For instance, if a particular area of your website goes through a maintenance mode, then you can command the bots not to index that area.
On the other hand, if you want all of the pages or posts you publish to be indexed instantly and appear on google search results, then you instruct the bots for the same.
However, indexing or de-indexing is not only restricted to what you command to bots for a specific page, but a webpage can still appear on search results if it is linked with any other already indexed page.
Once you acquaintance about robots.txt file and understand why robots.txt file necessary, now you may ask where is robots.txt in WordPress? Where can you locate this file?
The answer is: Your robots.txt file is located in your WordPress site root directory. To view your file, you need to connect your site using your cPanel’s file manager.
To view whether your website has a robots.txt file or not, just search on your browser (as shown above); yoursite.com/robots.txt, For example – basicwebguide.com/robots.txt.
I recommend using and modifying robots.txt files with SEO plugins rather than manually because plugins make it pretty easy to modify or create new robots.txt files. I am mentioning two popular plugins (Rank Math SEO & Yoast SEO).
These are the two popular plugins to find robots.txt files and answers to where is robots.txt in WordPress. However, you can also use dedicated plugins to locate the robots.txt file like Virtual Robots.txt, Better Robots.txt.
The structure of robots.txt depends on what you are allowed to crawl and what you restrict. But a typical robots txt file looks like the following:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://basicwebguide.com/sitemap_index.xml
Here, User-agent is a bot or crawler, and (*) means you instruct all the bots to crawl your website or webpages.
Allow, disallow means what you allowed to crawl or what to disallowed to crawl, and in the above case, the WordPress admin area is restricted to crawl.
Now you know where is robots.txt in WordPress and how to find this file using different plugins. But if you don’t see your robots.txt file, then you can submit it to Google using Search Engine Console. First, you need to create your WordPress robots.txt file as given above instructions or generate it from popular robots.txt generators.
To ensure you created or submitted the correct file, I recommend testing your file using Google’s robots.txt tester. It displayed your recently uploaded file; if you find any error or a different version, re-upload it. Uploading a correct file is a must to optimize robots.txt files.
It is all about where is robots.txt in WordPress? Why is it necessary, and how to use it. I hope you also understand well what is robots.txt file is and how it works.
Robots.txt file is an inevitable part of any website, whether it is self-hosted or hosted by Google (blogger). However, creating a robots.txt file in WordPress is not mandatory because it is a default feature of WordPress.
I do not recommend making any changes in your robots.txt files until it is extremely required. Because if you have less knowledge of robots.txt file, you will destroy your site rather than improve.
The better way is to read this article again and again till you fully understand it, then go for any changes. If you encounter any issues, let me know through the comment section.
It is pretty annoying when you scroll down Reddit and see 'delete posts' or 'deleted…
Do you know how to take a screenshot on Mac? Screenshot helps you save a…
Are you looking for how to upload pdfs to WordPress? Then stay connected with this…
Are you looking for how to hide internet activity from a router? Then you have…
Everyone uses the Internet though no one owns the Internet as we discussed in this…
Do you want to learn what is a static page in WordPress? If you just…
View Comments