{"id":789,"date":"2022-01-26T04:51:21","date_gmt":"2022-01-25T23:21:21","guid":{"rendered":"https:\/\/basicwebguide.com\/?p=789"},"modified":"2022-01-26T04:51:21","modified_gmt":"2022-01-25T23:21:21","slug":"where-is-robots-txt-in-wordpress","status":"publish","type":"post","link":"https:\/\/basicwebguide.com\/where-is-robots-txt-in-wordpress\/","title":{"rendered":"Where Is Robots.txt In WordPress? How To Edit Or Modify Robots.txt In WordPress?"},"content":{"rendered":"<p>If you are a blogging enthusiast or a beginner, you might know that <strong>robots.txt<\/strong> plays a critical role in indexing or de-indexing the site&#8217;s every post and page. But are you ever wondered <strong>where is robots.txt in WordPress?<\/strong> Why is it necessary, and how to use it?<\/p>\n<p>Robots.txt is an integral part of any website irrespective of the hosting platform. You may hear that visitors come to any website through the <strong>browser&#8217;s search results,<\/strong> but how a webpage appears there to the crawlers? How can one <strong>optimize the robots.txt<\/strong> for desired results and <strong>where to find robots.txt in WordPress?<\/strong> You will know all these in further section, read it completely.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_61 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title \" >Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/basicwebguide.com\/where-is-robots-txt-in-wordpress\/#What_is_Robotstxt_in_WordPress_How_does_robotstxt_work\" title=\"What is Robots.txt in WordPress? How does robots.txt work?\">What is Robots.txt in WordPress? How does robots.txt work?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/basicwebguide.com\/where-is-robots-txt-in-wordpress\/#Why_is_the_Robotstxt_file_important_Why_use_Robotstxt\" title=\"Why is the Robots.txt file important? Why use Robots.txt?\">Why is the Robots.txt file important? Why use Robots.txt?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/basicwebguide.com\/where-is-robots-txt-in-wordpress\/#Where_is_Robotstxt_in_WordPress_How_to_find_robotstxt\" title=\"Where is Robots.txt in WordPress? How to find robots.txt?\">Where is Robots.txt in WordPress? How to find robots.txt?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/basicwebguide.com\/where-is-robots-txt-in-wordpress\/#Access_robotstxt_file_using_Rank_Math_SEO\" title=\"Access robots.txt file using Rank Math SEO\">Access robots.txt file using Rank Math SEO<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/basicwebguide.com\/where-is-robots-txt-in-wordpress\/#Access_robotstxt_file_using_Yoast_SEO\" title=\"Access robots.txt file using Yoast SEO\">Access robots.txt file using Yoast SEO<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/basicwebguide.com\/where-is-robots-txt-in-wordpress\/#What_does_a_robotstxt_file_look_like\" title=\"What does a robots.txt file look like?\">What does a robots.txt file look like?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/basicwebguide.com\/where-is-robots-txt-in-wordpress\/#How_to_submit_the_Robotstxt_file_to_Google\" title=\"How to submit the Robots.txt file to Google?\">How to submit the Robots.txt file to Google?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/basicwebguide.com\/where-is-robots-txt-in-wordpress\/#Final_Words\" title=\"Final Words\">Final Words<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"What_is_Robotstxt_in_WordPress_How_does_robotstxt_work\"><\/span><strong>What is Robots.txt in WordPress? How does robots.txt work?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Robots.txt is a set of instructions or <strong>commands for web crawling bots<\/strong> (or robots) for a website that crawlers are crawling on the internet.<\/p>\n<p>The robots.txt page is the first page that bots consider before moving to other available pages of any site. So, you must know <strong>how to optimize the robots.txt file<\/strong>.<\/p>\n<p>The bots or crawlers look for the <strong>robots.txt<\/strong> file to confirm if there are any special instructions for a particular webpage, like either indexing a page or blocking any page or URL. It means you can <strong>allow or disallow any specific page<\/strong> or URL of your website that you want to display or hide from the search engines.<\/p>\n<p>After bots analyze the particular webpage or site, webmasters like <strong>Google Webmaster<\/strong> decide <strong>what to index or what to not. Accordingly<\/strong>, a webpage or website appears on search results for specific queries on the browser. If you instruct crawlers or bots not to index any page through the robots.txt file, it will not appear on search engines.<\/p>\n<p>So now, you may get a basic idea of what robots.txt is in WordPress and how robots.txt works. I <strong>highly recommend<\/strong> that new bloggers not do unnecessary meddling in robots.txt files until it is too necessary. However, hiding your webpage or website is not like <a href=\"https:\/\/basicwebguide.com\/how-to-block-a-website-from-google-search\/\">blocking a website from Google search results<\/a>.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Why_is_the_Robotstxt_file_important_Why_use_Robotstxt\"><\/span><strong>Why is the Robots.txt file important? Why use Robots.txt?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>As I said, robots.txt files play a crucial role in how your website or webpages appear on search engines. You can instruct or command search crawlers or bots to crawl a specific directory, page, or area of your website. For instance, if a particular area of your website goes through a maintenance mode, then <strong>you can command the bots not to index<\/strong> that area.<\/p>\n<p>On the other hand, if you want all of the pages or posts you publish to be indexed instantly and appear on google search results, then you instruct the bots for the same.<\/p>\n<p>However, indexing or de-indexing is not only restricted to what you command to bots for a specific page, but a webpage can <strong>still appear on search results<\/strong> if it is linked with any other already indexed page.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Where_is_Robotstxt_in_WordPress_How_to_find_robotstxt\"><\/span><strong>Where is Robots.txt in WordPress? How to find robots.txt?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Once you acquaintance about robots.txt file and understand why robots.txt file necessary, now you may ask <strong>where is robots.txt in WordPress?<\/strong> Where can you locate this file?<\/p>\n<p>The answer is: Your <strong>robots.txt file<\/strong> is located in your <strong>WordPress site root directory<\/strong>. To view your file, you need to connect your site using your cPanel\u2019s file manager.<\/p>\n<p style=\"text-align: center;\"><img fetchpriority=\"high\" decoding=\"async\" class=\"wp-image-793 size-full aligncenter\" src=\"https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-145841.png\" alt=\"Where is Robots.txt in WordPress\" width=\"622\" height=\"367\" srcset=\"https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-145841.png 622w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-145841-300x177.png 300w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-145841-20x12.png 20w\" sizes=\"(max-width: 622px) 100vw, 622px\" \/><\/p>\n<p>To view whether your website has a robots.txt file or not, just search on your browser (as shown above); yoursite.com\/robots.txt, For example &#8211;<em> basicwebguide.com\/robots.txt.<\/em><\/p>\n<p>I recommend using and modifying robots.txt files with SEO plugins rather than manually because plugins make it pretty easy to modify or <strong>create new robots.txt files<\/strong>. I am mentioning two popular plugins (<strong>Rank Math SEO &amp; Yoast SEO<\/strong>).<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Access_robotstxt_file_using_Rank_Math_SEO\"><\/span><strong>Access robots.txt file using Rank Math SEO<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li><em>Login<\/em> to your WordPress admin dashboard and head to the left side menu bar.<\/li>\n<li>Navigate to the <strong><em>Rank Math<\/em><\/strong> option(activate if not yet).<\/li>\n<li>Inside Rank Math, look out for <em>General Settings<\/em> and click on it.<\/li>\n<\/ul>\n<p style=\"text-align: center;\"><img decoding=\"async\" class=\"aligncenter wp-image-795 size-full\" src=\"https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-151848.png\" alt=\"Where is Robots.txt in WordPress\" width=\"724\" height=\"466\" srcset=\"https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-151848.png 724w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-151848-300x193.png 300w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-151848-20x13.png 20w\" sizes=\"(max-width: 724px) 100vw, 724px\" \/><\/p>\n<ul>\n<li>You can see various settings, click on Edit robots.txt.<\/li>\n<li>It is the default robots.txt file; you can edit it, update it if required, and click save changes.<\/li>\n<\/ul>\n<p><img decoding=\"async\" class=\"wp-image-796 size-full alignnone\" style=\"text-align: center;\" src=\"https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-152127.png\" alt=\"Where is Robots.txt in WordPress\" width=\"1363\" height=\"399\" srcset=\"https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-152127.png 1363w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-152127-300x88.png 300w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-152127-1024x300.png 1024w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-152127-768x225.png 768w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-152127-20x6.png 20w\" sizes=\"(max-width: 1363px) 100vw, 1363px\" \/><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Access_robotstxt_file_using_Yoast_SEO\"><\/span><strong>Access robots.txt file using Yoast SEO<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li>Locate the <strong>SEO plugin<\/strong> as mentioned above steps.<\/li>\n<li>Under the <strong>SEO<\/strong> section, look for <strong>Tools<\/strong> that leads you to the new page where you can get your robots.txt file.<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-798 size-full aligncenter\" src=\"https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-160136.png\" alt=\"how to find robots.txt file\" width=\"639\" height=\"349\" srcset=\"https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-160136.png 639w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-160136-300x164.png 300w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-160136-20x11.png 20w\" sizes=\"(max-width: 639px) 100vw, 639px\" \/><\/p>\n<ul>\n<li>Click on <strong>File Editor,<\/strong> and your robots.txt file will be displayed on the screen. You can either edit, modify, or create a new one, then save it finally.<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-799 size-full aligncenter\" src=\"https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-160633.png\" alt=\"edit robots.txt file\" width=\"567\" height=\"348\" srcset=\"https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-160633.png 567w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-160633-300x184.png 300w, https:\/\/basicwebguide.com\/wp-content\/uploads\/2022\/01\/Screenshot-2022-01-24-160633-20x12.png 20w\" sizes=\"(max-width: 567px) 100vw, 567px\" \/><\/p>\n<p>These are the two popular plugins to find robots.txt files and answers to where is robots.txt in WordPress. However, you can also use dedicated plugins to locate the robots.txt file like <a class=\"rank-math-link\" href=\"https:\/\/wordpress.org\/plugins\/pc-robotstxt\/\" target=\"_blank\" rel=\"noopener\">Virtual Robots.txt,<\/a>\u00a0<a href=\"https:\/\/wordpress.org\/plugins\/better-robots-txt\/\" target=\"_blank\" rel=\"noopener\">Better Robots.txt<\/a>.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"What_does_a_robotstxt_file_look_like\"><\/span><strong>What does a robots.txt file look like?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The structure of robots.txt depends on what you are allowed to crawl and what you restrict. But a typical robots txt file looks like the following:<\/p>\n<pre>User-agent: *\r\nDisallow: \/wp-admin\/\r\nAllow: \/wp-admin\/admin-ajax.php\r\n\r\nSitemap: https:\/\/basicwebguide.com\/sitemap_index.xml<\/pre>\n<p>Here, User-agent is a bot or crawler, and <strong>(*) means you instruct all the bots to crawl<\/strong> your website or webpages.<\/p>\n<p>Allow, disallow means what you allowed to crawl or what to disallowed to crawl, and in the above case, the WordPress admin area is restricted to crawl.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"How_to_submit_the_Robotstxt_file_to_Google\"><\/span><strong>How to submit the Robots.txt file to Google?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Now you know where is robots.txt in WordPress and how to find this file using different plugins. But if you don&#8217;t see your robots.txt file, then you can submit it to Google using <strong>Search Engine Console<\/strong>. First, you need to <strong>create your WordPress robots.txt file<\/strong> as given above instructions or generate it from popular <a href=\"https:\/\/developers.google.com\/search\/docs\/advanced\/robots\/create-robots-txt\" target=\"_blank\" rel=\"noopener\">robots.txt generators<\/a>.<\/p>\n<p>To ensure you created or submitted the correct file, I recommend testing your file using <a href=\"https:\/\/www.google.com\/webmasters\/tools\/robots-testing-tool\" target=\"_blank\" rel=\"noopener\">Google&#8217;s robots.txt tester<\/a>. It displayed your recently uploaded file; if you find any error or a different version, re-upload it. Uploading a correct file is a must to optimize <strong>robots.txt files.<\/strong><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Final_Words\"><\/span><strong>Final Words<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>It is all about where is robots.txt in WordPress? Why is it necessary, and how to use it. I hope you also understand well what is robots.txt file is and how it works.<\/p>\n<p>Robots.txt file is an inevitable part of any website, whether it is self-hosted or hosted by Google (blogger). However, creating a robots.txt file in WordPress is not mandatory because it is a default feature of WordPress.<\/p>\n<p>I do not recommend making any changes in your robots.txt files until it is extremely required. Because if you have less knowledge of robots.txt file, you will destroy your site rather than improve.<\/p>\n<p>The better way is to <strong>read this article again and again<\/strong> till you fully understand it, then go for any changes. If you encounter any issues, let me know through the comment section.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you are a blogging enthusiast or a beginner, you might know that robots.txt plays a critical role in indexing or de-indexing the site&#8217;s every post and page. But are you ever wondered where is robots.txt in WordPress? Why is it necessary, and how to use it? Robots.txt is an integral part of any website [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":808,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[362],"tags":[1041,1058,1060,1059,1038,1055,1037,1045,1051,1056,1049,1042,1039,1057,1052,1036,1053,1047,1035,1054,1040,1048,1046,1044,1043,1050],"_links":{"self":[{"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/posts\/789"}],"collection":[{"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/comments?post=789"}],"version-history":[{"count":10,"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/posts\/789\/revisions"}],"predecessor-version":[{"id":809,"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/posts\/789\/revisions\/809"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/media\/808"}],"wp:attachment":[{"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/media?parent=789"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/categories?post=789"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/basicwebguide.com\/wp-json\/wp\/v2\/tags?post=789"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}