Do you like to know how to find hidden pages on a website? Then you have completed the first step by visiting this website. The hidden pages could be intentional or accidental; accidental means there could be a broken link that makes a particular page invisible).
Fortunately, there are numerous ways to find a hidden page or broken links on a website including using the robots.txt file, log files, sitemap, online tools, etc.
Without wasting much time, let’s explore the hidden pages and how to find hidden pages on a website.
Why Pages Are Hidden On A Website?
Pages that aren’t accessed through a menu or navigation are known as hidden webpages. They can really be hard to discover, even if a reader can see them via anchor text or internal links. As discussed above, hidden pages are not always intentional; they could be accidental.
Pages are usually hidden to keep users away from sensitive information shared among website administrators and management. However, that isn’t always the reality; broken links may exist that the site owner is unaware of.
Access to a certain website is not feasible or valuable when they’re in construction or maintenance status. Another reason might be that accessing specific pages on a typical website necessitates login information. Whatever the reasons, we will resolve how to find hidden pages on a website in a few steps.
How To Find Hidden Pages On A Website?
There are several ways to find hidden pages on a website; robots.txt, manual process, log files, Google search, online tools, etc. To assist you, I personally tested all of the techniques and discuss them below.
Also, you do not even have to be a nerd to use these methods as long as you follow all of the procedures thoroughly.
How To Find Hidden Pages On A Website Using Google Search?
Google search is the easiest method to find any hidden pages across the Internet. You just have to type the keyword, website, and then press the enter button. Whether you are using Windows, Mac, or even a smartphone, you can easily find a page.
- Open the Google browser on any device.
- Head to the address bar and start typing the following code.
(Replace the “website” with your page keyword and “basicwebguide.com” with your desired site)
- Type what page you want to find; for example, if you want to find a page related to the “website” then type website on the address bar as in the below screenshot.
- Add keyword, add space, type site, colon, website name, and then press enter.
- Every one of the pages of that particular website appears on your screen immediately after you tap the enter key. Explore any of the links to find out more.
How To Find Hidden Pages On A Website Using Robots.txt?
Robots.txt is the most effective method to find out the pages on a website. You just have to type the site name on the search engine with robots.txt which discloses the hidden page or tells you if the owner has restricted any pages.
- Go to the browser you are using on your device.
- In the search bar, type the site name along with robots.txt as shown below.
- Replace basicwebguide.com with your site and press enter.
- When you press enter, you will see different entries or codes including allow, disallow, etc as seen in the below screenshot.
- Robots.txt are bots that indicate to web crawlers what pages should index or what not to index.
- It does not directly help you to find hidden pages but tells you if the website owner actually hides any page intentionally.
In the above case, crawlers and visitors are restricted to the admin board only. It also displays pages under the “disallow” tag if any page is hidden then you can find the page by using other discussed methods.
How To Find Hidden Link On A Website Using Source Code?
Many times, how to find hidden pages on a website simply doesn’t work, you need to look for the hidden link. To find a hidden link on a website, follow the below steps.
- Go to any website on which you want to find a hidden link or page.
- Perform right-click and click on the view page source option.
- You can also use the shortcut key for the source page: Ctrl+U.
- On the next page, you can see the massive mesh of codes; don’t worry, you don’t have to deal with this.
- Now, press Ctrl+F and type “href” on the search bar.
- This will highlight all the links inside the web page.
- You can check the individual link if you think any of these are hidden. You can do this by checking the URL.
How To Find Hidden Pages On A Website Using Online Tool?
Online tools to find the hidden pages might not much effective yet it is worth trying. One such fantastic tool to find hidden pages is URL Fuzzer in which you just have to enter the website URL or broken link (if desired).
- Go to the URL Fuzzer.
- You have two options; Light scan and Full scan. Select an option based on your requirements.
- In both cases, enter the full website URL like www.yourwebsite.com.
- Check on the terms and conditions box and hit the Scan Target button.
- In a few minutes, depending on the size of the website, it reveals all the pages whether hidden or broken.
How To Find Hidden Pages On Your Own Website?
On several occasions, website owners accidentally hide their own pages. When your site grows larger, you completely forget about the pages or links you used to hide. In this case, you can try these quick follow-ups to find hidden pages on your own website:
- Go through your site map file.
- Refer to the log of your site using cPanel. It reveals all the previous pages you ever created.
- Use WordPress plugins like Export All URLs to find out any hidden page or broken link.
- Explore google analytics which stores all the activity since you created your site.
- As a last resort, you can mail your webmaster (Google, Bing, etc) about hidden pages or any broken link.
It is all about how to find hidden pages on a website using simple methods. If you’ve read this far, you’ve realized that there are a plethora of methods for discovering hidden pages and broken links on any website.
By using the above methods, you can not only find hidden pages on others but also can find your own website. The frequently used techniques are using the robots.txt file, Google search, source code file, log files, etc.
I hope you enjoyed this post, which includes a step-by-step method to identify any hidden pages on the website. If you have any trouble while putting these techniques into practice, please let me know in the comments section.