How to browse and check like Google bot? User agent change firefox extension
I recently got a query a friend that his website is getting crawled by the forum is showing “You are not authorized to view this page” under Google cache. (You can check google cache by either typing cache:http://www.example.com/page-example.html under Google search or clicking on the cache link that appears under SERP (search engine result pages))
Google Bot is an user agent
Google Bot (Program that Google uses to fetch your page) comes under the agent name “GoogleBot” (With some version number). This also allows some people to block Google agents from crawling the content. Some people block Google block from reading the content from privacy concerns, copyright issues and other agreements.
How to check as a Google bot
The easiest way is to use Firefox extension https://addons.mozilla.org/en-US/firefox/addon/59 and change the user agent to Google bot. Once the extension is installed, it will prompt for a firefox restart. Once the firefox is restarted then go to Tools > User Agent Switcher > Options > Option .
The add a new user agent as
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Tools > User Agent Switcher >
(Select Google Bot as the user agent and browse the page to ensure that Google bot is not blocked from reading the content)