Joomla! Forum - community, help and support

Hi everyone, all of a sudden, I am getting an error on webmaster tools that googlebot can't access my site. Any ideas on what I need to do? Sorry, i'm new!

Thanks!!

Show us your robot.txt

Hi,
My advice is (in the following order):

1.) Check your robots.txt for Disallow rules (i.e. 'Disallow:\' will ban all access)
2.) Check your source for no-index/no-follow tags
3.) If still no luck, go to WebMasterTools and try to "Fetch as Googlebot"
4.) Try to get a hold of your access logs and check for Googlebot user-agents to better understand your issue (when happened and why banned).

You can get full Googlebot user-agent list here (Botopedia.org)

haryono wrote:Show us your robot.txt

Is this what you mean? (see attached). Thanks!

Hello,

This is a known bug in Webmaster Tools. If you fetch as Googlebot Google will come back with a positive fetch. Then, a couple days will go by and you will get the same message. There are some threads on the Google Webmaster forum about this, but Google has not responded. It does not seem to be an issue with most sites and does not impact rankings.

flashgordonweb wrote:Hello,

This is a known bug in Webmaster Tools. If you fetch as Googlebot Google will come back with a positive fetch. Then, a couple days will go by and you will get the same message. There are some threads on the Google Webmaster forum about this, but Google has not responded. It does not seem to be an issue with most sites and does not impact rankings.

Thanks! I didn't know that. It has effected my site though. I was showing strong, and now nothing. Disappeared.

I just found this post in this forum! Could it be the cause? http://forum.joomla.org/viewtopic.php?f=618&t=757022

I'm totally confused now! (sorry...new to this ;-). And THANKS for the help!! I have one site that indexes/crawls fine. And the other site has an error message at the top of webmaster tools and no longer shows for searches (but showed strong up until this ocured). I'll also note that the site having this issue is an upgrade to 2.5.7 (this happened after i upgraded), and the site that works fine is not upgraded and is on 2.5.6. :

" Googlebot can't access your site
Aug 9, 2012"

They BOTH have the EXACT robots.txt contents. I'll paste them here:

GOOD SITE (CRAWLED):
# If the Joomla site is installed within a folder such as at
# e.g. http://www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. http://www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/


BAD SITE (NOT CRAWLED AND DISAPPEARED FROM SEARCH RESULTS AFTER SHOWING STRONG FOR A MONTH--and after I upgraded to 2.5.7):

# If the Joomla site is installed within a folder such as at
# e.g. http://www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. http://www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/


Let me know your thoughts. Thank you!!! BB

How many joomla sites in your host root? how you install them? in directory or use sub domain?

dbqt wrote:How many joomla sites in your host root? how you install them? in directory or use sub domain?

I have 3. I set them up through cloudaccess.net

sometimes google finds it difficult to crawl website because of server problems
the same has been with my websites too, contact your hosting company. however these are temporarily problems.