Googlebot can't access my site??!

Discuss Search Engine Optimization in relation to Joomla! 2.5. This forum will also have discussions on SEF/SEO Joomla! 2.5 extensions.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Locked
BrettBuc
Joomla! Intern
Joomla! Intern
Posts: 68
Joined: Thu Jun 28, 2012 5:02 pm

Googlebot can't access my site??!

Post by BrettBuc » Sun Aug 26, 2012 2:19 am

Hi everyone, all of a sudden, I am getting an error on webmaster tools that googlebot can't access my site. Any ideas on what I need to do? Sorry, i'm new!

Thanks!!
You do not have the required permissions to view the files attached to this post.

haryono
Joomla! Apprentice
Joomla! Apprentice
Posts: 12
Joined: Sun Aug 26, 2012 6:12 am
Location: Jakarta
Contact:

Re: Googlebot can't access my site??!

Post by haryono » Sun Aug 26, 2012 6:52 am

Show us your robot.txt

User avatar
Igal Zeifman
Joomla! Apprentice
Joomla! Apprentice
Posts: 6
Joined: Tue May 22, 2012 2:19 pm
Contact:

Re: Googlebot can't access my site??!

Post by Igal Zeifman » Sun Aug 26, 2012 2:06 pm

Hi,
My advice is (in the following order):

1.) Check your robots.txt for Disallow rules (i.e. 'Disallow:\' will ban all access)
2.) Check your source for no-index/no-follow tags
3.) If still no luck, go to WebMasterTools and try to "Fetch as Googlebot"
4.) Try to get a hold of your access logs and check for Googlebot user-agents to better understand your issue (when happened and why banned).

You can get full Googlebot user-agent list here (Botopedia.org)

BrettBuc
Joomla! Intern
Joomla! Intern
Posts: 68
Joined: Thu Jun 28, 2012 5:02 pm

Re: Googlebot can't access my site??!

Post by BrettBuc » Thu Sep 13, 2012 10:02 pm

haryono wrote:Show us your robot.txt
Is this what you mean? (see attached). Thanks!
You do not have the required permissions to view the files attached to this post.

flashgordonweb
Joomla! Intern
Joomla! Intern
Posts: 74
Joined: Sat Mar 29, 2008 3:52 pm

Re: Googlebot can't access my site??!

Post by flashgordonweb » Thu Sep 13, 2012 11:08 pm

Hello,

This is a known bug in Webmaster Tools. If you fetch as Googlebot Google will come back with a positive fetch. Then, a couple days will go by and you will get the same message. There are some threads on the Google Webmaster forum about this, but Google has not responded. It does not seem to be an issue with most sites and does not impact rankings.

BrettBuc
Joomla! Intern
Joomla! Intern
Posts: 68
Joined: Thu Jun 28, 2012 5:02 pm

Re: Googlebot can't access my site??!

Post by BrettBuc » Tue Sep 25, 2012 2:40 pm

flashgordonweb wrote:Hello,

This is a known bug in Webmaster Tools. If you fetch as Googlebot Google will come back with a positive fetch. Then, a couple days will go by and you will get the same message. There are some threads on the Google Webmaster forum about this, but Google has not responded. It does not seem to be an issue with most sites and does not impact rankings.
Thanks! I didn't know that. It has effected my site though. I was showing strong, and now nothing. Disappeared.

BrettBuc
Joomla! Intern
Joomla! Intern
Posts: 68
Joined: Thu Jun 28, 2012 5:02 pm

Re: Googlebot can't access my site??!

Post by BrettBuc » Thu Sep 27, 2012 10:39 pm

I just found this post in this forum! Could it be the cause? http://forum.joomla.org/viewtopic.php?f=618&t=757022

BrettBuc
Joomla! Intern
Joomla! Intern
Posts: 68
Joined: Thu Jun 28, 2012 5:02 pm

Re: Googlebot can't access my site??!

Post by BrettBuc » Fri Sep 28, 2012 3:03 pm

I'm totally confused now! (sorry...new to this ;-). And THANKS for the help!! I have one site that indexes/crawls fine. And the other site has an error message at the top of webmaster tools and no longer shows for searches (but showed strong up until this ocured). I'll also note that the site having this issue is an upgrade to 2.5.7 (this happened after i upgraded), and the site that works fine is not upgraded and is on 2.5.6. :

" Googlebot can't access your site
Aug 9, 2012"

They BOTH have the EXACT robots.txt contents. I'll paste them here:

GOOD SITE (CRAWLED):
# If the Joomla site is installed within a folder such as at
# e.g. http://www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. http://www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/


BAD SITE (NOT CRAWLED AND DISAPPEARED FROM SEARCH RESULTS AFTER SHOWING STRONG FOR A MONTH--and after I upgraded to 2.5.7):

# If the Joomla site is installed within a folder such as at
# e.g. http://www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. http://www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/


Let me know your thoughts. Thank you!!! BB

dbqt
Joomla! Apprentice
Joomla! Apprentice
Posts: 11
Joined: Wed Oct 03, 2012 12:53 am
Location: Jakarta
Contact:

Re: Googlebot can't access my site??!

Post by dbqt » Wed Oct 03, 2012 2:00 am

How many joomla sites in your host root? how you install them? in directory or use sub domain?

BrettBuc
Joomla! Intern
Joomla! Intern
Posts: 68
Joined: Thu Jun 28, 2012 5:02 pm

Re: Googlebot can't access my site??!

Post by BrettBuc » Wed Oct 03, 2012 4:48 pm

dbqt wrote:How many joomla sites in your host root? how you install them? in directory or use sub domain?
I have 3. I set them up through cloudaccess.net

apnavarun
Joomla! Apprentice
Joomla! Apprentice
Posts: 15
Joined: Fri Sep 28, 2012 6:08 am
Location: delhi

Re: Googlebot can't access my site??!

Post by apnavarun » Sat Oct 20, 2012 3:12 am

sometimes google finds it difficult to crawl website because of server problems
the same has been with my websites too, contact your hosting company. however these are temporarily problems.


Locked

Return to “Search Engine Optimization (Joomla! SEO) in Joomla! 2.5”