The Joomla! Forum ™

Download

Demo


Board index » Joomla! Older Version Support » Joomla! 1.5 » Search Engine Optimization (Joomla! SEO) in Joomla! 1.5

Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  Page 1 of 1
  [ 5 posts ] 
  Print view Previous topic | Next topic 
Author Message
goosed
PostPosted: Sat Feb 18, 2012 4:02 am 
Joomla! Apprentice
Joomla! Apprentice

Joined: Mon Sep 07, 2009 9:07 pm
Posts: 21
Location: South Carolina
I was looking at the crawl errors for my website, and saw some things that I was not sure about. I have recently restricted some URL's that I did not want Google to crawl, and they now show up as errors. Here is an example:

Code:
http://www.mdtechteam.com/components/com_comment/joscomment


I have "Disallow: /components/" listed in my robots.txt. I was under the impression that this means "do not crawl any URL with /components in it". Why would it still crawl this URL? And more importantly, why would it show up as an error?

_________________
www.mdtechteam.com
Top
 Profile  
 
Raton
PostPosted: Sat Feb 18, 2012 11:20 am 
User avatar
Joomla! Enthusiast
Joomla! Enthusiast

Joined: Tue May 04, 2010 8:36 pm
Posts: 127
the rule you have set will stop Google from crawling /components/ and it's subdirectories.

this will not stop Google from crawling any url that has "components" in it as you say. Eg. somesite.com/directory/subdirectory/components will still be crawled.

If you want to do that (stop Google from crawling any url that has "components" in it) you should use this robots.txt rule: Disallow: /*components

You should not be concerned by that error. But, you should make sure Google does not index crap urls with no usefull content. To do that, do a site:yoursite.com in Google and have a look at the urls indexed.

_________________
Retete culinare, mancaruri => http://reteteaz.net
Legislatie actualizata gratis la zi => http://legeaz.net
Top
 Profile  
 
john-wh
PostPosted: Sat Feb 18, 2012 12:43 pm 
User avatar
Joomla! Apprentice
Joomla! Apprentice

Joined: Sat Feb 18, 2012 11:31 am
Posts: 20
Location: Bradenton, FL
In my opinion you have to clear all the errors from your Google WM,
or at least as much as possible.
The disallow you have added should do the job for this error, but even Google's software is not without bugs.
Top
 Profile  
 
goosed
PostPosted: Thu Feb 23, 2012 2:03 am 
Joomla! Apprentice
Joomla! Apprentice

Joined: Mon Sep 07, 2009 9:07 pm
Posts: 21
Location: South Carolina
So there is no real way to clear the errors off Google's end though right? All you can do is preventative maintenance. IE - set URLs that you do not want crawled in your robots.txt.

_________________
www.mdtechteam.com
Top
 Profile  
 
john-wh
PostPosted: Thu Feb 23, 2012 10:27 pm 
User avatar
Joomla! Apprentice
Joomla! Apprentice

Joined: Sat Feb 18, 2012 11:31 am
Posts: 20
Location: Bradenton, FL
Give it some time, maybe Google picks your revised version of robots.txt later on.

_________________
Proudly using Joomla at
http://www.webhosting-top10.com/
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  Page 1 of 1
  [ 5 posts ] 


Board index » Joomla! Older Version Support » Joomla! 1.5 » Search Engine Optimization (Joomla! SEO) in Joomla! 1.5

Who is online

Users browsing this forum: No registered users and 6 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
Powered by phpBB® Forum Software © phpBB Group