A description for this result is not available because of this site's robots.txt

[Root25]

Hello.

This show up in the a Google search for my website:A description for this result is not available because of this site's robots.txt.

Any ideas how to resolve this?

[sozzled]

You need to allow Google to index your website by fixing the problem within your website's robot.txt file. You created this problem; you need to undo what you did to fix the problem.

[Root25]

I didn't make any changes, and I don't know how to fix it. Could someone kindly explain how I can fix this?

[sozzled]

I didn't make any changes ...

A standard Joomla installation does not make it impossible for Google to index Joomla websites. If that were the case then no-one would be able to have their websites indexed by Google. Therefore even if you "didn't make any changes", someone changed something. My guess is that you've put index=nofollow on your page links.

Irononically, Google is the cure for this problem: when I searched for "A description for this result is not available because of this site's robots.txt" I discovered hundreds of places where this problem has been discussed and has been solved. I suggest that people should learn to make Google their friend.

Here's one possible answer (there may be others): https://productforums.google.com/forum/ ... JIO3uEnHsJ

[Root25]

sozzled, i'm not sure why you continue to respond with snarky remarks, but you are not being helpful. Could someone else kindly provide me with guidance on resolving this issue?

[numinousmedia]

Sozzled... this is common to all of my Joomla sites currently (several dozen). Your input blames the user without offering any solutions or thinking about alternative causes. At the very least, you could have found a decent Stack Overflow thread and posted a link. You can do better.

I'm digging into this as well Root25. I'm using the default robots.txt file on all of my sites.

Here are the contents of that file:

Code: Select all

# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

This is the default Joomla robots.txt. According to this robots.txt, all crawlers are blocked from crawling the folders in that list, but everything else should be available. But the Google messages suggest otherwise.

I'm finding some peculiar stuff as I dig. I'll post back here with a fix in the next twenty minutes I believe.

[numinousmedia]

The strange thing I'm finding is if I open up robots.txt in Sublime, I see the correct file. But if I simply visit the file in my browser, I see completely different contents.

You can see an example here: http://winesburgoh.com/robots.txt

[numinousmedia]

I've run into this issue now with sites on 2 different hosting providers. At Siteground, all of the robots.txt files on all of my sites contained this bad data telling all search engines not to crawl the site. In this case, I was able to update these files with a text editor and correct the problem.

At Inmotion, the files I'm attempting to update don't seem to be the file that Google sees. It looks like there's some sort of rule kicking in to replace the file when a browser (or crawler) hits it. I've not found any sort of rule in any .htaccess files to create this sort of rewrite. Inmotion support was stumped by it too. There's a support ticket out for it at the moment. I'll post back when I learn more.

Root25, as a first solution, I'd recommend you open up your robots.txt file in a text editor (Notepad, Sublime, whatever you prefer), and replace the contents with the following:

Code: Select all

# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

I would recommend resubmitting a site map in the Google Webmaster tools. Eventually your site meta description should reappear. It may take a few days to a couple weeks.

If you run into the same issue I did, where your robots.txt file is correct, but the browser/crawlers aren't seeing the right one, I'll try to provide a solution for this too.

[numinousmedia]

My final server related bug was just a dumb mistake on my part. I migrated several sites a long time ago between the hosts, and didn't realize who was hosted where, which led to the wrong robots.txt getting edited.

That still leaves the strange and slightly alarming problem of why so many of my Joomla sites had a bad robots.txt file. Here's a copy of what I was finding in those in case anyone is curious:

Code: Select all

User-agent: google
Disallow:

User-agent: yahoo
Disallow:

User-agent: msn
Disallow:

User-agent: *
Disallow: /

That last bit is the main problem. It tells all search engines not to index any part of your site. Obviously if you are wanting to be found online, this is a problem.

The fix is to replace this code with the default code that comes with Joomla (see above). The fact that nearly all of my Joomla sites had their robots.txt files changed like this is problematic and suggests a hack. I've run malware scans without turning up anything, but I'll post back if I find any trouble.

[Per Yngve Berg]

The file is installed as robots.txt.dist. You have to rename it.

Mod. Note: Relocated the topic to the SEO Forum.

[numinousmedia]

In all of my installs, there's already a robots.txt (as well as a robots.txt.dist). In some cases the original robots.txt was correct, but in many others, the robots.txt contained that "disallow all" line. This borked robots.txt doesn't ship with default Joomla, and I didn't change any of these.

[Per Yngve Berg]

The borked file is probably created by your host.

[Root25]

numinousmedia, thank you so much for your detailed response!!!! I really appreciate it!

I can't be sure, but I think the issue occurred after I did the latest Joomla update. I also host with Siteground and thought it was an error on their part.

I will try your steps, and let you know if i'm successful!