Robots problem after upgrade

[pot1906dk]

Hi

I just upgraded from J2.5 to 3.4. In the post installation notes it say that are changes in the robots file and that I should fix it manually.
However there no info on what to change nor where in the file. Somewhat poor information, I think. But after some search, I found that basically I should just copy the .dist file to my robots.txt.
After doing so, all seems fine. The reason for the changes should be new requirements from Google, to access places like /images and /plugins etc.
But when I run Googles test for responsiveness, it say that my site is responsive -- BUT there are resources that Google cannot access. It then list several files in /images and /plugins or subdirectories of these.
So it does not seem that the .dist file does things right. or how do I interpret this??

[imanickam]

If you just notice the differences in the robots.txt files between joomla! 2.5 and 3.x, you would notice that some of the directories (e.g., \images) for which the permission was denied for the robots are allowed now.

You would have noticed the following text in the file robots.txt or robots.txt.dist.

# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html

The document in the first link explains about the file robots.txt and the document in the second link validates the file robots.txt of a site.

[pot1906dk]

Thanks imanickam

I did notice all of this though, but it's not the solution.
In the robots.txt.dist file (which should work with J3) these are the directives :

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

With these in my robots.txt I still get lots of errors - even for directories not mentioned in the disallow directives.

If you test this page https://www.mindfulsolutions.dk/uddannelse you get several errors because Google cannot access /images, /plugins and /templates

Is it because I have set the IndexIgnore * in .htaccess? which I did because that was also what Joomla asked me to do after the upgrade.

/poul

[josh2015]

Google webmaster console have Robots Test Tool, it is a must use tool if you want to get everything right with your robots.
You can check of your robots.txt file status, it is funny and I don't know why
several time I have found that robots.txt test on physical drive is as update than one Google is reading.
Plz use this tool to check plus lots of other tweaking

[pot1906dk]

Hi Josh

I already did that. The Google test gives 0 errors and 0 warnings.
Everything seems to be as it should.

[ribo]

Hi

I just upgraded from J2.5 to 3.4. In the post installation notes it say that are changes in the robots file and that I should fix it manually.
However there no info on what to change nor where in the file. Somewhat poor information, I think. But after some search, I found that basically I should just copy the .dist file to my robots.txt.
After doing so, all seems fine. The reason for the changes should be new requirements from Google, to access places like /images and /plugins etc.
But when I run Googles test for responsiveness, it say that my site is responsive -- BUT there are resources that Google cannot access. It then list several files in /images and /plugins or subdirectories of these.
So it does not seem that the .dist file does things right. or how do I interpret this??

Just make the suggestion of joomla and also check this http://forum.joomla.org/viewtopic.php?f ... 0#p3319272 for Googles test for responsiveness

[pot1906dk]

Hi Ribo

Not sure exactly what you mean. I have made the exact changes that Joomla suggest, and still got the prolem.
The thread you link to suggest that these lines should do the trick:
Allow: /*.js*
Allow: /*.css*

However I already got them in my robots file, Here's the total file:
(which still gives errors for both /images and /templates)

Code: Select all

# If the Joomla site is installed within a folder such as at
# e.g. http://www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. http://www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/


Allow: /*.js*
Allow: /*.css*
Allow: /*.png*
Allow: /*.jpg*
Allow: /*.gif*
Allow: /*.js*
Allow: /*.css*
# Custom
Disallow: /download/
Disallow: /audio/
Disallow: /*?*      # Fjerner alle sider med ?parameter

# JSitemap entries
Sitemap: https://www.mindfulsolutions.dk/index.php?option=com_jmap&view=sitemap&format=xml
Sitemap: https://www.mindfulsolutions.dk/sitemap.xml

[ribo]

it did not said that.only for googlebot said

Code: Select all

# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

i just put at the top, you can put it at bottom if you want

[ribo]

Also you must check if your robots.txt is updated in gwt

[Per Yngve Berg]

Code: Select all

disallow: /*?*

This prevents Google from reading the Site Map.

[pot1906dk]

@Ribo
Isn't the code you suggest fully contained in mine? I use Useragent:* which mean all useragents, including Google. Otherwise the directives are just like mine.

@Per Ynge
Yeah, I know I will have to change it here too. As you can see, there are 2 sitemaps. In fact they are equal, so I ca just remove the one with a "?".
But that has nothing to do with my problem, I think?

[ribo]

@Ribo
Isn't the code you suggest fully contained in mine? I use Useragent:* which mean all useragents, including Google. Otherwise the directives are just like mine.

Unfortunately for some reason it works with "User-agent: Googlebot" and not with "User-agent: *" But my opinion is that these files it s better to allow only googlebot that google wants this for its reasons and not for other bots.

[pot1906dk]

OK, maybe the reason for Google Responsive Test giving errors constantly is in fact only temporary problems.
I tried to run the test a few times without changing anything in my robots file.
And the number of errors are not the same from to test to test.
Maybe it simply times out for some reason. So one should perhaps just look at the notification that the page is responsive, and then ignore all the error messages after that.

[ribo]

If you will do the robots as i show you google will see it as responsive if also your template is responsive, if your modules are responsive too. etc. Now don t ask me why google test likes "User-agent: Googlebot" and not with "User-agent: *" You can ask in google forums for that if you want. This is google issue and not joomla issue as when google say "jump" we must jump.

[pot1906dk]

Agree on the Google thing.
Anyway, Google has no problems with my page, and the test says that my page is responsive - which I basically knew.
I'm just wondering why Google still makes several errors about not being able to access this and that. But I'm pretty sure that it's got to do with their test being too slow and thus timing out occationally.
So I'll just leave it here :-)