Google cannot access CSS and JS Files

[swetasapra]

The robots.txt file is a handy little file located in the root directory of your joomla installation. At first glance in the file it looks to have very little information and very little use. However this is completely wrong, this file is very handy and it is actually looked for by the googlebot when it is scanning your website for pages.

[ribo]

As i understood google wants to have access to css and js files to see that we have a mobile friendly site. So in us in joomla except in templates that we have css and js, if we use third party extensions we can have these types of files also and in components, modules, plugins. So there are two ways to do it.
1. allow these file types in robots.txt (easy way)
2. change path for these file types (hard way)

[Jeroen07]

Adding these lines to robots.txt did the fix for our websites indeed, thanks for this!

User-Agent: Googlebot
Allow: /*.js*
Allow: /*.css*

Now, i've refetched as Google and these sources were still not crawlable:

templates/frontend/images/bg.jpg
templates/frontend/images/social_icons.png Afbeelding Tijdelijk onbereikbaar
templates/frontend/images/bg_news.png Afbeelding Tijdelijk onbereikbaar
https://scontent-iad3-1.xx.fbcdn.net/hp ... e=563ECEF5 Afbeelding Geblokkeerd
https://fbcdn-profile-a.akamaihd.net/hp ... 6474931b4a

All are images are "temporarily unavailable". The facebook images are "blocked".

- Regarding the Facebook images, this is normal I guess and there's no fix possible for this? I guess this won't harm our Google positions?
- Regarding the other images... They are in the template folder, and they are for sure there on the server... So why do we still get this error?

[Jeroen07]

Nobody who can give clarification here?

[Jeroen07]

Same for this website:

http://www.mobielebandenservice.be/

I get a temporarily unavailable for these files:

http://www.mobielebandenservice.be/temp ... rap.min.js
http://www.mobielebandenservice.be/temp ... ver-bg.png
http://www.mobielebandenservice.be/temp ... -title.jpg
http://www.mobielebandenservice.be/imag ... -hover.png

While this is robots.txt:

Code: Select all

# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

User-Agent: Googlebot
Allow: /*.js*
Allow: /*.css*


# JSitemap entries
Sitemap: http://www.mobielebandenservice.be/index.php?option=com_jmap&view=sitemap&format=xml

[winglonelion]

Same problem with me