Googlebot cannot access CSS and JS files

[DaveOzric]

That's the fun part. Google blocks themselves and says we need to fix it. How in the heck do you allow a resource on their server via my robots.txt file?

[VThomas]

I may have removed one too many folders, but had some css in the cache, so this is the list I have for J2.5 site and google is happy today. :
User-agent: *
Disallow: /administrator/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /logs/
Disallow: /tmp/

Right now (weeks later), I don't recall why I removed modules and plugins folder. Should have taken better notes...

[DaveOzric]

Yep, just need some clarification on what impact this will have on search results. Component folder is a must for the disallow but the modules & plugins folder????

Anyone?

[trogladyte]

Won't help them blocking their own resources, but just read this reply on a Google groups forum that may help everyone.

I have the same problem! I have been advised to add to my robot txt file:

Allow: *.css
Allow: *.js

Hope it works.

Hmmm, sounded brilliant, but didn't work for me.

[billedwin]

I believe google want access to the media folder.

This is the new Joomla 3.5.3
As you can see the media folder does not appear on the new robots.txt file

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

[HunterWillis]

Nothing new in the article but Search Engine Land reported on it: http://searchengineland.com/google-sear ... css-226227.

So this happened today and SEL reported on it today. That means that Mashable will report on it... Friday?

This article actually has some useful info:

http://www.thesempost.com/new-google-se ... -js-files/

[HunterWillis]

Try this and see if it works for you:

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

# global
User-agent: *
Disallow: /administrator/
Disallow: /includes/
Disallow: /[ANY OTHER DIRECTORY YOU SEE FIT]

[Wildcat1]

Try this and see if it works for you:

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

# global
User-agent: *
Disallow: /administrator/
Disallow: /includes/
Disallow: /[ANY OTHER DIRECTORY YOU SEE FIT]

Thanks HunterWillis this worked for me. Now the only pages Google is complaining that it can't access are it's own maps pages haha,

[genegun]

Not a specialist but would that work?

Code: Select all

User-agent: Googlebot
Allow: /plugins/ *.css
Allow: /plugins/*.js
Allow: /modules/ *.css
Allow: /modules/*.js

[chrispapa]

This is definitely a part of Panda. This is an issue I noticed but thought little of as there seemed to be no decline in rankings on mobile since all of our client sites are responsive. I was reviewing the rendering and noted that it was text only on our Joomla clients. Now I know why - the JS and CSS isn't being indexed.

I also noticed I haven't had a full crawl on a Joomla site recently and it has been nauseating.

Does this mean that Joomla (and WP) needs to move the JS and CSS files to a root directory like /public_html/css and js?

[chrispapa]

One temporary workaround would be this fun addendum ...

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /media/jui/
Allow: /media/jui/css
Allow: /media/jui/js

[cyberrider]

This video https://www.[youtube].com/watch?v=6Io2tH-guRI recommends:

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

[genegun]

This video https://www.[[youtube]].com/watch?v=6Io2tH-guRI recommends:

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

but it this going to overrride the

Disallow: /modules/
Disallow: /plugins/

?

[cyberrider]

but it this going to overrride the

Disallow: /modules/
Disallow: /plugins/

?

Yes, but just for Googlebot and just for .js and .css files

[grafixpro]

I have implemented the changes that cyberrider suggested and got a "complete" status so that seemed to help.

[Faldinio]

I too just added:

Code: Select all

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

And that got me from Partial to Complete status.

[ribo]

if we will put

Code: Select all

Allow: *.css

Allow: *.js

will this solve our issue?

[ribo]

google blocks css and js in folders components, modules and plugins and as i saw this on robots.txt

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

allow css and js in google
My question is this will impact positive or negative in search results?

[ribo]

also i had this warning
Fix mobile usability issues found on http://www.zante-celebrations.gr/
To: Webmaster of
Google systems have tested 4 pages from your site and found that 50% of them have critical mobile usability errors. The errors on these 2 pages severely affect how mobile users are able to experience your website. These pages will not be seen as mobile-friendly by Google Search, and will therefore be displayed and ranked appropriately for smartphone users.
Fix this now:
1
Find problematic pages
View a report of the non-mobile-friendly pages found on your site, and the issues discovered
View a report
2
Learn about mobile-friendly design
There are a variety of techniques you can use to make your site mobile-friendly. Specifically, look for information about the issues brought up in Search Console
make your site mobile-friendly
3
Fix mobile usability issues on your site
Fix the issues preventing your site from being mobile-friendly

When i was view a report the non-mobile-friendly pages the google not see a mobile friendly some pages.
So when i allowed now css and js files in robots.txt the report of google see that pages mobile friendly.

[DebW]

My site has a file called robots.txt.dist as well as robots.txt
Do I have to add the googlebot allow commands to both files?

When I retest I'm also getting a 'partial' result and when I look to see why there are items described as 'temporarily unreachable'. DO I need to do something about these or just wait and re-test?

[ribo]

No need to put it in robots.txt.dist. just put what @cyberrider and @ HunterWillis said in robots.txt.
After check in google webmaster tools for blocked resources and do what it says to fix it. When you update robots.txt from GWT it takes 1-2 minutes to update it. check it

[ataman79]

Try this and see if it works for you:

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

# global
User-agent: *
Disallow: /administrator/
Disallow: /includes/
Disallow: /[ANY OTHER DIRECTORY YOU SEE FIT]

This should really work.
The only scripts, css which will be blocked are external ones - for ex. like twitter js, facebook .js, so on...

[Chacapamac]

Code: Select all

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

Work for me on Joomla 2.5 and 3