No crawled pages = 0 of my web

[MartinBalko]

Hi all,

I was very surprised when I saw some results in ahref s tool. In section of Crawled Pages (i think that it means Google indexed pages) i have 0 crawled pages - and i have no idea why. You can see it in picture.

Then I go to google webmasters tools and there i can see another problem - Google has no access to CSS and JavaScript files - you can see print screen. Then I make changes in robots.txt -

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/sampledata
Disallow: /images/sampledata
Disallow: /images/sampledata
Disallow: /images/sampledata
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

User-Agent: Googlebot (I have ADD that)
Allow: /*.js*
Allow: /*.css*

but it seem that it doesnt help at all.

Please, can anybody help me?

i have Joomla! 2.5.25 and web with that issues is

Code: Select all

http://www.parfumylacno.sk

Thank you very much,
Have a nice day.

[dhuelsmann]

This is the standard robots.txt file that comes with the Joomla installation.

Code: Select all

# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

Have you created a site map and submitted it to Google? I would recommend OsMap.

[MartinBalko]

thanks for reply, of course i have site map and have it submittd to Google:
Sitemap:

Code: Select all

 http://www.parfumylacno.sk/index.php?option=com_jmap&view=sitemap&format=xml

Sitemap:

Code: Select all

http://www.parfumylacno.sk/index.php?option=com_jmap&view=sitemap&format=images

but when you say it, i checked it in webmastertools and there is any error too: (see picture) in picture is this error 404 url (i have no idea what kind of article it is and why it gets error)

Code: Select all

http://www.parfumylacno.sk/index.php?option=com_content&view=article&id=280:najlacnejsie-parfumy-pre-alexandra&catid=25&Itemid=138

it have to be that article

Code: Select all

 http://www.parfumylacno.sk/vyber-parfumu/podla-mena/667-najvyhodnejsie-parfumy-pre-alexandra

but why are there 2 links for it? i should try delete this article, but i dont think that it should solved problem.

[dhuelsmann]

I just did a Google search on the word parfumylacno and got About 5,540 results (0.59 seconds). Looks like you have a lot of links out there.

[MartinBalko]

yes, link are in google (web is working since 2010 ) but there are another problems too when you search site:parfumylacno.sk in google most of results have Https:// but it have to be only http://! why is there https even it doesnt work and I have never setting https

and first problem still exists - Why Google doesnt index/crawl all my pages (only about 1%) It is very bad

[MartinBalko]

Please, can anyone help?

[tonypartridge]

Hello,

The president Blum is your rite allows for both. Http and https, google prefers https since it's secure and it's also a ranking factor. If you wish to only support http then you need to redirect https traffic back.

Please add this code to your htaccess file in the custom redirects section:

Code: Select all

DISABLE HTTPS:
RewriteCond %{HTTPS} on
RewriteRule .? http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

[MartinBalko]

hi, thanks for advice, but it seems not help = when I add your code to .htaccess

DISABLE HTTPS:
RewriteCond %{HTTPS} on
RewriteRule .? http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

then web stop working and there was a 500 Error - you can see in picture.

And I have changed the file robots.txt to default joomla robots.txt file, we will see if it would help. By the now are not any changes in webmasters tools, still error about that google doesnt have access to css and js.

thanks to all,

still looking for solution

[tonypartridge]

Sorry I shouldn't have included the DISABLE HTTPS: text

Try this:

Code: Select all

RewriteEngine On
RewriteCond %{HTTPS} on
RewriteRule .? http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

At the top of your .htaccess file, so it's redirected before doing anything.

Many thanks
Tony

[MartinBalko]

Thanks you very much, so i added this to htaccess

RewriteEngine On
RewriteCond %{HTTPS} on
RewriteRule .? http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Now - web is working, but redirect from https to http does not - you can see in picture (when i try to click on any https link in google serp on keyword site:parfumylacno.sk

Thank for your job and help.

Still trying to find solution

[tonypartridge]

Hello,

Did you process past the warning and ignore it?

If google is finding the https with an invalid https certificates I suspect it is ignoring the validity of the certificate when indexing it. So my solution would work in that case.

Many thanks
Tony

[MartinBalko]

Sorry i m not good in English. I cant understand what did you mean with "Did you process past the warning and ignore it?"

I was never using https certificates and really dont know why google is indexig my site with https = that is problem for what im looking solution.

and when i use your solution, it doesnt help in case that i was testing = in google SERP for keyword "site:parfumylacno.sk" i had click on any https: result and it was not redirect to http version. I thought, that with your solution it will be working = you click on https link in google and it would be automaticly redirect to http link. But it doesnt, so I suppose that i understood it in wrong way.

thank you for you patience and helping

[tonypartridge]

Hello,

There is no solution to making your site work with https and redirecting to http without installing a validated ssl certificate on your site, you will always be prompted with that warning.

In the above warning you get with the https link when going to your site you can click to ignore this warning and proceed. If you do that it should redirect you back to the http version with the htaccess code I provided, you can just update the google urls from http. But if you ask google to reminder you site it should help improve the reindexing within google.

Your site has an ssl certificate so google is trying to access it since it is encrypted and safer. Even if it is self signed so it can't be verified.

Many thanks
Tony

[MartinBalko]

Ok, thank you.

But question is - why google gives in search result links with https: I dont understand it, my site has never have ssl certificate, never set to https, so why google "thinks" opposite and gives https.

To your reply "Your site has an ssl certificate so google is trying to access it since it is encrypted and safer. Even if it is self signed so it can't be verified." = my site has not ssl certificate, never set, never has.

[leolam]

Couple of things here:

Your robot.txt is incorrect. The current default robot.txt in the Joomla distribution packages is adjusted to googles new algorithm. The content is (behind the blah, blah,

Code: Select all

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

the main change is that images and templates should be indexed.

Re your current site I do not many broken links on the site while surfing neither in Chrome nor on Firefox nor on the W3C link checker

The article you see indexed is an article that is located in category with id: 25, article id: 280 and menuitem id: 136

See if you have this still in your menu links maybe as trashed item.

Also it seems you site has been indexed while SEO unabled and with enabled. Dead links still exist with Google and you might need to reindex the site or ask Google to review this

Leo

[tonypartridge]

MartinBalko,

Your site has a self signed SSL Certificate hence why it loads over https:// you may have not installed it personally. But it is there. The certificate belongs to: WebSupport, s.r.o. with no domain specified. So I suspect it's your host who installed it.

My solution above should sort out your https issue in the long term.

Many thanks
Tony

[leolam]

I can concur that finding by Tony (nice one btw)

Leo

[MartinBalko]

thank you very much, i had write to my webhosting (you are right, it is websupport.sk) and they are searching for solution - by now they advised me to install SSL certificate. it is one of solutions, it should resolve problems with non working https: sites and it should little help of SEO effect. But, i think it should have some disavantages, i have to study something about it

thanks to all, For now i will be looking for solutions to remaining problems with non indexing

[MartinBalko]

The article you see indexed is an article that is located in category with id: 25, article id: 280 and menuitem id: 136

See if you have this still in your menu links maybe as trashed item.

thanks, yes i found article with id: 280 in trash - and many other articles = what is best way to do with them? should i remove them definitely even from trash?

Also it seems you site has been indexed while SEO unabled and with enabled. Dead links still exist with Google and you might need to reindex the site or ask Google to review this

Leo

yes, this sould be possible too, how can i fix it now? how can i reindex old dead links? i have redirect 404 sites to working sites, or homepage with joomla redirect component. is this solution of reindexing, or i have to do something else?

thanks you very much