No crawled pages = 0 of my web

Discuss Search Engine Optimization in relation to Joomla! 2.5. This forum will also have discussions on SEF/SEO Joomla! 2.5 extensions.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Locked
MartinBalko
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Tue Nov 24, 2009 10:24 am
Contact:

No crawled pages = 0 of my web

Post by MartinBalko » Fri Sep 02, 2016 3:27 pm

Hi all,

I was very surprised when I saw some results in ahref s tool. In section of Crawled Pages (i think that it means Google indexed pages) i have 0 crawled pages - and i have no idea why. You can see it in picture.

Then I go to google webmasters tools and there i can see another problem - Google has no access to CSS and JavaScript files - you can see print screen. Then I make changes in robots.txt -

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/sampledata
Disallow: /images/sampledata
Disallow: /images/sampledata
Disallow: /images/sampledata
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

User-Agent: Googlebot (I have ADD that)
Allow: /*.js*
Allow: /*.css*

but it seem that it doesnt help at all.

Please, can anybody help me? :)

i have Joomla! 2.5.25 and web with that issues is

Code: Select all

http://www.parfumylacno.sk
Thank you very much,
Have a nice day.
You do not have the required permissions to view the files attached to this post.

User avatar
dhuelsmann
Joomla! Master
Joomla! Master
Posts: 19643
Joined: Sun Oct 02, 2005 12:50 am
Location: Omaha, NE
Contact:

Re: No crawled pages = 0 of my web

Post by dhuelsmann » Fri Sep 02, 2016 6:02 pm

This is the standard robots.txt file that comes with the Joomla installation.

Code: Select all

# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/
Have you created a site map and submitted it to Google? I would recommend OsMap.
Regards, Dave
Past Treasurer Open Source Matters, Inc.
Past Global Moderator
http://www.kiwaniswest.org

MartinBalko
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Tue Nov 24, 2009 10:24 am
Contact:

Re: No crawled pages = 0 of my web

Post by MartinBalko » Fri Sep 02, 2016 7:22 pm

thanks for reply, of course i have site map and have it submittd to Google:
Sitemap:

Code: Select all

 http://www.parfumylacno.sk/index.php?option=com_jmap&view=sitemap&format=xml
Sitemap:

Code: Select all

http://www.parfumylacno.sk/index.php?option=com_jmap&view=sitemap&format=images
but when you say it, i checked it in webmastertools and there is any error too: (see picture) in picture is this error 404 url (i have no idea what kind of article it is and why it gets error)

Code: Select all

http://www.parfumylacno.sk/index.php?option=com_content&view=article&id=280:najlacnejsie-parfumy-pre-alexandra&catid=25&Itemid=138
it have to be that article

Code: Select all

 http://www.parfumylacno.sk/vyber-parfumu/podla-mena/667-najvyhodnejsie-parfumy-pre-alexandra
but why are there 2 links for it? i should try delete this article, but i dont think that it should solved problem.
You do not have the required permissions to view the files attached to this post.

User avatar
dhuelsmann
Joomla! Master
Joomla! Master
Posts: 19643
Joined: Sun Oct 02, 2005 12:50 am
Location: Omaha, NE
Contact:

Re: No crawled pages = 0 of my web

Post by dhuelsmann » Fri Sep 02, 2016 7:48 pm

I just did a Google search on the word parfumylacno and got About 5,540 results (0.59 seconds). Looks like you have a lot of links out there.
Regards, Dave
Past Treasurer Open Source Matters, Inc.
Past Global Moderator
http://www.kiwaniswest.org

MartinBalko
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Tue Nov 24, 2009 10:24 am
Contact:

Re: No crawled pages = 0 of my web

Post by MartinBalko » Sat Sep 03, 2016 6:04 am

yes, link are in google (web is working since 2010 ) but there are another problems too :D when you search site:parfumylacno.sk in google most of results have Https:// but it have to be only http://! why is there https even it doesnt work and I have never setting https :)

and first problem still exists - Why Google doesnt index/crawl all my pages (only about 1%) It is very bad :)
You do not have the required permissions to view the files attached to this post.

MartinBalko
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Tue Nov 24, 2009 10:24 am
Contact:

Re: No crawled pages = 0 of my web

Post by MartinBalko » Mon Sep 05, 2016 6:31 am

Please, can anyone help? :)

tonypartridge
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 138
Joined: Sat Sep 03, 2016 7:37 am

Re: No crawled pages = 0 of my web

Post by tonypartridge » Mon Sep 05, 2016 7:25 am

Hello,

The president Blum is your rite allows for both. Http and https, google prefers https since it's secure and it's also a ranking factor. If you wish to only support http then you need to redirect https traffic back.

Please add this code to your htaccess file in the custom redirects section:

Code: Select all

DISABLE HTTPS:
RewriteCond %{HTTPS} on
RewriteRule .? http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

MartinBalko
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Tue Nov 24, 2009 10:24 am
Contact:

Re: No crawled pages = 0 of my web

Post by MartinBalko » Mon Sep 05, 2016 11:25 am

hi, thanks for advice, but it seems not help = when I add your code to .htaccess

DISABLE HTTPS:
RewriteCond %{HTTPS} on
RewriteRule .? http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

then web stop working and there was a 500 Error - you can see in picture.

And I have changed the file robots.txt to default joomla robots.txt file, we will see if it would help. By the now are not any changes in webmasters tools, still error about that google doesnt have access to css and js.

thanks to all,

still looking for solution
You do not have the required permissions to view the files attached to this post.

tonypartridge
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 138
Joined: Sat Sep 03, 2016 7:37 am

Re: No crawled pages = 0 of my web

Post by tonypartridge » Mon Sep 05, 2016 3:36 pm

Sorry I shouldn't have included the DISABLE HTTPS: text

Try this:

Code: Select all

RewriteEngine On
RewriteCond %{HTTPS} on
RewriteRule .? http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
At the top of your .htaccess file, so it's redirected before doing anything.

Many thanks
Tony

MartinBalko
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Tue Nov 24, 2009 10:24 am
Contact:

Re: No crawled pages = 0 of my web

Post by MartinBalko » Mon Sep 05, 2016 3:49 pm

Thanks you very much, so i added this to htaccess

RewriteEngine On
RewriteCond %{HTTPS} on
RewriteRule .? http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Now - web is working, but redirect from https to http does not - you can see in picture (when i try to click on any https link in google serp on keyword site:parfumylacno.sk

Thank for your job and help.

Still trying to find solution :)
You do not have the required permissions to view the files attached to this post.

tonypartridge
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 138
Joined: Sat Sep 03, 2016 7:37 am

Re: No crawled pages = 0 of my web

Post by tonypartridge » Mon Sep 05, 2016 3:59 pm

Hello,

Did you process past the warning and ignore it?

If google is finding the https with an invalid https certificates I suspect it is ignoring the validity of the certificate when indexing it. So my solution would work in that case.

Many thanks
Tony

MartinBalko
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Tue Nov 24, 2009 10:24 am
Contact:

Re: No crawled pages = 0 of my web

Post by MartinBalko » Mon Sep 05, 2016 4:14 pm

Sorry i m not good in English. I cant understand what did you mean with "Did you process past the warning and ignore it?"

I was never using https certificates and really dont know why google is indexig my site with https = that is problem for what im looking solution.

and when i use your solution, it doesnt help in case that i was testing = in google SERP for keyword "site:parfumylacno.sk" i had click on any https: result and it was not redirect to http version. I thought, that with your solution it will be working = you click on https link in google and it would be automaticly redirect to http link. But it doesnt, so I suppose that i understood it in wrong way. :)

thank you for you patience and helping :)

tonypartridge
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 138
Joined: Sat Sep 03, 2016 7:37 am

Re: No crawled pages = 0 of my web

Post by tonypartridge » Tue Sep 06, 2016 7:14 am

Hello,

There is no solution to making your site work with https and redirecting to http without installing a validated ssl certificate on your site, you will always be prompted with that warning.

In the above warning you get with the https link when going to your site you can click to ignore this warning and proceed. If you do that it should redirect you back to the http version with the htaccess code I provided, you can just update the google urls from http. But if you ask google to reminder you site it should help improve the reindexing within google.

Your site has an ssl certificate so google is trying to access it since it is encrypted and safer. Even if it is self signed so it can't be verified.

Many thanks
Tony

MartinBalko
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Tue Nov 24, 2009 10:24 am
Contact:

Re: No crawled pages = 0 of my web

Post by MartinBalko » Tue Sep 06, 2016 7:35 am

Ok, thank you.

But question is - why google gives in search result links with https: I dont understand it, my site has never have ssl certificate, never set to https, so why google "thinks" opposite and gives https.

To your reply "Your site has an ssl certificate so google is trying to access it since it is encrypted and safer. Even if it is self signed so it can't be verified." = my site has not ssl certificate, never set, never has. :)

User avatar
leolam
Joomla! Master
Joomla! Master
Posts: 19087
Joined: Mon Aug 29, 2005 10:17 am
Location: Netherlands/ UK/ S'pore/Jakarta/ North America
Contact:

Re: No crawled pages = 0 of my web

Post by leolam » Tue Sep 06, 2016 8:53 am

Couple of things here:

Your robot.txt is incorrect. The current default robot.txt in the Joomla distribution packages is adjusted to googles new algorithm. The content is (behind the blah, blah,

Code: Select all

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/
the main change is that images and templates should be indexed.

Re your current site I do not many broken links on the site while surfing neither in Chrome nor on Firefox nor on the W3C link checker

The article you see indexed is an article that is located in category with id: 25, article id: 280 and menuitem id: 136

See if you have this still in your menu links maybe as trashed item.

Also it seems you site has been indexed while SEO unabled and with enabled. Dead links still exist with Google and you might need to reindex the site or ask Google to review this

Leo 8)
Joomla's #1 Professional Support Provider:
-> Joomla Professional Support: https://gws-desk.com -
-> Joomla Specialized Hosting Solutions: https://gws-host.com -
Member Joomla Bug Squad & Joomla CMS Release Team

tonypartridge
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 138
Joined: Sat Sep 03, 2016 7:37 am

Re: No crawled pages = 0 of my web

Post by tonypartridge » Wed Sep 07, 2016 10:45 am

MartinBalko,

Your site has a self signed SSL Certificate hence why it loads over https:// you may have not installed it personally. But it is there. The certificate belongs to: WebSupport, s.r.o. with no domain specified. So I suspect it's your host who installed it.

My solution above should sort out your https issue in the long term.

Many thanks
Tony

User avatar
leolam
Joomla! Master
Joomla! Master
Posts: 19087
Joined: Mon Aug 29, 2005 10:17 am
Location: Netherlands/ UK/ S'pore/Jakarta/ North America
Contact:

Re: No crawled pages = 0 of my web

Post by leolam » Wed Sep 07, 2016 11:07 am

I can concur that finding by Tony (nice one btw)

Leo 8)
Joomla's #1 Professional Support Provider:
-> Joomla Professional Support: https://gws-desk.com -
-> Joomla Specialized Hosting Solutions: https://gws-host.com -
Member Joomla Bug Squad & Joomla CMS Release Team

MartinBalko
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Tue Nov 24, 2009 10:24 am
Contact:

Re: No crawled pages = 0 of my web

Post by MartinBalko » Wed Sep 07, 2016 12:55 pm

thank you very much, i had write to my webhosting (you are right, it is websupport.sk) and they are searching for solution - by now they advised me to install SSL certificate. it is one of solutions, it should resolve problems with non working https: sites and it should little help of SEO effect. But, i think it should have some disavantages, i have to study something about it :)

thanks to all, For now i will be looking for solutions to remaining problems with non indexing :)

MartinBalko
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Tue Nov 24, 2009 10:24 am
Contact:

Re: No crawled pages = 0 of my web

Post by MartinBalko » Wed Oct 05, 2016 10:04 am

leolam wrote:
The article you see indexed is an article that is located in category with id: 25, article id: 280 and menuitem id: 136

See if you have this still in your menu links maybe as trashed item.
thanks, yes i found article with id: 280 in trash - and many other articles = what is best way to do with them? should i remove them definitely even from trash?
leolam wrote:
Also it seems you site has been indexed while SEO unabled and with enabled. Dead links still exist with Google and you might need to reindex the site or ask Google to review this

Leo 8)
yes, this sould be possible too, how can i fix it now? how can i reindex old dead links? i have redirect 404 sites to working sites, or homepage with joomla redirect component. is this solution of reindexing, or i have to do something else?

thanks you very much :)
Last edited by pe7er on Wed Dec 28, 2016 8:32 am, edited 1 time in total.
Reason: URL removed


Locked

Return to “Search Engine Optimization (Joomla! SEO) in Joomla! 2.5”