Google is crawling a lot of non-sef URLs (more than I have SEF-URLs)

Discuss Search Engine Optimization in relation to Joomla! 3.x. This forum will also have discussions on SEF/SEO Joomla! 3.x extensions.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Windows Defender SmartScreen Issues <-- please read this if using Windows 10.
Post Reply
Thomas_VDB
Joomla! Apprentice
Joomla! Apprentice
Posts: 5
Joined: Wed Oct 14, 2020 6:35 am

Google is crawling a lot of non-sef URLs (more than I have SEF-URLs)

Post by Thomas_VDB » Wed Oct 14, 2020 6:43 am

Hi,

I've tried to optimise our website (www.ninix-tech.com) for SEO as much as possible.
One of the things remaining is that Google search console is listing a lot (hundreds) of non-sef URLs as blocked.
In my sitemap I've only listed our SEF-urls, which are only about 60 pages. These are all the pages that exist on our website.

I've configured SEF-urls in Joomla as they should be configured .
(https://docs.joomla.org/Enabling_Search ... (SEF)_URLs)

However I've got a feeling that there are many non-sef URL's possible to get to an article, and that Google somehow finds them all.

How do I completely stop Google to crawl any page but the ones in my sitemap?

waarnemer
Joomla! Hero
Joomla! Hero
Posts: 2870
Joined: Sun May 04, 2008 12:37 pm

Re: Google is crawling a lot of non-sef URLs (more than I have SEF-URLs)

Post by waarnemer » Thu Oct 15, 2020 4:10 pm

That indeed is true.. the ones with ID and such are still resolvable..
How google got them? Perhaps from the past or an extension that did/does not repsect the ROUTE in the code.

One I added to get rid of indexed "odd urls" is by adding this to the bottom of robots.txt

Code: Select all

Disallow: /*?

Allow: /*/component/osmap/*?
The last one to have it see the sitemaps from osmap

User avatar
Webdongle
Joomla! Master
Joomla! Master
Posts: 39210
Joined: Sat Apr 05, 2008 9:58 pm

Re: Google is crawling a lot of non-sef URLs (more than I have SEF-URLs)

Post by Webdongle » Thu Oct 15, 2020 6:35 pm

Create a site map and add it to google webmaster tools.
http://www.weblinksonline.co.uk/
https://www.weblinksonline.co.uk/updating-joomla.html
"The definition of insanity is doing the same thing over and over again, but expecting different results": Albert Einstein.

waarnemer
Joomla! Hero
Joomla! Hero
Posts: 2870
Joined: Sun May 04, 2008 12:37 pm

Re: Google is crawling a lot of non-sef URLs (more than I have SEF-URLs)

Post by waarnemer » Thu Oct 15, 2020 6:51 pm

@webdongle, OP has that in place already

User avatar
Per Yngve Berg
Joomla! Master
Joomla! Master
Posts: 27296
Joined: Mon Oct 27, 2008 9:27 pm
Location: Romerike, Norway

Re: Google is crawling a lot of non-sef URLs (more than I have SEF-URLs)

Post by Per Yngve Berg » Thu Oct 15, 2020 7:07 pm

About the Contact Form: Disable the "Send a copy to yourself" feature. It may be abused by spammers.

Thomas_VDB
Joomla! Apprentice
Joomla! Apprentice
Posts: 5
Joined: Wed Oct 14, 2020 6:35 am

Re: Google is crawling a lot of non-sef URLs (more than I have SEF-URLs)

Post by Thomas_VDB » Wed Oct 21, 2020 2:10 pm

Thx for the suggestions.

However I still did not find the root cause of all the non-sef urls that have been crawled (and blocked) by Google.

waarnemer
Joomla! Hero
Joomla! Hero
Posts: 2870
Joined: Sun May 04, 2008 12:37 pm

Re: Google is crawling a lot of non-sef URLs (more than I have SEF-URLs)

Post by waarnemer » Wed Oct 21, 2020 6:35 pm

root cause is hard to tell.. it can all be crawled in a past when proper tech was not in place yet.
it is better to concentrate on issues in current.


Post Reply

Return to “Search Engine Optimization (Joomla! SEO) in Joomla! 3.x”