Page 1 of 1

Links Excluded By Google

Posted: Sat Oct 13, 2018 10:32 pm
by Newman123
Hello, I was wondering if anyone can help me understand how to unexclude links, a lot of my tags seem to be excluded from indexing, I have it set to index and follow.. Thank you..

Re: Links Excluded By Google

Posted: Sun Oct 14, 2018 12:47 am
by Per Yngve Berg
Mod. Note: Relocated the topic to the SEO forum.

Edit robots.txt file

Re: Links Excluded By Google

Posted: Wed Oct 17, 2018 2:28 am
by delacroix1505
You can look at here: https://docs.joomla.org/Robots.txt_file

Open the robots.txt file (must be in the root folder). Then edit -> type: Allow: /tags/ (It really depend on your URL structure - for example: if I want to indexing my tag: I should type: Allow: /blogs/tags/)

Re: Links Excluded By Google

Posted: Fri Oct 19, 2018 9:49 pm
by Newman123
thanks everyone, just added that one line, let's see what happens..

Re: Links Excluded By Google

Posted: Wed Oct 24, 2018 5:20 pm
by occultfish
You can also use the joomla option for meta data by creating a menu item for the tags. In the meta robots box, just type in "index,follow".

Thanks

Re: Links Excluded By Google

Posted: Wed Oct 24, 2018 7:21 pm
by sozzled
delacroix1505 wrote:
Wed Oct 17, 2018 2:28 am
Open the robots.txt file (must be in the root folder). Then edit -> type: Allow: /tags/ (It really depend on your URL structure - for example: if I want to indexing my tag: I should type: Allow: /blogs/tags/)
Hmmm ...

Everything I've read about the Robots Exclusion Protocol suggests that there's no use for writing "Allow". Four other important points we should also make are:
  1. The /robots.txt is a de-facto standard, and is not owned by any standards body.
  2. The /robots.txt standard is not actively developed.
  3. robots can ignore your /robots.txt especially malware robots that scan the web for security vulnerabilities and email address harvesters used by spammers will pay no attention.
  4. The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.
The source for the above list is http://www.robotstxt.org/robotstxt.html.