How to allow Google to index protected content

Discuss Search Engine Optimization in relation to Joomla! 3.x. This forum will also have discussions on SEF/SEO Joomla! 3.x extensions.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Post Reply
maestroc
Joomla! Explorer
Joomla! Explorer
Posts: 370
Joined: Sun Dec 04, 2005 7:27 pm
Contact:

How to allow Google to index protected content

Post by maestroc » Fri Dec 07, 2018 12:37 pm

I have a new information/news site that is behind a paywall. How do we allow Google to index the content on it when everything is locked down by RSMembership? At this point we have not put in any --ReadMore-- tags and really didn't think we could do it that way as the information in the articles is mostly images, audio files, or video files (very little text).

I have installed OSMap and have it set to list all articles (including the protected ones) in the XML sitemap so that Google can at least index the titles. However I have the Add Images to sitemap option turned off. We don't want google or any other search engine to index the images, videos in a way that might make them visible through search results (like google's image search, etc.) as they are the primary thing that makes our site stand out from other similar sites. Is that all that I can do or am I missing something?

waarnemer
Joomla! Hero
Joomla! Hero
Posts: 2115
Joined: Sun May 04, 2008 12:37 pm

Re: How to allow Google to index protected content

Post by waarnemer » Fri Dec 07, 2018 4:56 pm

Still, the readmore method is the one you need.
Since the images and videos are only shown when logged in, there will be no URL link available on 'public'.
Your intro text and intro image will be available for indexing. All below Readmore is not.

I know the images folder is publically accessible... you can try the following:
Try this. Create an .htaccess file in the images folder and add this:

Code: Select all

order deny,allow
deny from all
allow from (subdomain.)domain.com
Test if you can still see the images from within the site. Test if you can download an image when you use the direct URL.

This is a trick I used on simple download managers in the past to make restricted downloads. But it may also be useful on images.

or...

For indexing bots you can read the following;
https://en.wikipedia.org/wiki/Robots_exclusion_standard
https://support.google.com/webmasters/a ... ic=6061961

User avatar
lefteriskavadas
Joomla! Intern
Joomla! Intern
Posts: 83
Joined: Fri Feb 05, 2016 6:10 pm
Location: Athens, Greece
Contact:

Re: How to allow Google to index protected content

Post by lefteriskavadas » Fri Dec 07, 2018 5:06 pm

Hi,

You can't have Google index protected content. The only way to do this is detect the Google bot and let it pass to access your content. But, keep in mind that this means that someone who knows a couple of things can use this hack to access your content. It's risky.

Giving just the links to Google through sitemaps will fail because when Google will try to fetch that link it will get nothing. Giving just a link to Google is not enough.

Regards

shivani_01
Joomla! Fledgling
Joomla! Fledgling
Posts: 1
Joined: Thu Jan 17, 2019 4:47 am

Re: How to allow Google to index protected content

Post by shivani_01 » Thu Jan 17, 2019 4:50 am

If you have content that is protected by an index you would still want it to be looked at by Google. However, this is something that not everyone knows how to do and you would need to think about and know what you should do. There are a few tools that you can use that would allow Google to see all of the content as it should be seen without having to worry about what user they are showing it to. First Free Click is a good tool that allows Google to see everything that you are posting without issues.

waarnemer
Joomla! Hero
Joomla! Hero
Posts: 2115
Joined: Sun May 04, 2008 12:37 pm

Re: How to allow Google to index protected content

Post by waarnemer » Sun Jan 20, 2019 11:18 am

https://www.theguardian.com/technology/ ... ree-policy

So that is ruled out.

Again, introduce in the articles above the readmore, then have all protected content below.
There are a few technologies that allow you to control..

Of course the readmore, but also
Intro Image, Full article Image (protected? use the .htaccess I described above), Your meta description and tags... so much you can do....

User avatar
Webdongle
Joomla! Master
Joomla! Master
Posts: 36439
Joined: Sat Apr 05, 2008 9:58 pm

Re: How to allow Google to index protected content

Post by Webdongle » Sun Jan 20, 2019 3:54 pm

lefteriskavadas wrote:
Fri Dec 07, 2018 5:06 pm
...
You can't have Google index protected content. ...
Actually you can but there would be no point as the visitor would not be able to see it on the site. Best option is the Readmore with 'Show Unauthorised Links' 'Yes' will allow intro text to bee seen. Set it in the menu item(s) not Article Options otherwise all Articles that are restricted (but have no readmore) will show.
http://www.weblinksonline.co.uk/
https://www.weblinksonline.co.uk/updating-joomla.html
"The definition of insanity is doing the same thing over and over again, but expecting different results": Albert Einstein

User avatar
lefteriskavadas
Joomla! Intern
Joomla! Intern
Posts: 83
Joined: Fri Feb 05, 2016 6:10 pm
Location: Athens, Greece
Contact:

Re: How to allow Google to index protected content

Post by lefteriskavadas » Fri Feb 01, 2019 3:42 pm

Webdongle wrote:
Sun Jan 20, 2019 3:54 pm
lefteriskavadas wrote:
Fri Dec 07, 2018 5:06 pm
...
You can't have Google index protected content. ...
Actually you can but there would be no point as the visitor would not be able to see it on the site. Best option is the Readmore with 'Show Unauthorised Links' 'Yes' will allow intro text to bee seen. Set it in the menu item(s) not Article Options otherwise all Articles that are restricted (but have no readmore) will show.
No you can't. Unless you expect Google bot to register to your site and login. Google Bot will see what your site guest users see. Just the intro text not the protected content.

User avatar
Webdongle
Joomla! Master
Joomla! Master
Posts: 36439
Joined: Sat Apr 05, 2008 9:58 pm

Re: How to allow Google to index protected content

Post by Webdongle » Fri Feb 01, 2019 4:36 pm

lefteriskavadas wrote:
Fri Feb 01, 2019 3:42 pm
...
No you can't. Unless you expect Google bot to register to your site and login. Google Bot will see what your site guest users see. Just the intro text not the protected content.
You can because you can create a plugin that allows google to access Registered content. I was given the code once for a module that I wrote. That was a few years ago and I did not keep the code. Basically the plugin allows Googlebot to be treated the same as a 'Registered user' without creating a user for it.
http://www.weblinksonline.co.uk/
https://www.weblinksonline.co.uk/updating-joomla.html
"The definition of insanity is doing the same thing over and over again, but expecting different results": Albert Einstein

User avatar
mandville
Joomla! Master
Joomla! Master
Posts: 14763
Joined: Mon Mar 20, 2006 1:56 am
Location: The Girly Side of Joomla in Sussex

Re: How to allow Google to index protected content

Post by mandville » Fri Feb 01, 2019 5:39 pm

not sure where in the search you would find the latest answer but .,.,
https://extensions.joomla.org/extension ... rfilters=1
HU2HY- Poor questions = Poor answer
Un requested Help PM's will be reported, added to the foe list and possibly just deleted
{VEL Team Leader}{TM Auditor }{ Showcase & Security forums Moderator}


Post Reply

Return to “Search Engine Optimization (Joomla! SEO) in Joomla! 3.x”