How to allow Google to index protected content

Discuss Search Engine Optimization in relation to Joomla! 3.x. This forum will also have discussions on SEF/SEO Joomla! 3.x extensions.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Post Reply
maestroc
Joomla! Explorer
Joomla! Explorer
Posts: 366
Joined: Sun Dec 04, 2005 7:27 pm
Contact:

How to allow Google to index protected content

Post by maestroc » Fri Dec 07, 2018 12:37 pm

I have a new information/news site that is behind a paywall. How do we allow Google to index the content on it when everything is locked down by RSMembership? At this point we have not put in any --ReadMore-- tags and really didn't think we could do it that way as the information in the articles is mostly images, audio files, or video files (very little text).

I have installed OSMap and have it set to list all articles (including the protected ones) in the XML sitemap so that Google can at least index the titles. However I have the Add Images to sitemap option turned off. We don't want google or any other search engine to index the images, videos in a way that might make them visible through search results (like google's image search, etc.) as they are the primary thing that makes our site stand out from other similar sites. Is that all that I can do or am I missing something?

waarnemer
Joomla! Hero
Joomla! Hero
Posts: 2033
Joined: Sun May 04, 2008 12:37 pm

Re: How to allow Google to index protected content

Post by waarnemer » Fri Dec 07, 2018 4:56 pm

Still, the readmore method is the one you need.
Since the images and videos are only shown when logged in, there will be no URL link available on 'public'.
Your intro text and intro image will be available for indexing. All below Readmore is not.

I know the images folder is publically accessible... you can try the following:
Try this. Create an .htaccess file in the images folder and add this:

Code: Select all

order deny,allow
deny from all
allow from (subdomain.)domain.com
Test if you can still see the images from within the site. Test if you can download an image when you use the direct URL.

This is a trick I used on simple download managers in the past to make restricted downloads. But it may also be useful on images.

or...

For indexing bots you can read the following;
https://en.wikipedia.org/wiki/Robots_exclusion_standard
https://support.google.com/webmasters/a ... ic=6061961

User avatar
lefteriskavadas
Joomla! Intern
Joomla! Intern
Posts: 82
Joined: Fri Feb 05, 2016 6:10 pm
Location: Athens, Greece
Contact:

Re: How to allow Google to index protected content

Post by lefteriskavadas » Fri Dec 07, 2018 5:06 pm

Hi,

You can't have Google index protected content. The only way to do this is detect the Google bot and let it pass to access your content. But, keep in mind that this means that someone who knows a couple of things can use this hack to access your content. It's risky.

Giving just the links to Google through sitemaps will fail because when Google will try to fetch that link it will get nothing. Giving just a link to Google is not enough.

Regards


Post Reply

Return to “Search Engine Optimization (Joomla! SEO) in Joomla! 3.x”