Web Scraper for Joomla 3.x

This forum is for general questions about extensions for Joomla! 3.x.

Moderators: pe7er, General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Post Reply
tpaljr63
Joomla! Apprentice
Joomla! Apprentice
Posts: 11
Joined: Mon Sep 19, 2016 9:34 pm

Web Scraper for Joomla 3.x

Post by tpaljr63 » Mon Jan 30, 2017 3:30 pm

Hello All,

I am looking for a extension/module/plugin to be able to scrape real time information from one website to my joomla site.


I have looked (googled) - looked in the Joomla Extensions, any ideas of what to use to grab the information one website and post it on my site.

Thanks,

Tom

User avatar
sozzled
Joomla! Champion
Joomla! Champion
Posts: 5888
Joined: Sun Jul 05, 2009 3:30 am
Location: Canberra, Australia
Contact:

Re: Web Scraper for Joomla 3.x

Post by sozzled » Mon Jan 30, 2017 4:53 pm

Obtaining original web content from a website (using "scraping" techniques), without the owner's permission, is illegal. If the content from the "other" site is owned by you then there are many techniques you can use to display that content. If, on the other hand (and, as I suspect) this "other site's" content is not owned by you then you must first seek the owner's permission before you can use it on your own site.

If the original owner's site provides some kind of API (or RSS feed, perhaps) then this is something you take up with the owner of the original content and clarify any restrictions that the original owner may place on that site's content copyright.

No, I am not aware of any extensions that permit content scraping in Joomla. Have a nice day. 8)
https://www.kuneze.com/blog
Former member of Kunena project team
If you think I’m wrong then say “I think you're wrong.” If you say “You’re wrong!”, how do you know?

tpaljr63
Joomla! Apprentice
Joomla! Apprentice
Posts: 11
Joined: Mon Sep 19, 2016 9:34 pm

Re: Web Scraper for Joomla 3.x

Post by tpaljr63 » Mon Jan 30, 2017 5:37 pm

Site Owner does allow it for the purposes that we are going to use it for since it deals with our users.

I already checked with the Site Owner for the permission and no api or rss feed..

and for you to assume that someone is doing it illegally is a crock.

Appreciate your info ..not your attitude.

User avatar
sozzled
Joomla! Champion
Joomla! Champion
Posts: 5888
Joined: Sun Jul 05, 2009 3:30 am
Location: Canberra, Australia
Contact:

Re: Web Scraper for Joomla 3.x

Post by sozzled » Mon Jan 30, 2017 5:50 pm

As I wrote earlier, if a website owner explicitly allows people to copy and republish their original content and the site owner provides a mechnism for other people to copy that content, there shouldn't be a problem. That's something you need to discuss with the original copyright owner.

If, however, the material is copyrighted that's an entirely different question. We only have your word that the original copyright owner allows you (or anyone) to copy and republish/repackage their content.

I repeat: I am not aware of any extensions that facilitate one Joomla website to automatically "scrape" contents from another website that's not owned by you.
https://www.kuneze.com/blog
Former member of Kunena project team
If you think I’m wrong then say “I think you're wrong.” If you say “You’re wrong!”, how do you know?

tpaljr63
Joomla! Apprentice
Joomla! Apprentice
Posts: 11
Joined: Mon Sep 19, 2016 9:34 pm

Re: Web Scraper for Joomla 3.x

Post by tpaljr63 » Mon Jan 30, 2017 10:50 pm

I like to apologize ...I came from doing wordpress and moved unto Joomla. I shouldn't have used the word scraping.

What i am trying to do is basically wrap a paticular section of a webpage that the public has access to and post it in iframe on the joomla site for our league players.

Our league stats are hosted on another site..instead of having my users scroll through the frame, i was looking to just have a paticular section in the frame for my users to view.

User avatar
dhuelsmann
Joomla! Master
Joomla! Master
Posts: 19646
Joined: Sun Oct 02, 2005 12:50 am
Location: Omaha, NE
Contact:

Re: Web Scraper for Joomla 3.x

Post by dhuelsmann » Mon Jan 30, 2017 11:01 pm

If it is just an iframe view you want of a particular page on another site, that capability already exists in the menu options of Joomla.
Regards, Dave
Past Treasurer Open Source Matters, Inc.
Past Global Moderator
http://www.kiwaniswest.org

tpaljr63
Joomla! Apprentice
Joomla! Apprentice
Posts: 11
Joined: Mon Sep 19, 2016 9:34 pm

Re: Web Scraper for Joomla 3.x

Post by tpaljr63 » Tue Jan 31, 2017 1:27 pm

I saw that yesterday - just trying to get paticular content off the page and not the whole page.

I googled it and found some examples - appreciate all the help.

User avatar
sozzled
Joomla! Champion
Joomla! Champion
Posts: 5888
Joined: Sun Jul 05, 2009 3:30 am
Location: Canberra, Australia
Contact:

Re: Web Scraper for Joomla 3.x

Post by sozzled » Tue Jan 31, 2017 7:43 pm

@dhuelsman: remembering also that the original content provider may employ anti-clickjacking methods by using a .htaccess rule such as this:

Code: Select all

Header append X-FRAME-OPTIONS "SAMEORIGIN"
If someone did that, it would stop IFRAME/wrappers dead-in-the-water.
https://www.kuneze.com/blog
Former member of Kunena project team
If you think I’m wrong then say “I think you're wrong.” If you say “You’re wrong!”, how do you know?

User avatar
uaintgotthisid
Joomla! Explorer
Joomla! Explorer
Posts: 351
Joined: Wed Sep 10, 2008 6:05 pm
Location: Essex, England, United Kingdom
Contact:

Re: Web Scraper for Joomla 3.x

Post by uaintgotthisid » Tue Feb 13, 2018 6:14 pm

The trouble with iFrames is that they don't automatically resize the height. Which then means it's hard to make them responsive. It won't take long to try it anyway I have seen the .htaccess block in the past but it only takes seconds to find out if it works or not.

I'm prepared to take your word for it that you are allowed to use the other sites data.
Just another lonely website designer trying to make his way.
https://www.squareballoon.co.uk
JOIN US at Joomla! User Group London or on G+
https://www.joomlalondon.co.uk


Post Reply

Return to “Extensions for Joomla! 3.x”