WARNING: Be careful with 'stats' components for Joomla!

[Rochen]

I thought I would write up a quick post from a web hosting providers prospective and try to give you an insight into a major problem we have been seeing with a particular type of Joomla component - stats. At the present time we host several thousands websites powered by Joomla including the official joomla.org site so I am basing this post on a fair bit of experience when it comes to Joomla-based hosting.

One type of component available for Joomla is a 'stats' component which allows users to gain statistical data about their site visitors directly through Joomla. A fair number of Joomla users would think "fantastic" and go ahead and install it. I mean being able to get all that statistical data about your visitors directly through Joomla sounds great.

However, what a lot of people don't seem to understand is that most of these stats components increase the load on the MySQL server by a dramatic amount above and beyond the core Joomla install. In some cases we have seen these stats components bring down entire managed servers hosting large Joomla sites. I should point out that while this particular issue is most noticeable on larger sites it affects all Joomla-powered sites - large or small. I should also point out that while this particular problem could apply to any Joomla component we have seen it most noticeable with the stats components.

I want to stress that this is not a problem with the Joomla core itself which we believe to be very stable. The problem is with the stats components that are being released through the official Extensions and other websites as add-ons for Joomla. In my opinion statistical data about site visitors should continue to be gathered and processed at the server level and not through a platform such as Joomla which adds excessive server loads to core services such as MySQL every time a visitor hits your website. This added load not only hits server performance but it also impacts the speed of your website.

We receive sales emails from people at least a couple of times per week who are looking to move their hosting because their current host tells them their Joomla installation is killing the server. It isn't in fact the Joomla installation that's at fault most of the time - it is a stats component which is installed under Joomla. It is often hard for a hosting provider who is not familiar with Joomla to tell that the component is to blame and not Joomla itself. This isn't their fault at all and it doesn't mean they are a bad host, it just means they aren't experts with Joomla.

My advice to anyone using Joomla would be to steer well clear of these 'stats' components to avoid running into problems with your web hosting provider's abuse / policy enforcement team. I would also advise anyone already running any of these stats components to drop them from your Joomla installation. You may not have encountered a problem yet because your site doesn't receive enough concurrent traffic at the moment but I can guarantee it will become a bigger issue as your site grows. I would also like to kindly ask that the developers of these components take this issue to heart before releasing any further components like this on unsuspecting victims.

Thanks for taking the time to read my post and I hope it may have proved useful to a few of you

[stingrey]

It should be noted that this is the reason we do not include a detailed statistics gathering fuction into the Joomla! core.


As noted by Chris, this kind of functionality it best dealt with at server level and also the functionality of server level statistics gathering applications is far more complete also most good web host provide such server level statistics gathering as part of their packages.

[infograf768]

I have asked all International Partners and Forum mods to forward this Warning to their respective communities.

[pe7er]

[..] The problem is with the stats components that are being released through the official Extensions and other websites as add-ons for Joomla. In my opinion statistical data about site visitors should continue to be gathered and processed at the server level and not through a platform such as Joomla which adds excessive server loads to core services such as MySQL every time a visitor hits your website. This added load not only hits server performance but it also impacts the speed of your website.
[..]
My advice to anyone using Joomla would be to steer well clear of these 'stats' components to avoid running into problems with your web hosting provider's abuse / policy enforcement team. I would also advise anyone already running any of these stats components to drop them from your Joomla installation. You may not have encountered a problem yet because your site doesn't receive enough concurrent traffic at the moment but I can guarantee it will become a bigger issue as your site grows. I would also like to kindly ask that the developers of these components take this issue to heart before releasing any further components like this on unsuspecting victims.

When you state that stats component have a major negative impact on the speed of your webserver, you are referring to all kinds of Joomla stats components?
And not just one in particular from http://extensions.joomla.org/component/ ... Itemid,35/ ?

[brian]

The problem is that all the J stats components work by grabbing the data and then storing them in a mysql database. If your site is even moderately busy this will result on lots of mysql entries and a rapidy increasing database.

Traditionaly webstats are gathered at the server level by apache in flat files. A stats analysis program then reads those files and presents you with the results ONLY when you access it. Which for most people will only be on an occasional basis.

Brian

[leolam]

@ Rey;

would it be possible to place a sort of "warning" button on the extention site pointing to this excellent information provided by "Rochen" or get a pop-up with the warning on some of the components etc with similar information? This might avoid many problems and reduce issues at the forum....Should not be too much of a task...?

2ct's

Leo

[dknight]

Leo,

We have already updated the description to inform users about this potential performance issue:

http://extensions.joomla.org/component/ ... Itemid,35/

[leolam]

Leo,

We have already updated the description to inform users about this potential performance issue:

http://extensions.joomla.org/component/ ... Itemid,35/

Good call Lee! Tnx

[keliix06]

I couldn't agree with this more. We're also hosting thousands of Joomla and Mambo installs, and it is the single biggest problem we see.

[focalguy]

Thanks for the information. My site is quite small so I haven't noticed anything yet. I guess I have been using both the stats gathered in cPanel scripts and the Joomla Components because I feel the components display the information I am most interested in in a better format. The information is there with the other scripts, but I haven't found a way to get at it as quickly or easily as the components. If this were not the case, I would drop the component immediately.

[ilox]

Can I please have a little bit of clarification? In the Admin backend, Site --> Global Configuration --Statistics, there are options to collect stats.

• Statistics:    No  Yes  Enable/disable collection of site statistics
• Log Content Hits by Date: No Yes Warning! Large amounts of data will be collected
• Log Search Strings: No Yes

Is this also likely to be a contributor to the problem?
If so then perhaps from 1.1 onwards we could have more than just a warning symbol appear next to the stats option warning people of likely consequences for activating that option? Yes, it does say, "Warning! Large amounts of data will be collected" but this doesn't really explain just what that might mean to efficiency or server work.
A newcomer (like I was)  is unlikely to be aware of the hit that such an option might bring to the site and the servers. Maybe this page needs an introductory text warning placed above the options so that users are very clearly aware of the consequences?
Like.
Collecting statistics through the use of these options is likely to cause extra workload on the servers. Some service providers may take regulatory action in the event of such extra workload. It is suggested that these options only be selected if your server provider is unable to provide you with site statistics that are usually collected by the server software.

[Cafemonkey]

Could we blend the solutions to create a better one?

I mean, the non-Joomla stats read from a flat file, which means no MySQL resources chewed up.
The Joomla stats provide better information for the Webmaster.

Can we put them together and let Mambo use flat files so we get better performance and better information?

[stingrey]

Can I please have a little bit of clarification? In the Admin backend, Site --> Global Configuration --Statistics, there are options to collect stats.

The short answer is yes this also can affect the performance of a site and is why it is `off` be default.

[Mr. Magic City]

Since installing stat components is not advised what would be a suitable solution (or remedy) for a webmaster who needs this function? Also, I'm interested in knowing how reporting components (e.g. JamboReport) affect MYSQL and server loads.

[trompete]

On behalf of BSQ Sitestats, I'd like to defend my architecture a bit. I created the component because the component I was using at the time, TFSForMambo, was generating 20 database queries per page. BSQ Sitestats does a single INSERT for a normal page load and an extra SELECT/INSERT in the rare case that a hit comes from a search engine. This performs very well. I'm currently showing 45 other queries per page on my Joomla 1.0.8 installation.

The other reason I use BSQ Sitestats instead of the native stats is that the native stats cause an update query to occur the mos_content table, invalidating Mysql's query cache for that record. The mysql query cache is wonderful, but it only works if you aren't updating records each time you view them.

I'd love to hear any feedback on performance you guys have as Joomla administrators.

I'm sorry to see that all of the stats components are grouped together as one malicious entity.

Brent

[eyezberg]

I am starting to think this might be the reason my previous host kicked me out..
Would me nice to have more detailed exaplantions and tests about Core stats and each (of the not so many) components.
I agree with trompete that it is not fair to just issue a general statement about stats collection without differentiating how these are collected and or stored etc.. According to Brians post, it looks like using flat files would solve most issues? What about that, is that correct, or are there other solutions also?
Good to warn, better to continue this discussion with the dev's and come up with (a) solution(s).

[masyomo]

On behalf of BSQ Sitestats, I'd like to defend my architecture a bit.

I'd love to hear any feedback on performance you guys have as Joomla administrators.

I'm sorry to see that all of the stats components are grouped together as one malicious entity.

Brent

We removed BSQ stats for the same reason - our database went from 2mb to 30mb and its only a small userbase. Most of the stats are available through cpanel anyway although it was interesting to see user reports but not crucial.

The other component beefing up our database is remository which now stores files in mysql. On their site they say that databases will run with no problem up to 4GB so is it the load on the database thats the problem rather than the storage? Our main worry was backing up the database but that was before I'd thought we could just exclude the big tables from the backup & still have all the crucial info 

[trompete]

We removed BSQ stats for the same reason - our database went from 2mb to 30mb

What about server load? The above statement has to do with disk space. Server load is not the same as disk space.


Disk space can be an issue, and is a valid reason not to use a stats product. But don't confuse that with peformance. For example, the "stuff" I work on at my day job generates 50 GB of recorded data a day, but it does 150k transactions per second. This is great performance, even though it eats up a lot of disk.

[masyomo]

yeah thats what I meant - I removed it because doing a backup was taking forever but hadn't thought that I could just leave out the heavy tables - and then when looking into Remository I saw the thing about the database still working fine up to 4GB and so thought maybe it wasn't such a problem afterall.

I didn't know anything about the server load so can't say unfortunately but now I'm learning!!

[CCI]

The little statistics module that comes with the Joomla install which show count of members, server OS, number news items, etc., isn't a culprit, is it?

[Websmurf]

The little statistics module that comes with the Joomla install which show count of members, server OS, number news items, etc., isn't a culprit, is it?

No that information is allready registered anyway

[Michiel_1981]

This warning might as well go out for most sef components.

regards,
Michiel

[notexa]

This warning might as well go out for most sef components.

Could you ellaborate on this one ?
How do eg Opensef, advanced Sef, ... have a negative impact on server load ?

[Websmurf]

Well, mostly due to the amount of queries that it will use to build op the sef urls. Most sef components will querie the database a few times for each link on the page. These aren't very heavy queries (simple selects, so cached in Mysql 4.1 and above) but they will be there.
If you're running a high traffic site, this will play a role.

[kenmcd]

This warning might as well go out for most sef components.

Could you ellaborate on this one ?
How do eg Opensef, advanced Sef, ... have a negative impact on server load ?

Performance Comparison of MySQL Queries by SEF Components
http://www.open-sef.org/forum/opensef-d ... rison.html

[Michiel_1981]

Hi kenmcd,

I was just searching for the that post.
If you look at the data you can concluded this, this avarage queries extra.

• opensef rc4 -  181
• opensef rc5 -  18
• sef advance -  50
• 404sef        -  126

Compared those to a 1 update query bsq sitestats makes on a shared host or high volume site. That update will most likely be faster then any of those sef components.
opensef might stand a change (rc5) because it only ads 18 queries, but all those others.
Now picture a shared host running a lot of joomla sites with those sef components. What will happen to your cache mechanisme? Does it matter, as 50+ individual select statements are slower then 1 update statement, more so if the database isn't on the same server!

BSQ sitestats has a problems, I admit,  it gets fat very fast. Brent is working on this. The next release will group more urls together and has a better filter so post queries won't be counted as unique urls. The release after that the database structure will be changed and the hits be stored more efficiently.

but speed/serverload it will beat the sef components and that's were this topic was about.
regards,
Michiel

[edit]
typo

[kenmcd]

All of the SEF queries can be cached with sufficient memory.
Inserts are another story.
I have some experience tuning high performance ad servers - lots of inserts.
But again this can be tuned for high performance.

Unfortunately most shared MySQL servers are not tuned well, and on top of that they are usually overloaded with too many users. They usually take an average machine with average memory and then don't tune it properly for what they do have. Add way too many users = sloooowwwww. The silly solution - add query limits.

Ironically the users who need a Joomla Stats component the most - those without log file access - are the ones most likely to be affected by the query load/limits.


Don'cha know it's always sumthin'
                              - Rosanne Rosannadana

[trompete]

Ironically the users who need a Joomla Stats component the most - those without log file access - are the ones most likely to be affected by the query load/limits.

Agree. On the querying side, BSQ Sitestats uses the Cache lite libraries to minimize database IO by only doing it every 30 seconds or so. I've found that file IO works a lot better than database IO for a couple reasons:
1. Most people do database IO, so that cache fills up quickly and is invalidated in a circle of thrashing.
2. Database IO is socket-based. Yeah, it is just a handle, but TCP has its slow transmission ramp up...etc. Mysql really needs to make a working shared memory interface between the client libraries and their server for maximum throughput. At that poing, mysql would be just as good as a file system.
3. Reading of files is implicitly cached thanks to our friend, Mr. virtual memory.
4. Profit???

[Michiel_1981]

Unfortunately most shared MySQL servers are not tuned well, and on top of that they are usually overloaded with too many users. They usually take an average machine with average memory and then don't tune it properly for what they do have. Add way too many users = sloooowwwww. The silly solution - add query limits.

Ironically the users who need a Joomla Stats component the most - those without log file access - are the ones most likely to be affected by the query load/limits.

What exactly is your point, that they both are affected by misconfigured Shared host Setups or only the stats?
[little off topic maybe]
I'm willing to run a test on my local server to compare speed/load between these components and a clean joomla install, if you are so kind to provide some details what should be fare. How much memory  should mysql get to cache per domain and how much hits/connections would be fare in this case. Just to see what the affects are.
Short, How many domains share how much of mysql cache on how many concurrent connections does it have. So we can scale down.

regards,
Michiel

[kenmcd]

What exactly is your point, that they both are affected by misconfigured Shared host Setups or only the stats?
[little off topic maybe]
I'm willing to run a test on my local server to compare speed/load between these components and a clean joomla install, if you are so kind to provide some details what should be fare. How much memory  should mysql get to cache per domain and how much hits/connections would be fare in this case. Just to see what the affects are.
Short, How many domains share how much of mysql cache on how many concurrent connections does it have. So we can scale down.

regards,
Michiel

HUH?
Are you upset about something?
If that is so, I do not have any idea of what or why.

What exactly is your point?