Joomla! Discussion Forums

Hi

I am not a programmer, but I think I grasp the issue being discussed here.  If the argument is that stats are collected by the server anyway AND can be processed via an external program WITHOUT unnecessarily adding to the Joomla database burden, then the following issues need to be addressed or documented (if someone already knows the answers):

1. How to work with server stats that simply record all visits as being to index.php (i.e. ignoring the parameters in the URL).  For example, I use AWStat, and I can't see which pages are being visited.  Even if I could see the parameters, I would still not know the title of the page (note: I know about SEF options, but can't find an open source option that works with all the CMT's I use).

[UPDATE] A partial solution appears to be to configure the server stats analyzer to include the URL paramaters as part of the url, thereby tracking index.php?option=foo1 as a different url to index.php?option=foo2.  For AWStat, see http://awstats.sourceforge.net/docs/aws ... LWithQuery (haven't tried it yet).

2. How to intergrate the stats obtained at a server level with the stats pertaining to each registered user's use of the site.  In fact, the integration of any statistics gathered by Joomla's default stats module and those only gathered by the server stats.

Without a solution to both of these problems, there will always be a demand for a Joomla component that gathers all stats.

Any ideas?

Regards
Brendon

No No, not upset about anything, sorry if I made it sounds like that.

Just a little curious why All the stats components are being hammerd and the sef components not, well those can make a server go lame as hard as those stats components.
Both component groups have better and worse in it's catogries.
opensef rc5 <-> sef404
bsq_sitestat <-> The old JoomlaStats (I haven't checked the latest release, so if it's 1 fast query now don't shoot me)

The problem here is I encounter more sef404 then opensef usage. So awarness and user education is in order but don't limit it to category and/or make the complete category "the black sheep".

I try to keep the "weight" of my sites as low as possible but I use bsq_sitestats on most of them to get a better view on which pages are viewed more often as brendonhatcher states, further most host will give you form of stats viewing but you can get the raw data and somethimes this is a must. bsq_sitestats just fills the gap "some" hosting companies aren't filling.

kind regards,
Michiel

No one has "hammered" stats components and "not" hammered SEF components.
I have no agenda or opinions regarding the stats components.
Where you are getting this is beyond my apparently limited capabilities.

Simple observations of the MySQL realities.

Take a walk, have a beer or something.

First of all, i'm not attacking you or anybody. Just the decision to put a big alarm on the sitestats component categorie with out looking deeply into the matter or make a distinction. I just dragged sef components in as example of components that also add a big mysql extra to joomla.
Just a simple MySQL reality.

But i'll walk now and have that beer.
Regards,
Michiel

Please relax all.

Basically the point is that some components can cause an extra load in Mysql queries, something most people don't take into account when installing them. For that matter it doesn't really matter if it is a stats component, SEF, or something else entirely. An improperly configured Mysql database will be hit more than a properly configured one, just as shared hosting can be overpopulated.

Hi Tonie,

See my first post in this Topic.
I agree to make users aware but don't make such a drastic step with the "sitestats" category and leave other category with the same kind of issue completely untouched.

There is big differnce between those components, old and new(both releases and components) why should a new and improved component suffer because of the old reputation of the components there.

for example opensef newest release I try to switch as many clients or people who ask advice to switch to that as a lot of users still use sef404. Same counts for stats components.

kind regards,
Michiel

If that is indeed the point, then state it that way. Stating it as "stats components" means all stats solutions, even ones that strive for high performance. You can't make an outrageous claim like that and not expect a response. I'd be happy if you changed the topic to reflect what you stated above. Both michiel and I can run performance numbers similar to what the guys over at opensef did for RC5 vs RC4, advanced SEF, and Joomla SEF. He has dedicated hosting, and I have shared.

Let's make something productive out this this thread so we won't discuss it again.

Regards,
Brent

I would propose adding the informations to the performance enhancenment thread in this forum (The other sticky) That way we would have all neatly tucked away at one place and only one URL to referr newbies to. I'm thinking about redoing that one anyway with correct informations on folder permissions. What do you guys think?

Sure. I'll post the info here as a base reference and you can copy it.

I profiled BSQ Sitestats and JoomlaStats on my local site with Zend studio. I purposely disabled query caching to simulate a shared-hosting environment where the cache is constantly being thrashed. My setup is Apache 1.3.5, mysql 4.0.20, and PHP 4.3.x.

BSQ Sitestats
Queries per page load: 23
BSQ Queries: 1



JoomlaStats
Queries per page load: 29
JoomlaStats Queries: 7



More to come when I can figure out where my ab.exe is so I can do some real multithreaded profiling.

So, Brent what you are saying is that BSQ Sitestats is adding ONE query per page load?
In the major scheme of things, that is nothing.

If this is putting someones MySQL server over the edge, there is definitely some server capacity issues.

Yes. That's exactly what I'm saying.

Yes, I'm not an expert on server performance, but one query per page seems rather small to me. Is there some other explanation of why this would hurt the server more than any other component that runs queries every page load?

It's a simple insert shouldn't hurt to much, difference with other components that might perform selects is that mysql caches those queries most of the time (although will a large number of requests at a time, the effect will be unnoticable)

There's always at least one INSERT or UPDATE for stats components. That's the nature of the game.

I think the point is that BSQ Sitestats isn't a big resource hogger as opposed to other stat components (or even sef components).  That's what the warning is for.
But I guess trompete has proven here that BSQ Sitestats is well written concerning query handling. 

Please read again what cafemonkey wrote, because I totally agree and no one is responding:



seems like a great solution to me! everybody happpy.

Does this issue affect the administrator?
It doesn't make much sense for me but I had been having very slow performance in the admin side and after reading this post I disabled the stats and now my site admin is running like a charm!
There is a possibility that it's a coincidence sinse I just wrote to my hosting reporting bad database performance.
What do you think?

:( Ok guys,

This is my sad story. My little site has been suspended twice this week due to overload. At first I wasn't sure what the problem, till 2nd time they showed me this:


  I love BSQ so much, but I have to give up this time. I'm thinking to switch host also. What do you guy think? because of purely BSQ stats or my host or both?

I fixed this query about a month ago in SVN. It is, in fact, a slow query on mysql 4.0.x. I'm just finishing up BSQ Sitestats 2.0.0, which has fixes for all of the slow queries. For now, turn off the mod_bsq_sssummary module on your front page or set the caching time to more than 60 seconds.

How many hits in your database?

Hey thanks for reply 

Where do I have to look at in the database  On the front page summary it says about 97 800 hits, not sure it's the same thing as you asked 

Good to see the author is concerned with improving site performance. So it is only the module that has the slow query, not the component itself?

The only part of the component that's affected is the part that displays the Site Stats Summary report (just like the module). However, you can disable that report on the front end so only your backend administrators will trigger it.

I will stick with Google Analytics   It is great and I dont see anything wrong with having it yet.  Doesn't seem to put any load on my server at all that I am aware of.

But I am going to figure out how to make my rendition of 404SEF, 404SEFx, have less queries.  It puts out many useless queries I notice when I turn debug on.  Sometimes 4 times more than it should need.  But I am not sure how to get rid of them.  Any ideas?

Thanks for the info Brent.

Ty2u, it's gettin a little off topic but you could try to look at OpenSEF RC5 - not as many queries.

Thanks but it does not do what I want exactly not to mention I cant get it to work.  I don't know that I am off topic at all considering 404SEF does collect stats and that is why I came to this thread.  There is also much mention of SEF URL extensions in this thread including 404SEF   It collects hits to each page.  It collects stats about 404 errors.  That is partly the problem I suppose.  I have implemented a feature just now that lets you shut off the hit counter.  Only reduces it by one query though but if anyone can help teach people how to go about reducing the number of queries in their stats programs one step at a time I am willing to bet they won't mind doing it.

Useful thread, thank you all. I have a few questions from a technical challenged person:

1) What is the threshold where Joomla stat components will significantly slow down a site in terms of database connections? 30 concurrent uniques? 40, 50, 100? Or are local conditions so important nothing can be stated about this..?
2) Suppose I use external scripts, will this not likely be as slow as any given Joomla stat script with the difference that the script is pulling external resources (external from one's host)? In other words: using external scripts does not make any difference in site load speed from a user perspective, only your host will thank you?
3) I use Tracewatch of Arash Dekam, a nifty little stats tool with many & great features. This is implemented by one line of PHP code in my template that connects to a seperate (from Joomla) database. Will this have any effect on the speed (from user perspective) of the site loading compared to using a Joomla stat component? Joomla loads from one db, stats are inserted into another...
4) Should increased database connections not be a simple fact of life to hosters, the same way new game technology drives pc technology upwards? Or is it my responsibility, as a user, to think for my host in terms of his server load and how to tune it? To me, hosting has three qualifications: slow, acceptable or fast. Too much of the slow status will make me move to the next hoster asap. You cannot really expect users like me to analyze DB connections;
5) which brings me to a fifth question: how can I quickly analyze the number of DB connections on my shared server space, with a low level tool, please?
6) Server level stats (Raw Access Logs etc) are a b**ch to analyze. You need a good datamining tool like Sane's Nettracker to get the information you need from them.

Lastly, I respect the guys who make stat components and I have used them with great satisfaction. I cannot blame them for my own ignorance about DB connections and the implications on my site speed.
Nevertheless, I would really hate to go back to server level stats cause they are for serious professionals or data hobbyists only. I want it quick and crystal clear (without having to use crystal reports).

FYI: BSQ Sitestats 2.0.0 Public Beta will be available tomorrow. Feature list at the thread below:

http://forum.4theweb.nl/showthread.php?t=5

1) local conditions are indeed so important that nothing can be stated about this..
2) using any (internal or external) script does make use of the PHP parser (software / process on the server that interprets the code and returns HTML to the site's visitor) and could make some difference in site load speed from a user perspective. It depends on the length of the code, the processes it uses, and how many database connections/queries. Some hosters do ban customers with certain PHP scripts because those use to many resources. Again, it depends on the script you use.
3) Any connection and query to a (external or internal) database might effect the speed (from user perspective).
4) It's not all about increasing database connections. It's about processor speed, RAM memory, amount of other sites (and their appetite for the server's resources) hosted on the virtual shared server. New technology and competition of other (faster) hosting companies will drive server/hosting technology upwards.
5) As far as I know, on a well configured shared server, you will not be able to analyse any other connections beside your own (if possible).

Your arguments 1, 2 and 3 are absolutely correct.

In relation to 3 and 4 I want to add some comment.

At the time being employed as Technical Manager (CTO position) in ISP environment I tracked down some techniques:

Take a given time slot. Compare server's output performance (pages/time) in dependancy of technique being used.

100% pure HTML
~70% SSI (depends very much on how many files are included, I got IIS to crash for not being able to handle the number of files )
~30% dynamic output from database

Percent values ensure to not to depend on RAM or CPU.

Best (most comfortable) solution has always been a CMS working with database in backend but, creating more static content being used at frontend (i.e. through caching, maybe automatic update of cache after changes). One of the most importing problems with such a solution is templating when user (visitor) can change template. Putting more inteligence into caching could improve performance a lot. SSI is a way to improve situation further on.

Related to 5:
Depends. Depends very much on what you get in your hosted environment. At the very moment you're able to run programs being installed on server you should be able to analyze each connection of the very server (assuming knowledge needed is given). There's almost no way to disable it without much hazzle. Such hazzle would cause higher prices. Than you're better off getting a dedicated server.

Related to SEO:
Why do such SEO applications not create a corresponding .htaccess file?

Most of SEO parameters are kind of static nature. Therefore it isn't such a problem i.e. to filter those hardcoded names like paths, component or module names to increase security of a site at least (hardcoding paths is a veryveryvery bad idea, btw. There's something like a configuration file to make such things variable ...). Special rewrites can be bound to ids and are static for this as well.

This would shift load to server (no problem even in hosted environment) without having the need to connect to any database only to show a page. Functionality is very same what such scripts do on the fly now ...

No offence intended, my $0.05 only.

cu, diri

Thanks Peter and Diri. Insightful.

Regarding your point

Don't leave out the greed factor Since the nice Bushflation of the dollar I have a clear preference to use services of US mass hosting companies. And US suppliers/companies are delightful predictable in the sense that everything is about maximizing profits.

In terms of shared hosting this means they will fill up servers with users to the rim (rim=the trade of where every defecting customer will diminish their returns). Add more RAM and memory, have 20% more resources? Yihaa, let's put 10% more customers per server. That is the way it works with our American brethern (more than anywhere else)  and you can always bet on that. Count on greed. That's why I like them.

My point is: hoster user density management is more important to performance than a bit of Rammabam in mass hosting True technical optimization is more an issue in your high end kind of business, not in my mass hosting environment.