Joomla! Discussion Forums

I read in another post a comment by moderator Brad "It mostly comes down to hardware. If setup correctly and with enough boxes, Joomla will handle huge amounts of traffic."

I am proposing Joomla as a CMS for a new project for my company. The site receives an enormous amount of traffic. The site gets, on average, 3.3M page views a day and 900K visits.

Is there any webpage or forum thread that outlines the architecture for installing Joomla on multiple boxes?

Ultimately Joomla runs from the database

So why not create an architecture as follows:-

[Load balancer 1] -> webserver 1, webserver2,webserver3 .......  all webservers share the same joomla codebase replicated by rsync from a single server that you update.

webservers then talk to the database through their own load balancer..or for simplicity run their own database locally.

[Load balancer 2] -> db server 1, db server2, db server 3 ....etc.....

All db servers replicate as slaves to a single master server, so your read operations are nicely balanced

Now, your only bottleneck is write operations to the database. This may need some tweaks to the joomla database obkect, I've not looked into this at all and don't know how easy it is. Also, mysql 5.0 may offer a better replciation architecture, again, I haven't looked into it.

This can be done and I know that someone has been working on writing up exactly what they did. I have contacted them directly to alert them of this thread and hopefully they will post back here

Brian

Thanks both for responding. Brian I look forward to what you find out.

I have been doing more research and found architectures similiar to what Dave described.  One change was that the Joomla docroot was on a shared drive, behind all the webservers. This essentialy sync's their codebase instantly, without having to do an rsync.

When we finalize our architecture and it is load tested, I will diagram it and post it here. 

We use the following Configuration

[load balancer] Webserver 1, Webserver 2, Webserver 3

Joomla, DocRoot NFS Share on a SAN Cluster

1 Databaseserver for all Operations

Working very well with Nice Page Times, But one big problem you can't work with the backend If the Load balancer kicks you on another server while you are working, you have to login again and all work is lost. We never get a solution to this problem. In Frontend, this problem never happens....

just a thought.... what if you created a seperate Joomla install in a protected (not accessible to the public) folder that was not part of the cluster, but shared the same database and used that for administering the site.

Hi stconrad,

as Brian knows, this is something I have been working on for a little while - and now have a very nice working infrastructure, running pretty much what Dave Barter explained.

This comprises of;

LOAD BALANCER:  I have used a product called UltraMonkey 3 (http://www.ultramonkey.org/3/) to provide the load balancing for the web servers.  You can configure this for a "persistent" connection, which allows each user to continue accessing the same webserver they originally connected to during a session (which is important when you have users signing on and assigned a "session")

WEB SERVERS:  At the moment I have three Apache webservers running behind the LOAD BALANCER, each ensuring they have the same Apache configuration and file loaded onto them - this can be acheived by either RSYNC -or- a better solution that I have found is something called SSHFS - this allows you to mount drives between Linux clients via SSH (a much securer solution)

DB SERVERS:  I have two MySQL servers as well (ran out of boxes ) which are running MySQL in a Master-Slave configuration.  These servers also have the UltraMonkey software installed on them as well, which allows me to load balance the queries to them (a kind of intelligent round-robin).  I have also configured the MySQL tables to run InnoDB, rather than the default MyISAM, mainly to ensure that any locking on the tables are done on a per record basis rather than a whole table basis.

HACKING:  As Dave correctly points out, there is an issue when it comes to writing records to the databases.  In the Master-Slave configuration, all writes *MUST* be done to the Master, and not the Slave.  This means that a hack was required of Joomla!  What I did was to create a "Cluster DB Write" parameter, within the Global Configuration Admin Screen, which points Joomla! to the IP address of the Master MySQL Server.  This now allows all reads to be clustered, and writes to be directed to the Master.

The records are synced between the MySQL cluster in about 2-3 seconds - this could be longer in a larger cluster, but acceptable.  There is a better method of clustering mysql, but this involves moving the whole of the database into memory, which is fine for smaller sites - not so good for larger ones (this is called the MySQL NDB Storage Engine).

The hack also allows you to sign onto the backend and work as normal, using the persistent option within UltraMonkey, without being moved from server-to-server within the cluster.  Again, all writes are directed towards to the Master DB - via the new configuration parameter.

CURRENT STATUS:  I am currently in the process of writing this all up into an article, a HOWTO, but am also continuing the testing of the solution.  There are still some issues to iron out, such as caching, session information, document management systems and the speed of uploaded file replication between clustered webservers, etc. that need to be ironed out.

However, it does work using opensource tools - and I have ran load stress tools against the solution (vs a standalone of the same site), and it does give significant improvements.  I will also provide a Proof of Concept (PoC) to the rest of the core team, when I have finished documenting and testing, to see whether the hacks are good enough to include into the core in future versions (not guaranteed as the inclusion process is fairly rigorous now - for good reason )

Hopefully this shows you that it can be done, and a pointer to the technologies that could be used - I will of course try and get the article finished as quickly as possible.

If i Understand your solution right, every "Unique" User is bound on 1 Backend Server, correct?

This would be inacceptable unfortunatly, cause this wouldn't be real load balancing and we couldn't change boxes on the fly, or do i miss something?

It would be better, that only the Admin Backend could be connected to one server, or, the session handling there should be exact the same as on the frontpage (remeber, there isn'T a problem with logouts)

Thanks everyone for their replies. I have the budget for nine servers to run this, so we may even extend to two locations. Troozers, your "how "to" article would be invaluable.

Incorrect.  (Hopefully I am not reading your question incorrectly). 

If your Joomla!, or any web based service for that matter, doesn't require you to login - then the following won't apply.  However if you *do* need to login to your web service, you require a persistent option  - which is required to keep users going to the same web server when they log in. That is needed to preserve session information.

What I mean by this is, if User A just browses your site - he will be flicking from one load balanced server to another without even noticing.  However, as soon as he signs on - he needs to carry on using the server he has signed onto, to maintain his session information.  Other signed on users may be using completely different servers within the cluster, they just need to continue connecting to the same server they initially logged into.

if the server that User A is signed onto goes down, all that will happen is that he will be redirected to another server in the cluster and lose his session information (essentially signed off).  There is no solution that I am aware of, that will offload the session information to an external source to help with this situation (but thats probably just ignorance on my part )

*Now*, this problem may be showing up in my testing because of a misconfiguration of Apache or the Load Balancing tool, by myself - I am still investigating this fully.  But I am fairly sure, that most configurations would not take the sudden removal of a server from a cluster without some kind of whimper - the best that you can hope for is that the majority of your users would not be affected (cos they are spread across many servers), and those that were using that server can recover in some kind of manner (e.g. the worst they would see is them being signed off, but still able to view the website.

Strange,

if i log into the Joomla Frontend, or Vbulletin Forum (Frontend and Backend) i am "hopping" around the servers without getting logged of. I just get logged of in Joomla Backend (thats the problem )

...well then that means we have a solution between us 

I believe this is due to the nature of the Joomla session design.  I believe that Joomla uses MySQL entries for regular user sessions (which would be in the backend, and be possible to bounce between the various web servers), but the backend users, get a PHP cookie from the webserver, that they are logged into the back end, so, the back end users have to stay with the web server they logged in on.  But, if I remember correctly about a thread a while back having to do with user sessions, they are MySQL based, and should be able to be load balanced. 

I could be wrong about this, maybe a developer can step in and confirm or clarify this.

Good Luck,
Rob S.

No, you are right. Thats the "problem" at all .... why Backend is terrible to use in Load Balancing Situations

I wrote a simple session handler that worked with Joomla! for this exact problem.

(Found it: http://forum.joomla.org/index.php/topic,19282.0.html )

The biggest problem I had after that was uploading images using the media manager, the files go to the filesystem so only one server has it.  That required a massive overhaul to get it to work with the database.  I never finished it since we're migrating to a different CMS shortly and just set up a cron job to rsync between the the servers.

If I were to finish it, what I would do is add each image to the database when someone uploads it then when there's a request for it (using mod_rewrite to send the request to a php file), I'd check to see if the file existed on the local filesystem and read from there if it existed.  If not, read it from the database and write the file locally (less load on the DB and better performance.. the database usage has always been the downfall of my Joomla/Mambo sites).

Another option which we chose before implementing the session_handler was adding connection persistence on the load balancer so that any server we hit is the server we'd stay on.  Occasionally it'd bump you over, but not too often.  The downside to this is if you get a crazed spider just massively hammering your site, one server will take the brunt of it instead of spreading out the load.

A very timely thread as my client hopes to deploy "personal" Joomla! sites that could number in the thousands over the next 6-9 months.

Anyway, the route I am planning on taking uses a single file system that will be shared by all web servers.
Each server will boot with it's own base file systems, enought to run properly. Then each system will mount the file system that contains a single installation of Apache, PHP, Joomla! etc.

I am going to be looking at the offering from http://www.lustre.org/ as the networked file system solution.

I am hoping that by taking this approach that everthing will be worked out inherantly. Well everything but the limitations of load balancing MySQL. I will be studing the other solutions posted here to find some good resolution to that.

Any comments?

There is one problem with this type of configuration. Any user that comes from a provider such as AOL gets bounced around IPs for each of thier HTTP requests. This means that an end user of AOL will probably not have the same IP address across a session.

I currently use the offering from http://www.keepalived.org/ for my load balancing of a Java application. The persistence configuration works great, until a AOL user hits the site. Form them I have a filter that ties ALL AOL traffic to a single server so that their sessions don't expire between their IP changes. Of course that will become a large problem for sites that attract AOL type users.

The problem with AOL should be adressed in Joomla 1.0.8. Since the sites all use the same db (if i understand it correctly, you use a cluster of mySQLs?) and the sessions are handled through the db, this shouldn't be a problem.

@greergan
Could you be more specific with the load you are expecting? If you say thousands of hits, do you mean per minute, second? Because I think a good server with a properly configured Joomla can handle a thousand hits per minute without much problems.

I am glad to hear that 1.0.8 should hopefully take care of any AOLish issues with sessions.

My client is expecting to publish possibly thousands (and they hope tens of thousands) of "Personal" Joomla! sites. I.E. a user signs up with their main site and then they automatically have their own Joomla! site created.

The load balancing issues that I am up against are possibly huge amounts of video file upload/download/streaming.

I'd doubt that.  I think the best you'd be able to handle with just one (powerful) server is probably less than 300hits/min with a relatively small database.  Once you begin to grow to thousands of articles, performance will take a serious hit.  I haven't kept up recently with Joomla as my site is migrating CMSs, and I'm also not running the latest-and-greatest Joomla (It's kind of a combination of Mambo/Joomla).  Even with the site stripped down, our mysql server begins to fail when we get 15,000 hits an hour (however, I suspect it's not performing as well as it could, it wouldn't be able to handle much more).

This brings up a good idea, though.  Are there any benchmarks for Joomla?  Because I'd be interested in doing some.

What CMS are you migrating to?
What are the stats on your database server?

Thanks.

@greergan
Ok, I guess in your situation you would need such a server-cluster....  My area is more in the single page resort with a bit content...

@gharding
On a local server that is somewhat crappy, I need 0.15 seconds to generate a Joomla page with the solarflare template with default data installed. There is a lot of optimization potential in that, starting at hand-optimizing some querys, streamlining the template (the usual...) and ending at installing something like zend optimizer.
I would be so optimistic as to say that a time of 0.08 seconds is possible and with Joomla 1.1 and 1.2, this could improve more. (I'm so keen on the next versions of Joomla, cause they will come over so many restrictions in the framework and introduce so many new techniques...)

I'm migrating to OpenCms since the majority of the company's software is powered by J2EE/Oracle, and that my boss believes that PHP/mySQL sucks.  It's not really my decision, but oh well.

I think the database server is a single xeon 3.0ghz, 2gb of RAM, and 2x 74gb 7200 RPM drives in RAID 1 running FreeBSD 4.9.  I'm awaiting the DB server upgrade to a dual xeon 3.4ghz, 5x 36gb 15krpm RAID 5 on FreeBSD 6.0.  It was supposed to happen a little while ago, but it hasn't yet.  For anyone running mySQL on FreeBSD 4.9 and earlier, there are some performance issues related to the way FreeBSD handles threads (I think Jeremy Zawodny [author of High Performance MySQL] went over it in detail on his blog) which I believe are resolved in 5.0 and greater.  Sadly, I don't have access to the database server (I'm in development, not IT), so I can't tweak settings myself.

Gharding - does your site run with 300 rough hits per minute? How does it copy with this? Do you have a dedicated database server?

Also -



How does openCMS compare to Joomla!?

Hey Troozers,

Would you be willing to share this hack with me?  I think I am in a situation similar to you but, fewer servers.  I just can't really figure out what to do about the MySQL write problem and don't have the resources to go to the NDB Storage Engine.  If you would be willing to share this hack I would greatly appreciate it. 

Thanks,
Rob S.

We just upgraded our database server to a dual 3.4ghz Xeon like I had mentioned and with my cache, there's very very little load.  When I turn the cache off, the site still runs fine (with a noticeably higher load on the DB server) with 300 hits/minute.

Joomla is kind of a portal/CMS whereas OpenCms is purely a CMS.  There's very little functionality in OpenCms not related to managing content.  If you want the content to do something special (ie: hit counter, polls, forum, etc..), you're going to have to write your own (or find 3rd-party stuff which isn't nearly as abundant as Joomla's 3PD community).  We're going to begin using OpenCms as a publishing system so that we can easily put content on any of our dozens of sites from one central place.  It's really hard to compare to Joomla because they do things from a totally different perspective (Joomla, in my opinion, seems to put page layout & page functionality ahead of managing content whereas OpenCMS puts managing content ahead of everything else).  Neither is better in general, but they both do their own things, and one is probably better for the job than the other.  There have actually been quite a few features from Joomla I've implemented in OpenCms!

Well, I think it is a good time to give a little hope to any of you guys out there that have been intrigued by the idea of running Joomla in a 'cluster' (multi-server) environment.  After a few days of struggling with all of the issues that surface with this type of setup, I believe I have found a way to deal with them cleanly. 

I think I have figured out a very usable setup for Joomla in a multi-server environment.  I have managed to overcome that annoying problem of constantly getting logged out of the administration backend and only being able to write to one MySQL server.  Which, I believe were some of the biggest unresolved problems that others were having.  The best part about my fix is that it is all in software configuration and requires no modification to Joomla code.   

If any of you guys are interested in how I managed this, I am working on a little how-to to get it up and running.  Currently I have it running on two servers with Joomla 1.1-Alpha2, though I don't see why it wouldn't work in 1.0.8.  If you want, you can pm me or whatever and I will send you the write up when I finish it.

For a little taste test, check out http://www.artisteco.com and notice the page titles, "Artisteco - An Artistic Community" Followed by either "Slug" or "Ant" depending on which server you end up on.  After a few refreshes and random clicks you should end up on another server.  The page titles was a manual edit to configuration.php on each server.  That modification, and the IP address of the MySQL server are the only differences in the Joomla files on each server.  Note, I chose not to store the files on a NFS mount for a few reasons that will be more evident in my full write-up.

Good luck,
Rob S.

I have read a post regarding the load balancing of a cluster of servers including web servers & mysql servers.

According to your points "As Dave correctly points out, there is an issue when it comes to writing records to the databases.  In the Master-Slave configuration, all writes *MUST* be done to the Master, and not the Slave.  This means that a hack was required of Joomla!  What I did was to create a "Cluster DB Write" parameter, within the Global Configuration Admin Screen, which points Joomla! to the IP address of the Master MySQL Server.  This now allows all reads to be clustered, and writes to be directed to the Master."...

I think it works out for my installations because I use ultramonkey + apache + mysql + php as my web stack. Currently, I have two mysql servers running as master / slave replication. What I don't understand is how to write the PHP code so as to redirect all Write / Insert / Update queries to the master server and how to redirect the SELECT queries to the slave servers.

Can you send me the php code that you wrote?

I am looking forward for your reply. Cheers!

Thanks,
Ricky

Did you ever post article I would be very interested in reading on your findings.

Thanks for all of the information. It's like reading my own notes :).

My thinking was a little different (not using load balancing) but I've not been able to find the right tools to do this yet. I've spent weeks looking for solutions, putting together notes and ideas on how to build this for growth.

I have a multiple site (portal) Joomla setup working now but need to make sure it will handle growth before making it public.

>LOAD BALANCER, WEB SERVERS, DB SERVERS

Rather than load balancing, I was thinking more along the lines of an SSI (single server image) solution for each service.

While I have hardware based load balancers, I thought it would be much nicer to manage one single copy of Joomla for example and spread it across multiple PC's.

If I did use load balancing, what I've not found yet is now I would manage load balanced servers to see each httpd.conf as it's own. Each server has it's own IP/name, etc. I've created a small script which uses SED to modify the conf file for each server but that's not very effective. Running one single conf (unless there's a way of sharing it?) seems to be best.
I know there's a way os splitting the conf file up so that there are two parts. I could have the main part and the smaller custom part on each server? I think?

On MySQL, I was thinking that rather than getting into replication, again, using SSI to spread the load across servers.
Rather than replication, each server would see the DB as it's own. Replication would be in the form of storage backups and instant fail overs.

All storage is central using fibre channel. The servers each have small boot/OS drives and everything else is on shared storage. This way, it's quick to build a new server, fire it up and once connected to central storage, it's got all it needs to be online. Also, I can then easily add as well as replicate storage.

I have dozens of servers available for the task. I would be interested in sharing my findings with everyone once I get it working. I would also be interested in help if someone out there is a network guru with ideas but no access to much hardware.

Mike