Joomla! Discussion Forums

Well, I never quite finished the document that I mentioned earlier.  It is mostly complete but I didn't have time to get into more complex systems and to address the file system synchronization issues but there are a few ways to do it that I think people could find by searching around on Google.  So, if anyone is still interested in it to get an idea of one way to do this, I will make the document available.  Unfortunately I don't really have time to finish it now as we are very busy dealing with security issues but I would be happy to talk to people about it if you want to get in touch with me through an IM service or something.  I have many listed in my profile.

See attached.

Rob,

Thank you for taking the time to document your efforts, and then share this with the community!

Best,
Ryan

This was a perfect thread to find for me... But I dont think that this solution would work for me...

I share a similar problem to other users who posted... my site gets ALOT of media uploaded....ALOT, Songs, Photos, more photos, and soon video! lol .... So for me rsynch would just not be adequate...What might be more worth while is a true Apache Cluster, not a load balanced cluster?

Has any one tried something similar?

For you, a good solution might be to get a NAS unit (Network Accessible Storage).  Then, all of the cluster nodes can serve the website from one location (on the nas) and upload files to one location (the nas).  This introduces a single point of failure but NAS systems are pretty simple compared to servers and the higher end units are very reliable.  If you really need redundancy, there is always the option of a backup NAS as well.

Rob have I got a deal for you..!!

As Rob mentioned NAS is a pretty good solution, with RAID5 integration you will have parity protection if you lose a disk within the RAID Group and there are many reasonably cheap units out there that can have a hot spare disk to rebuild to to reduce even the effects of a disk failure in the RAID5.

In addition you can play around with the RAID5+0 arrangements,  where the RAID5 Group is subsequently mirrored for extra protection or 1+0 where you strip for performance first them mirror each stripe. This can be a very highly redundant solution and extremely well performing, especially id dedicated to a server cluster and using a Gbe NIC connection.

We use RAID5 mirrored for our DB table spaces and Striped, Mirrored for our File space, provides heaps of protection whilst performing very well under stress, NFS mounted at the moment, but planning on going to iSCSI mounted pretty soon.

Without mentioning brand names, we are lucky enough to have a reasonably high-end NAS devices (AU$10 upwards for about 2TB, 1.6TB usable) for our higher-end production but also make use of some mid-tier 1TB (850GB usable) all-in-one box units for under AU$1.5K for backup and snapshots also Development environments.

These are within Unix/Linux environments but also play nicely through CIFS and DFS for Windows platforms, the higher end devices also share across platforms concurrently also.

I personally made a dummy "load balancing", which is not good, but it is easy solution.

Intitially I started my website on a shared hosting server. Since my website became popular I very quickly overloaded the SQL limits. So I purchased a dedicated server and moved the database only on the server (and optimized the server for SQL usage mainly)
Hint: it's good if both servers are at the same network, so there is minor delay in the connections

This is somehow a load balancing, isn't it?

I would recommend everybody the following configuration (for very overloaded sites):
1. One dedicated server for the SQL only. All services stopped except the SQL server and all resources dedicated to it
2. Second dedicated server for the joomla frontend. Highly optimized for apache and php usage. All other not needed services closed
3. Optionally if your site generates too much bandwidth (like mine) get a cheap shared hosting from a hosting company that promise a lot of bandwidth (like mine promise 2500 gigs on $35 shared hosting, while the $69 P4 value dedicated server has 500 gigs only). Make a subdomain storage.yourdomain.com and host all your data for downloads there
Hint: Using subdomain for storage is very useful if you are also implementing a system for hotlinking prevention. It is very easy to set .htaccess hotlink prevention on all files using storage separate from the main site. You can even prevent the "Save target as" trick


I know it's dummy, but this is working configuration and it is very easy to maintain

Hi,

I want to thank you guys for making life easier for newbies like me 

However, may I suggest that anyone of you upgrade/reformat this thread to be suitable for placing it within the Performance FAQ's sub-forum?
http://forum.joomla.org/index.php/board,325.0.html

This will be a Time-Saver for a lot of people, who spend a lot of time searching the whole forum (avec )  before posting/asking.

Thanks a lot! 

I'm wondering if these cluster issues regarding admin sessions being dropped has been solved in subsequent versions of Joomla.

My ISP uses a clustered server configuration, and one of my users just reported this problem.

Hi,

I have one query according to my environment. I have 2 WebServers and one Database Server. I would like to put these two webservers behind the loadbalancer and will do the RSYNC for static content. I believe the frontend will work but how to deal with the backend, because this is being accessed by some domain name and I have hosted multiple sites on the same webserver. Here's an example to make it more clear of the current setup:

domain name          -      IP Address        backend URL
x.domainname.com  -      x.x.x.1              http://x.domainname.com/administrator
y.domainname.com  -      x.x.x.1              http://y.domainname.com/administrator

now i need to put one more server (x.x.x.2) in the same cluster which will have a similar setup to x.x.x.1 for the loadbalancing the application. The site can be accessed but my query is how to access the backend?

Can you please give me your inputs !!!

Regards

Sanju

Well, there are 2 possibilities.  Since everything is RSynced anyway, just access everything from one specific server (say x.domainname.com/administrator, which always points to the same physical server), but http://www.domainname.com points to the load balancer... 

You also should setup Memcache for session handling between the 3 servers (for the frontend sessions).  1.5 is better suited to this layout, but you don't NEED it...

I'm actually in the process of setting up a load balancing cluster for a site of mine...
1xWebserver (Lighttpd) -> 2x PHP servers (FastCGI), each running an implimentation of MySQL 5.1 Cluster -> 1x Backup DB server (Row-Replicated from the cluster) also running as a NAS server (to keep files in sync for the FastCGI servers).

The beauty of this setup, is that since the front end box is actually handling the connections, it's trivial to point mysite/administrator/* to the same backend box every time. 

Thanks for the reply.

my problem is how to handle the backend application... my sites runs on the subdomains - x.domainname.com & y.domainname.com and both the subdomains will point to virtual IP address where my backend need to be pointed to a static IP address so that the static content can be uploaded on the one server only and can be synced with other server later. How to handle this situation?? or if you tell me what is possible with the current joomla application for load balancing.

thanks

Sanju

If I'm understanding this thread, my configuration for should be as follows, if I were to flowchart it.
       
              Config 1
      Load Balance Server        <- Public connects to this server
Server .... Server .... Server ... <- These servers run apache,php
    MYSQL Master ... Slave ..... <- This is the Database for all servers
      Server Joomla Admin        <- Manage Joomla from behind Load Balance Server

              Config 2
      Load Balance Server        <- Public connects to this server
Server .... Server .... Server ... <- These servers run apache,php
  Shared File System Server .... <- Joomla files goes here, So no Sync is needed
    MYSQL Master ... Slave ..... <- This is the Database for all servers
      Server Joomla Admin        <- Manage Joomla from behind Load Balance Server

Is this about correct?
             

Config 1
Load Ballancer
Server1 --- Server2 --- Server N
DB Master --- DB Slave

Config 2 (the way I do it)
Server1 -- Server2 (server 1 primary, server 2 hot spare) (both run Lighttpd with FastCGI Load Balancing)
Server3 -- Server4 -- Server N (These run PHP by itself)...
DB Master -- DB Slave
File Server/Backup server

The nice thing about config2, is that you can direct all requests for /administrator/* to serverN (whatever server you want).  Using heartbeat to control the primary and hot spare, you cant get much more reduandt without geographic distrobution...

Thank you. That's just what I needed to know.

Hi All,

I am trying to get the PDF document that RobS had posted in 2006. That doc is no longer accessible (d'oh!). Is there any way I can get a hold of it? RobS, could you please repost the document?

Here is my problem: I have this installation:

{Internet}
<Firewall> (just one strong box)
Load Balancer1 and LoadBalancer 2 (failover)
<Apache/PHP Server1> <Apache/PHP Server2> <Apache/PHP Server3>
<no load balancing in between - RO taken care of by the code>
<MySQL Server Master> <MySQL Server Slave-Read Only>

Please help!

I'm running into the same situation.... Here's my configuration:
Sorry, my ASCII art is no good... This is a typical architecture for redundancy and reliability. Keep in mind, the Database is on a separate network from the SAN from the Webservers, etc. for security purposes.

ARCHITECTURE:



SCENARIO:

The issue that I need to overcome when I run anything in a distributed/redudant environment is the fact that you'll need to allow uploads to be posted and replicated across multiple servers.

My situation is that I don't want to make code changes to anything I'm hosting (Joomla, any other CMS, Forum, etc.) and I wanted to work around the SAN in case of Storage failures (just in case) as I've had an issue in the past with a different SAN environment and had all my eggs in one basket, making me rely on backups and recovering data. Not a nice time. Our new SAN is iSCSI with 4TB of RAID-5 + a Hot Spare making it 3TB usable storage and a very nice product, but just recently implemented. Our old SAN was a DAS (Direct Attached Storage), so now we have at least 2 hosts directly attached to the iSCSI SAN with the capability of taking over the other host if the host has a problem. All running Solaris 10 and very relibable.

Regardless, and to be independant of the SAN/network configuration, this is what I'm thinking about doing to upgrade our environment to make this easy and reliabile across all products/platforms/etc.:

1) Modify all web server cofigurations to reflect the new SAN storage as their primary source of the data via NFS mounting the storage locations on each web server.

2) Changing my httpd.conf files to reflect the new storage location so that any writes are made to the backend are immediately available to the other servers but keep a secondary httpd.conf file referencing the content stored locally.

3) Keep the SAN copy and Local copies in sync in case of a failure to the SAN, the web servers are finding the content locally. Probably sycn every hour or so.

CONCERNS:

* NFS Locking, but I only had issues when (in the past) I was NFS mounting a logging directory, but everyting else seemed to work ok.

* If the SAN had network connectivity issues or a failure on that side, then I would need to manually make 2 changes. 1) to the httpd.conf files to relfect the new (local) storage information and 2) update the database IP to the slave server.

I'm sure there's more, but this is what I've been working on so far to make it easy and no code changes to any product running in that environment.

Well, there's another issue, which is speed. Every file access requires network traffic... I am working on a setup (using a proposed change I made in the white paper forum, JFileBackend) that will eliminate all this. Basically, the idea is to use SVN, and svnserv's trigger system to update the servers. The number of advantages are large...

Hello All, This is my first post on the Joomla forum....

In regards to performance, my take on the subject is in regards to serving many more static vs dynamic pages.
As mentioned earlier in this thread, you can serve a hell of a lot more static pages on a server platform than dynamic with Apache.

My solution would simply be the use of mysql server -> joomla server -> caching server -> Public.

I am sure that the caching server would be able to bypass the cache for sessions and could be tweaked etc.

It goes without saying that if your site is highly dynamic then this type of setup may not meet your needs, although it probably would help reduce the load and increase performance for non session browsing etc.

All comments on this are more than welcome.

Regards

Zemadeiran

Hi Zemadeiran,

Are you referring to a reverse-proxy? This would be good to distribute globally, all connecting to the "backend" being defined by the reverse-proxy config of course. Kinda' like an Alkami or some other edge network configuration.

But in my case, I'm not sure how the posting or uploading of files will replicate back to the "backend" then get stored propertly unless the backend used shared storage such as an NFS server/cluster, etc.

I'm still working on this and currently my config looks like this (not yet in production):

(2) Sun V210's with 4GB memory / 2 1.2Ghz UltraSPARC-IIIi CPUs- having dedicated connections to the gigabit NFS network and gigabit iSCSI SAN.
(1) iSCSI Infortrend 4TB Array (RAID 5+hot spare for 3TB usable) (notice only 1, so a single point of failure!)

The two V210's will share the NFS server functionality while the other (not listed here) V210s will be load-balanced and connected via Gigabit to the NFS network.

Waiting on more hardware at this point before upgrading from our existing infrastructure. I know it will work out a lot better than I have today, but it's just the process of migrating now.. ugh.

I'll try and keep everyone posted on the hopeful success.

Shane

Hi Rob,

I'd love to be able to share your knowledge of how to effectively load-balance Joomla. I currently have my Juniper DX routing traffic to address /administrator/ to a single server in the cluster. I have a separate MySQL server that services the cluster, but the one sticking point is synchronizing media uploaded through the administrator interface or any other way.

We have a marketing group that are not very technically endowed, so I have got the new job of synchronizing the file systems across servers every single time they upload a PDF, an image, an mp3 file, or install a new mambot toy or whatever.

Basically all of the functions that are file-system-based are a problem when load balancing. How have you been able to tackle this? I've seen the posts about using a NAS device (or I guess a file server would accomplish the same thing), but part of the load-balancing concept is delivering content from multiple servers and I'm not sure I want to create another shared resource under my cluster, which is designed to create more capacity.

Any clues to another approach would be much appreciated.

(BTW I'm running on Windows Server 2003 and IIS with ISAPI-PHP5 and MySQL.)

Thanks,


Terry

Well, traditionally, you can either mount everything on shared partition (NFS, GFS, etc), or use rsync...

One thing I'm pusing for is a JFileSystemBackend for 1.6 (similar to the JFTP now, but more flexable), so you could add the library to make distributed writes in each frontend (so each user can then just act as normal, and let the codebase worry about it)...

Hi,

I have a problem with user session handling with load balancer. I have the following server set up.

Load balancer (www.mysite.com)
|
-----------------------------
| | |
Server-1 Serever-2 Serever-3(Database)

We same Joomla setup are replicated in Serever-1 and Serevr-2. Serevr-3 is user as a common DB server.
In this setup we are facing only one problem ie, "Session". We are daeling with user session in both front end and back end. We are using Joomla 1.0.15. Is there any ways to replicate user sessions in both the servers at the time of login in?

Yes... Use 1.5, with the memcache session driver...

Thank you for the info you provided..
But I am looking for a solution with Joomla 1.0.15(1.X.X.X) only. Is ther any possiblity with Joomla 1.0.15(1.X.X.X) and memcached?

we installed a peeplink with our company and it did not help with voip because the bandwidth was very slow..

we are looking at other load balancing and failover for voip and internet. plese could you let me know about

xroad network
xrio ubm
ande lfiq

for any help i am grateful

any help with load balancing....

I also have some older 1.0.15 sites that I am not ready to upgrade due to customizations and 3rd party components not ready for 1.5+.

But I'd love to load balance them!

I read Rob's PDF and got one question.

Considering a user could post a picture in Joomla to any servers on the "cluster", rsync would have to behave like a multi-master configuration, replicating the content to all other servers.

Has anyone ever used rsync this way?

Thanks a lot,
Levy

There appears to be a solution for the backend admin sessions issue for load balanced Joomla configurations here: http://blog.mattbeckman.com/2008/02/26/ ... h-haproxy/


How to Use the Administrator Control Panel in a Joomla 1.0.x Cluster

Many people understand that it’s a super big pain to work with the administrator control panel in a Joomla clustered environment. First of all, you’ll keep getting kicked out every few page requests, even while using sticky/persistent load balancing. Second, working with backend WYSIWYG rich-text editors is nearly impossible. I figured out how to do it, and here’s what I did. In Joomla 1.5, we don’t use a “live site” variable anymore, so steps 3-5 only apply to Joomla 1.0.14 and below.

1.Decide upon the management node
2.Give the management node a public host entry in DNS (e.g. node1.yourdomain.com)
3.Open configuration.php for editing
4.Locate the “live site” variable ($mosConfig_live_site)
5.Replace with “http://” . $_SERVER["HTTP_HOST"];
6.Save

Using the current host as the live site allows you to use node1.yourdomain.com as an access point for the control panel. You can work in the control panel without doing this, but you will run into tons of problems with rich-text editors and custom components that request the live site URL in their underlying code.