[GSoC 2009] Native Joomla Multisite Support

[insomn1a]

Hello everyone,
My name is Naveen Gavini, and I’ll be working on adding Multi-site support to Joomla. I am very interested in the community’s suggestions and feedback. Take a look below for more details.

Abstract
Large organizations, businesses, universities and more are establishing a web presence for every facet of their operation. Many of these organizations seek multisite support, for the ability to have one content management system/codebase that can be installed and manage more than one website. The idea for adding multisite support to Joomla would make it a viable solution for these organizations.

Since multi-site support is a very large undertaking with many subsets of the problem, I’ve decided to have this summer’s project focus on adding multisite support for a single codebase and single database. The project can be further expanded following the summer to add various combinations of multi-site support, examples include: one codebase w/multiple databases, multiple codebases w/shared database.

Outline of planned approach:
Step 1: Add siteid fields to Joomla core tables that need to be distinguished between different site installations (menus, content sections/categories)
Step2: Build component that can be used by super administrators to create/edit multi-sites
Step 3: Modify the way Joomla renders the website by limiting menu’s to specific sites
Step 4: Modify the way Joomla creates URLs (JRoute)
Step 5: Create Joomla Multi-site API, which will allow developers to add multi-site functionality to their components. This will be primarily for data separation amongst sites and enhancing multi-site functionality of components.
Step 6: Add site filtering options to modules in the administrator area
Step 7: Modify Joomla core components to allow for data separation amongst sites

Site administrators will be able to create new sites and configure data sharing or limit data to specific sites. The project will be designed for use in Joomla 1.6. Changes to the core code will be kept to a minimum and attempt to be non-invasive.

I will be blogging about the project and it's status on the community site.

Please feel free to let me know any suggestions or feedback you may have.

[csplrj]

Can there be a plugin support for enabling/disabling modules/modifying output?
Is there any roadmap/dates for the multisites/Joomla 1.6 to be launched?

Thanks in advance

CSJakharia

[insomn1a]

I believe the answers to your questions as well as more information about the project can be found here: http://community.joomla.org/gsoc2009/na ... pport.html

[MajorFusion]

This is an interesting concept. I believe that the only way that this will gain wide acceptance is if it can handle itself independently from existing components and extensions. The example that you gave as a disadvantage is what got me thinking. You suggest that code would have to be introduced into extensions to make them "multi-site aware". I really think that would cause the acceptance of this feature to stall. I mean, look at how many extensions are still stagnating in the J! 1.x world that haven't been ported to the 1.5. Looking in from the outside, I think what you are doing is fantastic and I look forward to seeing what is in store for us. I don't know how difficult it would be to isolate the "multi-site" from the extensions, but I think opening up a multi-site J! to all of the existing extensions out of the box would go a long way.
Thanks for you exciting work on this!

[robo1999]

Multisite support will be an excellent addition to Joomla CMS and I'm very excited that you took on this project.
I read your project description on how the Multisite will work and I don't see your problem with a pool being display on all sites.
But let me start from beginning.
If I understand correctly, Joomla is "menu-centric" and so is your multisite component. What is display on each site is directly related to menu assign to each site. To test this I create Pool #1 and Pool #2 and link those two pools to different menus. The results, each pool was display on a differ page (different site for multisite). Same goes with modules. To display pool in module on particular page you do this via menu (Menu Assignment).
I find one problem with pools component though. When you go to see the results you can see other pools. But this also would be a problem for ACL (different groups, different pools) so I think this is a "design" flow.

Other solution to work with components on a multisite would be to "clone" them.
This would allow to use same component and its modules with completely data separation. I don't know if you can do this with all components but you can with Sobi2. This technic is use to have more than one component working on a single site.

I agree that extra code would be helpful but you could do a lot without it.

Also, because this will be heavily menu dependent there should be some additional aids to select / display menus from particular site and better way in module edit view to assign module to menus. I saw single website having 20 and more menus. Imagine this on a multisite.

Thank you,
Robert

[gamupstate]

Consider limiting multi-sites to a MySQL v5 database and accomplishing much of your goals with views instead of php code.

A view is a virtual table built using a select statement.

So, for example, each site would run with it's own table prefix, which joomla already supports, instead of jos_XXX


For example:

jos_sometable
id integer,
sometext varchar(255)

jos_sitemapping
id integer,
sitename varchar(255),
tablename varchar(255),
rowid integer


So for site mysecondsite.com you establish that it's prefix is jos2

So you create the view
jos2_sometable
defined as "select * from jos_sometable where id in (select rowid from jos_sitemapping where sitename = 'mysecondsite.com' AND tablename ='jos_sometable')"

There you go, you can simply keep making new sites by creating a new set of views for all the tables and mapping them into sitemapping

[mariusvr]

Consider limiting multi-sites to a MySQL v5 database and accomplishing much of your goals with views instead of php code.

This is a clever idea, but the downside of this is that you are creating a read-only Joomla site. MySQL views are for viewing only and do not allow for edits. Therefore the multi-site would work, but any shared tables are "locked-down" and can not be altered (issue for instance when changing the user details, last login time, password, etc). Shared content would only be able to be edited on the main site, even though the Joomla multi-site would try to edit it, but give a fatal error when saving.

Other solution to work with components on a multisite would be to "clone" them. This would allow to use same component and its modules with completely data separation.

This is a "drupal like" solution with its own problems. With this solution, nothing is shared by default and code must be added through core hacks to allow for data sync. This would "balloon out" very quickly, as when you would create a new user on a Joomla site it executes 3 SQL queries. However if you then have 10 multi-sites any new user creation takes 30 SQL queries, plus you have the possibilities of conflicts. Then what should happen if the user add fails on one of the multi-site? Rollback all user changes on all 10 multi-sites and fail registration?

This proposed solution has a "share data by default" system, where no extra code is needed to keep the multi-sites completely synchronised (they use the same tables). Then you can use the Joomla 1.6 ACL system to limit content from the site1_usergroup to specific content. This works for modules, menus, templates and content. However this does not apply to components and they might have to add some code in order "serve up" different content to different sites. Naveen has talked about providing API functions to Joomla Extension Developers to allow them to easily add this multisite capabilities. Component will be work completely like normal with Naveens implementation (unlike the "cloned" drupal like implementation), but components need some help if they want to restrict certain views to specific sites.

Pick your poison Do you want a completely cloned site that requires a lot of code to keep synchronised and components are not compatible "out-of-the-box". Or do you want a system that is synchronises data by default and is compatible with all Joomla Extensions (but might need some alteration to components in order to limit content).

Also, because this will be heavily menu dependent there should be some additional aids to select / display menus from particular site and better way in module edit view to assign module to menus. I saw single website having 20 and more menus. Imagine this on a multisite.

Yes Naveen has included this in his project.

Thanks for the discussions! Its always great to get some fresh ideas for the project, keep them coming Thanks, Marius

[gamupstate]

Consider limiting multi-sites to a MySQL v5 database and accomplishing much of your goals with views instead of php code.

This is a clever idea, but the downside of this is that you are creating a read-only Joomla site. MySQL views are for viewing only and do not allow for edits.

Incorrect.
http://dev.mysql.com/doc/refman/5.0/en/ ... ility.html


However, there would need to be some changes to the db driver for insert statements to add the row to the mapping table.

[mariusvr]

Incorrect.
http://dev.mysql.com/doc/refman/5.0/en/ ... ility.html

Thanks for correcting me, it has been 5 years since I last views (in PostgreSQL) and it is great to see mySQL adopted the same functions. Using views would definitely open up possibilities for multi-site configurations while minimising the amount of core hacks. Although the disadvantage would be that mySQL 4 users can not use it.

Cheers, Marius

[WebJIVE]

I'm late to the game on this but another BIG disadvantage to the current thinking on multi-site Joomla is it's assumed that each site would want to share data and databases. Not so..

If you have a hosting biz and want to keep Joomla installations up-to-date with patches, it's pretty time consuming and VERY expensive when you have over 50 sites as we do.

Here's the other downside to the current multi-site thinking. Most hosting companies want to limit bandwidth and disk space. If Joomla were to handle multi-domains/multi-sites with a single base which could handle lots of domain names, the current multi-site approach doesn't allow for limiting disk or bandwidth usage as a typical hosted site would need.

The question becomes, how do you accomplish this? The only reasonable way is to have a master installation which can PUSH out code and DB updates to managed sites. This solves the bandwidth and disk quota issues. This also gets ugly again with all the separate installations in their own directories..

Security..

If you have one installation and that code base is hacked, all sites are compromised and when you have to take it down, it affects all the sites (maybe a 100 or so). That's a lot of customers to take offline.

So the real solution to this is either build in disk and bandwidth management features or build a deployment and update product to manage these installations. This is the safest route with the least amount of risk.

Thoughts?

I have tried contacting the guys at Jentla (http://www.jentla.com) but haven't heard back. This seems to be the way to go.

We're close to diving in and developing our own product if we can't locate a purchasable product to do this.

[insomn1a]

Actually we have modified the approach to use MySQL views, much thanks to all the feedback. I posted an update to this here: http://community.joomla.org/blogs/commu ... tatus.html

[WebJIVE]

I have been trying to get this Holy Grail of Joomla multisites for 2 years now and unfortunately, the approach Joomla is taking is more along the lines of a DCMS (Departmental CMS) where you can easily deploy multiple sites say in a corporation or academic environment. This works great where you need a common codebase and sites can opt to share users and content.

It still doesn't solve the hosted scenario where you need to apply bandwidth and disk quota restrictions to accounts in a shared nothing environment (except possibly a single base like JMS does). cPanel, Plesk and others don't allow bandwidth quotas at the domain level, only the account level which is what throws a wrench into the whole works.

That's the problem. Right now and we're probably going to roll our own PHP FTP Sync component whereby files are deployed via PHP/FTP and any DB updates can be in stored and applied to affected sites as well during updates.

This allows for pushing updates like JCE out to 50 sites/accounts and then push a DB update against them as well. This is pretty much the only way disk and bandwidth quotas can be maintained in a hosted environment and it isolates hacked sites to just that account instead of it being system wide.

We have yet to also find a way where we can have a safe environment and use NIX symbolic linking to use a common base either where you could maintain accounts with a common code base.

Thoughts?

Good topic BTW...

[mariusvr]

It still doesn't solve the hosted scenario where you need to apply bandwidth and disk quota restrictions to accounts in a shared nothing environment (except possibly a single base like JMS does). cPanel, Plesk and others don't allow bandwidth quotas at the domain level, only the account level which is what throws a wrench into the whole works.

Setting up an automatic Joomla hosting server with bandwidth quotas, file separation, added PHP/apache security measures, etc is outside the scope of this project.

Thanks, Marius

[garyjaybrooks]

Naveen,

Im very happy to see you doing this task. Keep us updated I would love to see your project grow and work for the community.

Keep up the good work !

[comduit]

Interesting project and I look forward to seeing its progression.

Since I have a client that needs a solution that shares content across eight domains I've spend some time considering how this can be achieved and hope my input below is of value.

I think Naveen has clearly stated his objectives for this project as using a single database to maximize the sharing of data for the benefit of large institutions. (Incidentally, I'm working on another, similar project but without any sharing at a database level.)

Based on these parameters, if one accepts that:

• File and server level security is less of an issue (than in shared hosting environments) because management is contained within a single establishment.
• The Joomla 1.6 ACL system will be used to determine who sees what.
• For each component, unless it makes use of this system its all or nothing

... then I think these are the main challenges:

• Being able to administer at a domain level - Since each domain may have its own template, menus and modules it could get very confusing as the number of domain sites increases. (Marius has already indicated that this is being factored in)
• Different component configuration per domain - This is not a problem if the configuration is stored in the database because each domain can have it's own configuration table but some components use files and one would again have to simply accept that until the component is changed, all sites will have to have the same configuration.
• Different plugin configuration per domain - Perhaps MySQL views can handle one field being different?
• Handling separation of image, language, logs, tmp and cache directories per domain - Would each domain have its own image directory or is this assumed to be shared? Would each domain using the same language use the same words? And could there be conflicts in either of the logs, tmp or cache directories?
• Additions and restrictions to administrative functionality - Presumably one would want the installation of new extensions centrally controlled and managed. For components this is particularly tricky if certain tables are not shares as these would need to roll-out for each domain or have some form of 'activation' mechanism at a domain level.
• Search - Component search plugins needing to be ACL aware.

[comduit]

Two quick questions:

1. What number of website could be viable with this model? 100? 1000? Is there a point at which having everything in a single database becomes a problem?

2. In what way does Joomla Multi Sites (JMS) not meet the objectives you set out to achieve?

Also, I have no idea what's happening with Jentla, but have you taken a look at http://www.simplweb.com/ which seems to already be providing a Joomla multi-site environment?

[WebJIVE]

1. Count depends on how the hardware and network environment are engineered.

2. JMS can indeed do this now on 1.5. Jentla is partnering with a GRID/Cloud based company to provide this service. Your seeing Jentla in action on Simpleweb.

[katart]

I have been using JMS to create Multi-Sites and it works good (with some minor issues). The main issue I had with JMS was backing up using JoomlaPack. It would back up with a continuous loop because of the symbolic links. I would hope this Joomla core implementation would solve that issue.

Thanks for continuing to improve on an already great product.

[darb]

Can there be a plugin support for enabling/disabling modules/modifying output?
Is there any roadmap/dates for the multisites/Joomla 1.6 to be launched?

Thanks in advance

CSJakharia

Would Joomla 1.6 also be supported for this version or just J 1.5? I think people want to know bcs they want to be secure for updates for coming J 1.6 release compared to other solutions...

so what you say?

[LukeDouglas]

So basically this is not a 'true' multi-site Joomla but instead a Joomla with multi-site capability. Whether you are pushing content / upgrades or pulling content / upgrades, it all deals with a single Joomla installation. Am I correct?

[maguirre]

Hi guys,

Im working in a project thats works similar that JMS called mtwMultiple. It replicates multiple sites with a custom configuration (next versions) and 3rd party extensions support too. You can test it here: http://extensions.joomla.org/index.php? ... 4&Itemid=2

I am glad to know that the code mtwMultiple can help with JMS project.

Take care and happy coding

[arlibertylover]

So is this going to be integrated into Joomla 1.6 core or not? It seems like it is going to be a component with its own API for other components to tap into. Is this a correct interpretation on my part? I am very excited about this component/functionality and I'm looking forward to 1.6 and multi-site support.

Do you think multi-site support will be added to the core as a result of your work with this project?

[LukeDouglas]

Actually, I don't want to derail this thread as it seems everyone is more interested in Multi-site support. I'm more interested in just pulling content on one website and displaying it in another as if the content was on the 2nd site without having to use an iFrame.

Anyone know a plugin, module or component that will do this?

[SageMan]

1. Count depends on how the hardware and network environment are engineered.

2. JMS can indeed do this now on 1.5. Jentla is partnering with a GRID/Cloud based company to provide this service. Your seeing Jentla in action on Simpleweb.

Thanks for letting me know of Jentla.
The moment I saw SimplWeb, I was interested in how they managed that. And with these recent frequent updates, it has become a pain updating all of these sites, and then updating all the extensions and checking for incompatibilities.

I think Jentla is covering a different customer base, more towards Hosting and Server Management, while JMS and this new Native support are for basic grouping of sites.

I'll likely utilize both features.

Brian

[darb]

There are different other possible solutions depending on what you want to do.

Joomla Multi Sites and User Sharing (comercial extension) recommended
Joomla Multi Sites allows multiple domain names or sites to have their own joomla configuration file using the same joomla installation. Possibility to share the users, and extensions like VirtueMart, Community Builder, JomSocial, ...With this new feature you can share the joomla files and folders to create totally independent web sites. This simply the maintenance of Joomla as you have only one instance to update for all the websites.

http://extensions.joomla.org/extensions ... sites/5550

mtw free mtwMultiple is an extension that ... sites/7494

Synk from Discouri comercial too
a Joomla! Database Synchronization tool that allows you to synchronize your users *and* content across multiple databases.

http://extensions.joomla.org/extensions ... sites/5501

[ccoxhead]

MultiSites uses a Core Hack, which I am trying to avoid.

mtWMultiple is brand new and, for my part, too risky a platform, on which, to base many potential sites

Virtual Domains is also new...ditto

Synk does not accomplish the same thing as it still requires multiple instances of Joomla.

Jentla is an interesting idea. Any idea what the pricing is?

[maguirre]

MultiSites uses a Core Hack, which I am trying to avoid.

mtWMultiple is brand new and, for my part, too risky a platform, on which, to base many potential sites

Virtual Domains is also new...ditto

Synk does not accomplish the same thing as it still requires multiple instances of Joomla.

Jentla is an interesting idea. Any idea what the pricing is?

Hi there

Why do you think that mtwMultiple is risky?

[ccoxhead]

I am hoping for a stable multi-site platform, on which, I can rely for a reasonably long time. I'm sad to say that many extensions end up abandoned or stagnant.

If I am to base many client sites on multi-site solution, then I need to be confident that it will be around for a while.

Frankly, Virtual Domains seems easy and promising, but it is still in Beta. I'm starting to play with it now and I plan to Donate to the project if it works out. Likewise for mtwMulitple.

I'm perfectly happy to work with a Commercial product, but I am hoping to avoid one that hacks the Joomla core.

A multi-site feature in the Joomla core represents the greatest mix of long-term viability, support, and cost.

[maguirre]

I am hoping for a stable multi-site platform, on which, I can rely for a reasonably long time. I'm sad to say that many extensions end up abandoned or stagnant.

If I am to base many client sites on multi-site solution, then I need to be confident that it will be around for a while.

Frankly, Virtual Domains seems easy and promising, but it is still in Beta. I'm starting to play with it now and I plan to Donate to the project if it works out. Likewise for mtwMulitple.

I'm perfectly happy to work with a Commercial product, but I am hoping to avoid one that hacks the Joomla core.

A multi-site feature in the Joomla core represents the greatest mix of long-term viability, support, and cost.

The reason I created mtwMultiple is entirely personal. I needed a handler Joomla installations for people working with me in my small business but the idea is to be adapted for all kinds of needs.

mtwMultiple is not to be abandoned. Im trying to make a better solution for me and for users involved in this project.

Here a small roadmap of mtwMultiple.

# Apache virtual domain support
# Groups and Users permissions support
# Front end management
# Better template management
# Joomla 1.6 support (like JMS)

If you have any feature request its welcome.

[Kualaho]

This is indeed a very good idea. It will be nice to see it implemented in joomla 1.6.
Good luck