guidance on improving Joomla 1013 module manager performance

[Cheeky Monkey]

I’ve recently implemented an intranet site using the Joomla (version 1.013) CMS

We’re up and running and all is well, however we’re now seeing a backend performance issue with the module manager

It’s worth noting that we have a relatively large site with many menu modules defined.

Checking the MySQL tables I can see that we have 4147 records in jos_menu and 3525 records in jos_modules_menu

After some analysis I’ve isolated the menu manager performance bottleneck.

Here’s a summary:

After entering the module manager in the Joomla backend and then selecting a particular menu module we are now seeing around a 12 second delay between selecting the module and the module details/editing screen actually appearing

Looking at the source code, it’s now clear that this delay is caused by a code loop in the MenuLinks() function which can be found in ..\includes\joomla.php:

/**
* build the multiple select list for Menu Links/Pages
*/
function MenuLinks( &$lookup, $all=NULL, $none=NULL, $unassigned=1 )

I’ve injected some debug code directly into the MenuLinks() function to reveal the timings for various sections of code in this function with the following results

time required to collect sql data : 0.12906908988953 seconds

// get a list of the menu items
$query = "SELECT m.*"
. "\n FROM #__menu AS m"
. "\n WHERE m.published = 1"
//. "\n AND m.type != 'separator'"
//. "\n AND NOT ("
// . "\n ( m.type = 'url' )"
// . "\n AND ( m.link LIKE '%index.php%' )"
// . "\n AND ( m.link LIKE '%Itemid=%' )"
//. "\n )"
. "\n ORDER BY m.menutype, m.parent, m.ordering"
;

time required for first pass - collect child menu records: 1.6271748542786 seconds

// establish the hierarchy of the menu
$children = array();
// first pass - collect children
foreach ( $mitems as $v ) {
$id = $v->id;
$pt = $v->parent;
$list = @$children[$pt] ? $children[$pt] : array();
array_push( $list, $v );
$children[$pt] = $list;
}

time required to run mosTreeRecurse() function : 0.15198202690155 seconds

// second pass - get an indent list of the items
$list = mosTreeRecurse( intval( $mitems[0]->parent ), '', array(), $children, 20, 0, 0 );

time required to add menu name to Display of Page(s) : 10.289244890213 seconds

// Code that adds menu name to Display of Page(s)
$text_count = 0;
$mitems_spacer = $mitems_temp[0]->menutype;
foreach ($list as $list_a) {
foreach ($mitems_temp as $mitems_a) {
if ($mitems_a->id == $list_a->id) {
// Code that inserts the blank line that seperates different menus
if ($mitems_a->menutype != $mitems_spacer) {
$list_temp[] = mosHTML::makeOption( -999, '----' );
$mitems_spacer = $mitems_a->menutype;
}

// do not display `url` menu item types that contain `index.php` and `Itemid`
if (!($mitems_a->type == 'url' && strpos($mitems_a->link, 'index.php') !== false && strpos($mitems_a->link, 'Itemid=') !== false)) {
$text = $mitems_a->menutype .' | '. $list_a->treename;
$list_temp[] = mosHTML::makeOption( $list_a->id, $text );

if ( strlen($text) > $text_count) {
$text_count = strlen($text);
}
}
}
}
}

For anyone interested, I’ve collected the above timings by injecting a few lines of code into the function to record the before and after time and then display the elapsed time in the module page as follows (note, this will only work with PHP5.x only, previous versions of PHP have a slightly modified way of collecting the time data)

Put this line before the code segment where timings are being collected
$mytime_start = microtime(true);

Put these lines after the code segment where timings are being collected
$mytime_end = microtime(true);

$myelapsedtime = $mytime_end - $mytime_start;

I also added these lines to show me the elapsed time as an item in the pages/items list box

if ( $unassigned ) {
// prepare an array with 'all' as the first item
$mitems[] = mosHTML::makeOption( 999999999, 'Elasped time: '. $myelapsedtime.' seconds');
// adds space, in select box which is not saved
$mitems[] = mosHTML::makeOption( -999, '----' );
}

As you can see the bottle neck is clearly in the loop which “adds menu name to Display of Page(s)”

Looking at this section of the code, it appears, to me at least, to be an expensive way to execute this list processing.

If I’m reading the code correctly, an initial pass of all menu modules is made to identify all menu modules that have a parent menu module.

A subsequent pass is then made taking each menu module that is found and then looping through every menu module again for each menu module to identify a parent menu module, if one exists

I think I have this right, although my head is starting to hurt now working out the logic. I think I’m close though

By posting this analysis here on the forum, I’m hoping to see here if anyone has come across this as a bottleneck and whether there have been any hacks/fixes to improve the list processing code here

My initial thoughts are to see if there’s a way to recode this (without breaking anything else in the process) so that the list processing does not occur before the module manager screen opens. One suggestion would be to introduce a new list control where the user has the option to expand those sections of the list that have child records at run time.

This way the list processing penalty is not incurred whilst the module manager screen is opening

Last night I started to look at how J1.5 manages the list processing in the module manager and can see that the old J1.013 MenuLinks() function has been deprecated.

With further analysis of the J1.5 code, I’m hoping to find that a new improved list processing technique has been implemented from which I can get some clues on how to improve the J1.013 code

I've read the post here: http://forum.joomla.org/viewtopic.php?f=500&t=266002 : "Additional logic in Module Manager Menu Item Assignment" which references a suggestion for adding new functionality in the module manager re assigning modules to pages/items but this is very much a different request. It would be interesting to see if that particular request is taken up by the Joomla dev team or other parties whilsts at the same time looking at the performance issue I've described above

All constructive comments on this are gratefully received…

[abhineshm]

I changed the block of code which was taking more than 10 seconds. Now it takes maximum of 0.18 seconds.
The inner loop with a condition check was removed as it was just being used to fetch outer loop's current record by looping through another array. The changed and removed code is in bold as below. As a whole, in loading the Menu List, the prformance is increased 400% . All comments on this reply are thankfully received…

foreach ($list as $list_a) {
foreach ($mitems_temp as $mitems_a) { // Removed
if ($mitems_a->id == $list_a->id) { // Removed
// Code that inserts the blank line that seperates different menus
if ($list_a->menutype != $mitems_spacer) {
$list_temp[] = mosHTML::makeOption( -999, '----' );
$mitems_spacer = $list_a->menutype;
}
// do not display `url` menu item types that contain `index.php` and `Itemid`
if (!($list_a->type == 'url' && strpos($list_a->link, 'index.php') !== false && strpos($list_a->link, 'Itemid=') !== false)) {
$text = $list_a->menutype .' | '. $list_a->treename;
$list_temp[] = mosHTML::makeOption( $list_a->id, $text );
if ( strlen($text) > $text_count) {
$text_count = strlen($text);
}
}
} // Removed
} // Removed
}

[Cheeky Monkey]

I should add the following comments on the above notes:

We have been recently seeing that it can take up to 60 seconds now to enter the
module manager editor with the Apache httpd process also showing a 47% cpu
utilisation at the point where the double loop (shown in the first post) is doing it's processing

This in turn has a knock on effect on the webserver performance, making the site slow to access

My php developer has made a modification to the double 'foreach' loop as described in the post above
and now we see a signifcant speed benefit of some 400%

Where it would previously take around 1 minute to enter the module manager editor
screen it is now only taking around 15 seconds!

webserver CPU load is also improved with the loop code now only taking the cpu load up to 19%

Can the Joomla devs please take a look at the above code suggestion and give us feedback if the code enhancement is sound

For reference, there is a similar code loop in the template manager processing, which
would also significantly benefit fro the code enhancement

All constructive comments gratefully received

[dhuelsmann]

My question is: Is 1.0.15 the same code??