Page Cache and Debug

Discussion regarding Joomla! 1.5 Performance issues.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Security and Performance FAQs
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Locked
almooj-craig
Joomla! Guru
Joomla! Guru
Posts: 500
Joined: Mon Aug 11, 2008 3:05 pm

Page Cache and Debug

Post by almooj-craig » Fri Sep 17, 2010 4:40 pm

Hello All,

Most of my pages are cached using the Page Cache Plugin. Out of curiosity I ran the Debug Plugin on one of these pages and I was surprised to find all of the memory and sql activity. Here is the summary for that page.

Profile Information
Application afterLoad: 0.003 seconds, 0.24 MB
Application afterInitialise: 0.244 seconds, 2.62 MB
Application afterRoute: 0.284 seconds, 3.06 MB
Application afterDispatch: 0.947 seconds, 7.41 MB
Application afterRender: 1.838 seconds, 8.33 MB
Memory Usage
8785740
27 queries logged

After looking at the cache plugin in the debug mode it wasn't using the cached data so I changed it and it now gives this report.

Profile Information
Application afterLoad: 0.008 seconds, 0.23 MB
Application afterInitialise: 0.807 seconds, 2.88 MB
Application afterCache: 0.909 seconds, 3.18 MB
Memory Usage
3356752
8 queries logged

For an experiment, in the plugins/system/cache.php file I simply check to see if there is a cached page that's not expired. Grab the contents, run a couple of regexps and modify the headers as needed and echo out the contents then die. Here are the results.

Profile Information
Application afterLoad: 0.013 seconds, 0.24 MB
Application afterCache: 0.621 seconds, 2.55 MB
Memory Usage
2882372
8 queries logged

That's an approximate change from 3356752 to 2882372 and is about a 14% reduction in memory usage (would even be lower without debug running) and the afterCache time drops from .909 seconds to .621. Checking the browsers "Time to Render" using this new approach it seems to almost always shave off about 1/2 of a second.

Other than modifying the cache plugin, is there any downside to this approach that I am overlooking?

Thanks,
Craig

qed
Joomla! Apprentice
Joomla! Apprentice
Posts: 22
Joined: Mon Dec 15, 2008 9:01 am

Re: Page Cache and Debug

Post by qed » Mon Feb 21, 2011 12:32 pm

Don't suppose you could post the changes to your debug.php ? I'd like to see how my cached pages are performing!

(yes I saw the date on his post)

almooj-craig
Joomla! Guru
Joomla! Guru
Posts: 500
Joined: Mon Aug 11, 2008 3:05 pm

Re: Page Cache and Debug

Post by almooj-craig » Mon Feb 21, 2011 5:35 pm

I have changed the plugins/system/cache.php quite a bit but you can give this a try and see if it works for you.

In the onAfterInitialise function in the cache.php file

Code: Select all

		// if($mainframe->isAdmin() || JDEBUG) {
		 if($mainframe->isAdmin()) {
			return;
		 }
Then at the end of that function add the JDEBUG, so that it looks like this: I don't remember why I commented out those two lines but you can try it both ways.

Code: Select all

			if(JDEBUG)
			{
				//$_PROFILER->mark('afterCache');
				//echo implode( '', $_PROFILER->getBuffer());
				$this->cacheDebug();
			}
			$mainframe->close();
Add this function to the plgSystemCache class

Code: Select all

	function cacheDebug()
	{

global $_PROFILER;
		$profiler	=& $_PROFILER;
		$_PROFILER->mark('afterCache');
		echo implode( '', $_PROFILER->getBuffer());

		echo '<div id="system-debug" class="profiler">';
			echo '<h4>'.JText::_( 'Profile Information' ).'</h4>';
			foreach ( $profiler->getBuffer() as $mark ) {
				echo '<div>'.$mark.'</div>';
			}
			echo '<h4>'.JText::_( 'Memory Usage' ).'</h4>';
			echo $profiler->getMemory();
echo '<hr>';
			
			jimport('geshi.geshi');

			$geshi = new GeSHi( '', 'sql' );
			$geshi->set_header_type(GESHI_HEADER_DIV);
			//$geshi->enable_line_numbers( GESHI_FANCY_LINE_NONE );

			$newlineKeywords = '/<span style="color: #993333; font-weight: bold;">'
				.'(FROM|LEFT|INNER|OUTER|WHERE|SET|VALUES|ORDER|GROUP|HAVING|LIMIT|ON|AND)'
				.'<\\/span>/i'
			;
			
			
			$db	=& JFactory::getDBO();

			echo '<h4>'.JText::sprintf( 'Queries logged',  $db->getTicker() ).'</h4>';

			if ($log = $db->getLog())
			{
				echo '<ol>';
				foreach ($log as $k=>$sql)
				{
					$geshi->set_source($sql);
					$text = $geshi->parse_code();
					$text = preg_replace($newlineKeywords, '<br />&nbsp;&nbsp;\\0', $text);
					echo '<li>'.$text.'</li>';
				}
				echo '</ol>';
			}
			
		$lang = &JFactory::getLanguage();
			echo '<h4>'.JText::_( 'Language Files Loaded' ).'</h4>';
			echo '<ul>';
			$extensions	= $lang->getPaths();
			foreach ( $extensions as $extension => $files)
			{
				foreach ( $files as $file => $status )
				{
					echo "<li>$file $status</li>";
				}
			}
			echo '</ul>';
}			

In my cache version it looks to see if there is an existing cached page and if so it grabs the data then echos it out and then calls the cacheDebug() function and then dies, like so.

Code: Select all

echo $data;					
if(JDEBUG)
  $this->cacheDebug();
die;
Having said all of that the system I currently use looks for a cached page before the Joomla application files are even loaded and the memory usage is around a few hundred k and it's much quicker.
http://forum.joomla.org/viewtopic.php?f=433&t=564106

Craig

qed
Joomla! Apprentice
Joomla! Apprentice
Posts: 22
Joined: Mon Dec 15, 2008 9:01 am

Re: Page Cache and Debug

Post by qed » Wed Feb 23, 2011 11:24 am

Cheers!

I've actually been using JotCache due to it's ability to exclude specific modules and component pages (handy for Virtuemart cart/checkout etc.) so will need to have a play with that :)

almooj-craig
Joomla! Guru
Joomla! Guru
Posts: 500
Joined: Mon Aug 11, 2008 3:05 pm

Re: Page Cache and Debug

Post by almooj-craig » Wed Feb 23, 2011 3:07 pm

I haven't used JotCache but these are the components and Virtuemart pages that I exclude from the system caching plugin.

com_user|com_login|com_virtuemart|shop.cart|checkout\..*|account\..*|shop.ask|order\..*|com_communicator

Does JotCache cache the complete page like the system plugin? I think the "Start to Render" time is the most important, it would be interesting to see what times your pages get when you are running JotCache. Try running a few tests at www.webpagetest.org

qed
Joomla! Apprentice
Joomla! Apprentice
Posts: 22
Joined: Mon Dec 15, 2008 9:01 am

Re: Page Cache and Debug

Post by qed » Thu Feb 24, 2011 9:03 am

After looking at your code it was quite easy to modify Jotcache to append debug info to the cached pages. Jotcache does look a lot like a modified system cache plugin.

Ok so, the page I used was a Virtuemart search query. Our search pages seem to be the most resource intensive as they have been modded alot to add extra features (such as category breadcrumbs).

With no caching:

Image

Image

With Jotcache:

Image

Image

Page speed score: 94/100
Last edited by qed on Thu Feb 24, 2011 12:38 pm, edited 3 times in total.

qed
Joomla! Apprentice
Joomla! Apprentice
Posts: 22
Joined: Mon Dec 15, 2008 9:01 am

Re: Page Cache and Debug

Post by qed » Thu Feb 24, 2011 9:10 am

If anyone else wants to modify Jotcache, modify onAfterInitialise in jotcache.php as follows:

Code: Select all

        if ($mainframe->isAdmin() ||  $_SERVER['REQUEST_METHOD'] == 'POST' || $message || $this->_exclude) {
            return;
        }
(JDEBUG clause removed)

Code: Select all

//JResponse::setBody($data);
//echo JResponse::toString($mainframe->getCfg('gzip'));
if (JDEBUG) {
	$_PROFILER->mark('afterCache');
	//echo implode('', $_PROFILER->getBuffer());
	ob_start();
	echo '<h4>'.JText::_( 'Profile Information' ).'</h4>';
	foreach ( $_PROFILER->getBuffer() as $mark ) {
		echo '<div>'.$mark.'</div>';
	}
	echo '<h4>'.JText::_( 'Memory Usage' ).'</h4>';
	echo $_PROFILER->getMemory();
	$db	=& JFactory::getDBO();
	echo '<h4>'.JText::sprintf( 'Queries logged',  $db->getTicker() ).'</h4>';
	$debug = ob_get_clean();
	$data = str_replace('</body>', $debug.'</body>', $data);
}
JResponse::setBody($data);
echo JResponse::toString($mainframe->getCfg('gzip'));
$mainframe->close();
Note: this doesn't output the queries (only the count) or any language file info.

Oh and if you're having trouble reading the jotcache.php file, try running it through: http://www.beautifyphp.com/

almooj-craig
Joomla! Guru
Joomla! Guru
Posts: 500
Joined: Mon Aug 11, 2008 3:05 pm

Re: Page Cache and Debug

Post by almooj-craig » Fri Mar 04, 2011 8:23 pm

qed,

Thanks for posting the test data, it's interesting to see results from other sites.

Your Start to render time 1.583s without caching is fast. Why do you have so many queries (203 queries logged)? The after caching time of 1.013s is good.

When I run virtuemart pages that have not been page cached yet, I get:
Capture.jpg
Profile Information
Application afterLoad: 0.001 seconds, 0.90 MB
Application afterInitialise: 0.079 seconds, 5.21 MB
Application afterRoute: 0.087 seconds, 5.72 MB
Application afterDispatch: 0.334 seconds, 18.13 MB
Application Menu: beforeRender: 0.364 seconds, 18.77 MB
Application Menu: afterCache: 0.365 seconds, 18.77 MB
Application Menu: beforeRender: 0.422 seconds, 19.34 MB
Application Menu: afterCache: 0.422 seconds, 19.34 MB
Application afterRender: 0.444 seconds, 19.81 MB
Memory Usage 20818672 60 queries logged

Once a page is cached I can't run debug because it does not actually load the application framework, but the memory usage is about 798760.
Capture-cached.JPG
Update: I was just trying out a PHPCompressor https://github.com/technosophos/PHPCompactor and I was able to drop the memory usage down to 793168 and if I don't do the memory check it would probably shave off a few more bytes.

Another Update: I found that Apache could parse the php file better if the code wasn't on a single line, so I changed it so there is a return after each semicolon. I also went ahead and split the main index file and compressed the first part and loaded the application code in another file. This reduced the memory usage to 778,336 (with memory check). So by minimizing the two initial php files the memory usage is reduced by over 20,400.

Last Update: I removed the functions that only run about 5% of the time and put them in their own file. It now uses 739504 which is almost a 60k reduction by just compacting and splitting a couple of files.
You do not have the required permissions to view the files attached to this post.

qed
Joomla! Apprentice
Joomla! Apprentice
Posts: 22
Joined: Mon Dec 15, 2008 9:01 am

Re: Page Cache and Debug

Post by qed » Wed Mar 09, 2011 1:16 pm

what's your method for not loading the application framework?

we've had to move to a dedi server now as rochen were moaning about cpu usage being >2% avg on some days. good excuse for me to play with xcache now tho :D

almooj-craig
Joomla! Guru
Joomla! Guru
Posts: 500
Joined: Mon Aug 11, 2008 3:05 pm

Re: Page Cache and Debug

Post by almooj-craig » Wed Mar 09, 2011 4:19 pm

First with dbase then perl, I have always generated flat html pages because they are quick. Now with Joomla my goal is to do the same thing, use Joomla as a cms to generate flat pages.

(1) Make sure that page cache only stores the pages or aid levels that you want to store. Don't store any pages that use a security token, for example have your uncached login page load after they click on a login link. Save the cached data as a gziped file not serialized.

(2) Add a require once "cache_index" to the main index.php file before the require_once ( JPATH_BASE .DS.'includes'.DS.'defines.php' ); line. Currently I have the index.php file split in two, see below.

(3) In the cache_index file I basically do the following.

(a) Load the configuration file so I can get the mysql info and the secret string.
(b) Query to see if the ip address is not banned
(c) Check for the Joomla cookie. If it isn't there then they are a first time visitor, send them files that don't use external css files and set the cookie.
(d) Query the aid level of the user so I know what files to send them and then update their session data so that they don't time out.
(e) Check to see if the url they are requesting is cached, if not bail out and load Joomla.
(f) Check to see if they accept gzip files send them the gziped file directly, otherwise uzip it for them.
(f) Update the ip information using a register_shutdown_function
(g) die before the application is loaded. (index_application.php)

For 90% of pages it only needs to load 2 files and up to four queries. The split index.php file that's 247b and then the cache_index file that's 3725b.

For example this is what's in my main 247b index.php file.

Code: Select all

<?php
define( 'DS', DIRECTORY_SEPARATOR );
define('JPATH_BASE', dirname(__FILE__) );
define( '_JEXEC', 1 );
require_once ( JPATH_BASE .DS.'plugins'.DS.'system'.DS.'cache_index.php' );
require_once ( JPATH_BASE .DS.'index_application.php' );
The remainder of the original index.php file code in in the index_application.php file.

qed
Joomla! Apprentice
Joomla! Apprentice
Posts: 22
Joined: Mon Dec 15, 2008 9:01 am

Re: Page Cache and Debug

Post by qed » Wed Mar 09, 2011 5:04 pm

That's a nice solution, they should implement it, I'm certainly going to! Cheers :)

qed
Joomla! Apprentice
Joomla! Apprentice
Posts: 22
Joined: Mon Dec 15, 2008 9:01 am

Re: Page Cache and Debug

Post by qed » Thu Mar 31, 2011 8:05 am

Just to update, have implemented your solution and been running it for about a week...

MASSIVE improvement in performance and server load. Thanks for sharing! :D

For anyone who wants to give this a try, to get you started quickly, here's a couple of functions to get a cached page:

Code: Select all

//xcache method
function _getXCachedPage($config,$url){
	$uri = md5(urldecode($url));
	$name = md5('-'.$uri.'-'.$config->secret.'-en-GB');
	$cache_id = 'cache_page-'.$name;
	return xcache_get($cache_id);
}

Code: Select all

//file method
function _getCachedPage($config,$url){
	$uri = md5(urldecode($url));
	$name = md5('-'.$uri.'-'.$config->secret.'-en-GB').'.php';
	$dir = '/PATH/TO/YOUR/cache/page';
	$path = $dir.DS.$name;
	if (file_exists($path)) {
		$data = file_get_contents($path);
		if($data) {
			// Remove the initial die() statement
			$data	= preg_replace('/^.*\n/', '', $data);
		}
	}
	return $data;
}
The $url should be passed as "/path/to/your/page" or "/index.php?option=com_whatever" depending on your setup.

almooj-craig
Joomla! Guru
Joomla! Guru
Posts: 500
Joined: Mon Aug 11, 2008 3:05 pm

Re: Page Cache and Debug

Post by almooj-craig » Sun Apr 10, 2011 1:46 pm

Qed,

Glad to hear that you got it working. Did you happen to try to gzip the pages before they are saved and then serve them out directly. It really saves a lot of space and it's quicker. Another tip that I use that works very good is to sort the query before the md5 is created. It keeps the duplicate page storage down to a minimum, especially with Virtumart.

Out of curiosity what is the memory usage when you load the cached pages?

User avatar
ahmad
Joomla! Guru
Joomla! Guru
Posts: 902
Joined: Fri Apr 07, 2006 4:02 pm
Location: Egypt
Contact:

Re: Page Cache and Debug

Post by ahmad » Sun Feb 26, 2012 6:49 am

Can you post the code that you used in the plugin 'cache_index.php'?

almooj-craig
Joomla! Guru
Joomla! Guru
Posts: 500
Joined: Mon Aug 11, 2008 3:05 pm

Re: Page Cache and Debug

Post by almooj-craig » Wed Mar 14, 2012 2:33 pm

Ahmad,
I have been pondering your request but that 'cache_index.php' file would not work for another Joomla instillation without a lot of modification. If you have a question about a part of the method that I outlined, I could post the relevant code.

My newest idea at the end of February is to always make sure that all pages are cached in advance and that spiders that do not set cookies always get the same session_id number.

Here is a screen shot of a current Webmaster Tools screen shot from a site that has 6T different Virtuemart pages cached.
Capture.jpg
You do not have the required permissions to view the files attached to this post.

qed
Joomla! Apprentice
Joomla! Apprentice
Posts: 22
Joined: Mon Dec 15, 2008 9:01 am

Re: Page Cache and Debug

Post by qed » Wed Mar 14, 2012 4:05 pm

here's the basic part of what I use:

Code: Select all

if( !defined( '_JEXEC' ) ) die( 'Direct Access to '.basename(__FILE__).' is not allowed.' ); 
require_once('configuration.php');
$config = new JConfig();
$url = $_SERVER['REQUEST_URI'];
session_start();
//xcache method
function _getXCachedPage($config,$url){
	$cache_id = 'cache_page-'.get_name($url,$config);
	return xcache_get($cache_id);
}

//file method
function _getCachedPage($name){
	$data = '';
	$dir = '/PATH/TO/YOUR/cache/page/';
	$path = $dir.$name;
	if (file_exists($path)) {
		$expires = file_get_contents($path.'_expire');
		if ($expires > time()){
			$data = file_get_contents($path);
			if($data){
				// Remove the initial die() statement
				$data = preg_replace('/^.*\n/', '', $data);
				$data = trim($data);
			}
		}else{
			unlink($path);
			unlink($path.'_expire');
		}
	}
	return $data;
}

function get_name($url,$config){
	$uri = md5(urldecode($url).'-');
	$name = md5('-'.$uri.'-'.$config->secret.'-en-GB').'.php';
	return $name;
}

function check($data,$name){
	// for anything you might want to check
	return true;
}

if(!isset($_SESSION['cart']['idx']) && !preg_match('/(cart|checkout|password|wishlist|contact|activ|user|account|ssl|basket|order)/',$url) && $_SERVER['REQUEST_METHOD'] == 'GET'){
	$name = get_name($url,$config);
	$data = _getCachedPage($name);
	if ($data != '' && check($data,$name)){
		echo trim($data);
		die();
	}
}
I should point out that generally speaking my site doesn't use query strings so no need to sort them - although I agree it is a good idea if you do use them alot.

almooj-craig
Joomla! Guru
Joomla! Guru
Posts: 500
Joined: Mon Aug 11, 2008 3:05 pm

Re: Page Cache and Debug

Post by almooj-craig » Mon Mar 26, 2012 3:32 am

Here is some of the code that I use. In this part:

(1) First I clean up the URI. Don't cache the complete component/search, sort the query, get rid of anchor tags.

(2) Figure out their aid level

(3) Make the path. Currently I'm using different sub directories to reduce the amount of files in each directory.

(4) I think checking the filectime is quicker than file_exists and then reading the file.

(5) If they are requesting a sub domain that they don't have access to, then 301 them back to what is usually "www"

(6) If they are logged in don't store the page in the browsers cache otherwise store for only one day.

(7) If they can receive a gz file then send it directly to them otherwise unzip it and send it to them. Saves space and keeps your server from having to deflate it on the fly. Very rarely will you need to unzip the file.

(8) If they are a first time visitor then include the css with the file.

(9) Use register_shutdown_function to keep them logged in along with some other tasks.

Code: Select all

$cURL = $_SERVER['REQUEST_URI'];
if (strpos($cURL,'/component/search') !== false)
	$cURL = '/component/search';
else
{
	if ($_SERVER['QUERY_STRING'])
	{
		require_once( JPATH_BASE .DS. 'plugins' .DS. 'system' .DS. 'cache' .DS. 'cache_index_helper.php' );
		$cURL = cacheIndexHelper::sortQuery($cv);
	}
	else if (preg_match('/(.*)(#|%23)([^&]+)$/', $_SERVER['REQUEST_URI'], $a))
	{
		$cURL=$a[1];
	}	
	if (empty($cURL)) $cURL='/';
}	
$id  = md5('-'.md5($cURL) .'-'. $cv->secret);

if ($cv->cache_index == 1 and strpos($_SERVER['HTTP_HOST'],$cv->redirect.'.') !== false)
	$aid=0;
else
{
	$q = "SELECT data FROM {$cv->dbprefix}session
						WHERE session_id = '$sid'";

	$result = mysql_query( $q );
	$row = mysql_fetch_array( $result );			
	if (strpos($row['data'],'"aid";i:2')!==false)
		$aid = 2;
	else if (strpos($row['data'],'"aid";i:1')!==false)
		$aid = 1;
	else
		$aid = 0;
	if ($aid >= $cv->cache_index)
	{	
		mysql_close($link);
		return;
	}
}

//$path = $cv->cache_path.DS.'page'.DS.$id.'_'.$aid.'_en-GB.php';
$path = $cv->cache_path.DS.'page'.DS.substr($id, 0, $cv->sd_length).DS.$id.'_'.$aid.'_en-GB.php';

//if(file_exists($path.'_expire')) 
if (time() < filectime($path)+($cv->cachetime*60))
{
	if ($aid==0 and strpos($_SERVER['HTTP_HOST'],$cv->redirect.'.') === false)
	{
		$host = preg_replace('/^[^.]+/',$cv->redirect, $_SERVER['HTTP_HOST']);
		header('Expires: Mon, 1 Jan 2001 00:00:00 GMT'); 				// Expires in the past
		header('Last-Modified: '. gmdate("D, d M Y H:i:s") . ' GMT'); 		// Always modified
		header('Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0');
		header('Pragma: no-cache' ); 											// HTTP 1.0
		header('Status: 301 Moved Permanently', false, 301);
		header("Location: http://$host{$_SERVER['REQUEST_URI']}");
		exit();
	}
	if ($cv->debug)
		header('X-cURL: '.$cURL . ' ' .$id.'_'.$aid);
	
	//$time = @file_get_contents($path.'_expire');
	//if (time() < $time) 
	{
		header('Content-type: text/html; charset=utf-8');
		
		if ($aid or strpos($_SERVER['HTTP_HOST'],$cv->redirect.'.') === false)
		{
			header('Expires: Mon, 1 Jan 2001 00:00:00 GMT'); 				// Expires in the past
			header('Last-Modified: '. gmdate("D, d M Y H:i:s") . ' GMT'); 		// Always modified
			header('Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0');
			header('Pragma: no-cache' ); 											// HTTP 1.0
		}
		else
		{
			header('Expires: '. gmdate( 'D, d M Y H:i:s', time() + 86400) . ' GMT');  // Expires in 1 day
		}
		
		if ($cv->debug)
			header('X-Mem: '. memory_get_usage());
		
		if (empty($_SERVER['HTTP_ACCEPT_ENCODING']) || strpos($_SERVER['HTTP_ACCEPT_ENCODING'],'gzip') ===false)
		{	
			require_once( JPATH_BASE .DS. 'plugins' .DS. 'system' .DS. 'cache' .DS. 'cache_index_helper.php' );
			cacheIndexHelper::readGZ($path);
		}
		else
		{
				header('Vary: Accept-Encoding');
				if (!empty($nid))
				{
					if (file_exists($path.'_1'))
					{
						header('Content-encoding: gzip');
						header('X-Cache: Index 1');
						echo @file_get_contents($path.'_1');
					}
					else
					{
						require_once( JPATH_BASE .DS. 'plugins' .DS. 'system' .DS. 'cache' .DS. 'cache_index_helper.php' );
						cacheIndexHelper::headerCSS($path, $cv);
					}
				}	
				else
				{
					header('Content-encoding: gzip');
					header('X-Cache: Index');
					echo @file_get_contents($path);
				}	
		}	
		register_shutdown_function('indexSaveData', $cv, $sid, $id.'_'.$aid, $cURL);
		die;
	}
}
define( 'cURL', $cURL );


Locked

Return to “Performance - Joomla! 1.5”