EDIT NOTE:  This .htaccess file is really working well, thanks to all the input and testing others have provided.  Please continue testing and post problems in this thread.  We will continue to revise as needed.  If you want to skip this lengthy explanation, the code is at the bottom of this post.

I have spent the last couple of days working on this.  Although Joomla 1.07 returns 404 headers for non existant content pages, it still processes a lot of unwanted content (gif,jpg,css, etc...).  I wanted to reduce this processing and let the server handle it on its own.

On older sites (pre 1.07) I consistantly got the front page, even when typing in a nonexistant graphics filename.  In addition, subdomains of the primary domain and add-on domains, which live on my server in subdirectories of the main domain, did the same (the primary domain was pre 1.07).

I finally figured a few things out.  The existing .htaccess file has this line in it:

Code: Select all

RewriteCond %{REQUEST_FILENAME} !\.(jpg|jpeg|gif|png|css|js|pl|txt)$

Which is applied with this rule:

Code: Select all

RewriteRule ^(.*) index.php

This means that apache will look first to see if it has a rule applied to (.*) - everything, then start processing them according to its rules of inheritance.  In my case, my ISP has set the mod_rewrite module to inherit rule sets prom parent directories, and the virtual host directives.  I suspect that most do to ensure that the virtual host directives are not sidestepped..

So, my server was processing the rulesets for each parent directory, that led to the directory that contained the .htaccess file it was currently processing, in thier order starting from the docroot.  For example, running a test on a nonexistant file http://domain.com/test/nothere.jpg it was processing rules from the .htaccess files contained in /server_docroot/user_dir/public_html/primary_domain/add_on_domain/test_directory.  In my case, this meant it was processing the .htaccess files from the primary_domain, add_on_domain, and test_directory directories.

I used the standard (1.07) rewrite rules on the test directory, but was consistantly getting the front page no matter what I did.  After some futzing around (two days worth) I fnailly was able to determine that the Apache environment variable we were testing was changing with each successful RewriteRule in each of these directories.  So the {REQUEST_FILENAME} was getting constantly turned into 'index.php' leading to unmatched rulesets and thus not getting the right action for the rewrite.  In each case, the base ruleset for the accessed domain ended up winning the war, and I inevitably ended up on the front page for that domain, even when trying to access these non-existant css and jpg files.

To correct the problem, I started testing on a more constant variable, which in my case turned out to be {REQUEST_URI}.  It stayed constant and always contained the actual request (minus the domain).  But still had the problem, even though I was getting more succesful matches.  Non-existant jpg and css files still brought me back to the accessed domain's front page.

After spending some time (again) futzing with the RewriteCond's, I finally found a way to consistantly return the front page only on requests for non-existant .htm, .html, .php, and directory requests (like Joomla sef URLs).

I replaced the standard line:

Code: Select all

RewriteCond %{REQUEST_FILENAME} !\.(jpg|jpeg|gif|png|css|js|pl|txt)$

With this:

Code: Select all

RewriteCond %{REQUEST_URI} (/|\.htm|\.php|\.html|#.*|\?.*|/[^.]*)$  [NC]


I haven't had any problems since.  It has worked flawlessly since I made this change, no matter what directory or domain I have accessed it from.  I took the time to change this on each base installation, regardless of version, and it has worked like a champ, each still able to access its own SEF URL's.

Here's the code, last modified February 10th (for clarity - no functional changes):

Code: Select all

######COPYRIGHT##########
# .htaccess file for Joomla/Mambo SEF useage
# Copyright 2006 Steve Graham   Contact me at http:/coders.mlshomequest.com/
# Licensed under the LGPL.  You are free to use and include this file in
# any project or codebase subject to the LGPL license at:
# http://www.gnu.org/copyleft/lesser.html
######END COPYRIGHT#######

##
#  READ THIS COMPLETELY IF YOU CHOOSE TO USE THIS FILE
#
# The line just below this section: 'Options FollowSymLinks' may cause problems
# with some server configurations.  It is required for use of mod_rewrite, but may already
# be set by your server administrator in a way that dissallows changing it in
# your .htaccess file.  If using it causes your server to error out, comment it out (add # to
# beginning of line), reload your site in your browser and test your sef url's.  If they work,
# it has been set by your server administrator and you do not need it set here.
#
# Only use one of the two SEF sections that follow.  Lines that can be uncommented
# (and thus used) have only one #.  Lines with two #'s should not be uncommented
# In the section that you don't use, all lines should start with #
#
# For Standard SEF, use the standard SEF section.  You can comment out
# all of the RewriteCond lines and reduce your server's load if you
# don't have directories in your root named 'component' or 'content'
#
# If you are using a 3rd Party SEF or the Core SEF solution
# uncomment all of the lines in the '3rd Party or Core SEF' section
#
#####  SOLVING PROBLEMS WITH COMPONENT URL's that don't work #####
# SPECIAL NOTE FOR SMF USERS WHEN SMF IS INTEGRATED AND BRIDGED
# OR ANY SITUATION WHERE A COMPONENT's URL's AREN't WORKING
#
# In both the 'Standard SEF', and '3rd Party or Core SEF' sections the line:
# RewriteCond %{REQUEST_URI} ^(/component/option,com) [NC,OR] ##optional - see notes##
# May need to be uncommented.  If you are running your Joomla/Mambo from
# a subdirectory the name of the subdirectory will need to be inserted into this
# line.  For example, if your Joomla/Mambo is in a subdirectory called '/test/',
# change this:
# RewriteCond %{REQUEST_URI} ^(/component/option,com) [NC,OR] ##optional - see notes##
# to this:
# RewriteCond %{REQUEST_URI} ^(/test/component/option,com) [NC,OR] ##optional - see notes##
#
##

##  Can be commented out if causes errors, see notes above.
Options FollowSymLinks

#
#  mod_rewrite in use

RewriteEngine On

#  Uncomment following line if your webserver's URL
#  is not directly related to physical file paths.
#  Update Your Joomla/MamboDirectory (just / for root)

# RewriteBase /


########## Begin Standard SEF Section
## ALL (RewriteCond) lines in this section are only required if you actually
## have directories named 'content' or 'component' on your server
## If you do not have directories with these names, comment them out.
#
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
#RewriteCond %{REQUEST_URI} ^(/component/option,com) [NC,OR] ##optional - see notes##
RewriteCond %{REQUEST_URI} (/|\.htm|\.php|\.html|/[^.]*)$  [NC]
RewriteRule ^(content/|component/) index.php
#
########## End Standard SEF Section


########## Begin 3rd Party or Core SEF Section
#
#RewriteCond %{REQUEST_URI} ^(/component/option,com) [NC,OR] ##optional - see notes##
#RewriteCond %{REQUEST_URI} (/|\.htm|\.php|\.html|/[^.]*)$  [NC]
#RewriteCond %{REQUEST_FILENAME} !-f
#RewriteCond %{REQUEST_FILENAME} !-d
#RewriteRule (.*) index.php
#
########## End 3rd Party or Core SEF Section

I tested on Linux only, haven't tried this on my Apache/Windows test server yet, but wanted to pass this along for the benefit of others who were experiencing the same thing.

It works flawlessly, and I always get the server's 404 error for any nonexistant file/directory that is not .html, .htm, .php, or a directory.

If you are going to try this, I would suggest backing your old .htaccess file up (obviously), but I think I nailed it.

Consistant 404s, no more potential loops for nonexistant files, and no more Joomla instantiation (even with 1.07 and its 404 page) for these non-existant files.

GRAM

Just changed the new line in the code above. 

Changes will cover a SEF URL with a normal cannonical query string at the end, a URL that has a page bookmark in it, and a URL that does not have a trailing slash and also does not have a period in it (like a normal SEF URL that is missing its trailing slash).

GRAM

Thanks for this information Gram, I will move it to the 1.0.x forum. There are more people there with .htaccess problems.

Thanks Tonie, wasn't sure where this belonged.  Although it does benefit 1.07 too, seems like pre 1.07 users are having the major problem.

Just updated the RewriteRule/s and RewriteCond/s to optimize and reduce POSIX overhead and reduce scope for Rewrite.  Results are faster and less Apache processing.

GRAM

Thanks Gram!

Your suggestion solved my problem: http://forum.joomla.org/index.php/topic,30612.msg177867.html#msg177867

With 1.07 I cold not use  /administrator/ if I had password protected the administration folder in cPanel, had to use /administrator/index.php.

Glad to hear it Troll !

That is an unintended cosequence, but falls in with the normal problems encountered with the broad scope of the htaccess.txt file that is included with Joomla.

Problem is twofold:

The first RewriteCond:

Code: Select all

RewriteCond %{REQUEST_FILENAME} !\.(jpg|jpeg|gif|png|css|js|pl|txt)$

Is supposed to exclude files with those extensions, but does not work.  It ends up being the equivalent of (.*) for our purposes and allows processing to continue, even for files with those extensions.  I have not been able to get Apache to match a partial negative POSIX statement at all.  Seems like you have to match the entire string or it does not work at all.

The alternative I posted uses partial positive matches, which do work and will only allow processing to continue for file/directory requests that match the Statement.  This automatically excludes anything that does not match and stops the Rewrite processing when matches don't occur.

The RewriteRule:

Code: Select all

RewriteRule ^(.*) index.php

Is very broad.  It tells the Rewrite engine to look for qualified Rewrite conditions for every request.  This is needed with all of the advanced SEF engines, but not needed for standard Joomla SEF processing.  By reducing this to the two directory's used by standard SEF, for standard SEF, Apache won't even check for Rewrite Conditions unless one of those two directories exists in the requested URL.

If you used the Standard SEF RewriteRule, take note.  Your problem will most likely return if you convert to using an advanced SEF component and try to use the broader RewriteRule for the advanced SEF's.

GRAM

Code: Select all

<Files .htaccess>
order allow,deny
deny from all
</Files>

Options -Indexes FollowSymLinks

#
#  mod_rewrite in use
#RewriteEngine On

RewriteEngine On

#  Uncomment following line if your webserver's URL
#  is not directly related to physival file paths.
#  Update YourMamboDirectory (just / for root)

# RewriteBase /

#
#  Rules
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

#move following line to top of RewriteCond's for Advanced SEF's
#It will work here, but moving it will reduce Apache overhead
RewriteCond %{REQUEST_URI} (/|\.htm|\.php|\.html|#.*|\?.*|/[^.]*)$  [NC]

#Only use one of the two RewriteRules below, comment the other out
#First line is for Joomla/Mambo standard SEF
#Second line is for Advanced SEF usage
RewriteRule ^(content/|component/) index.php
#RewriteRule (.*) index.php

So I copy and pasted this into my .haccess file and now I only get 'Forbidden' errors everytime I try to go to a page.  What am I missing?  This is also effecting my forum that is in a subdomain, forum.mysite.com

Charles,

Try mixing this file with your original .htaccess file.  For example, your server may be having a problem with the -indexes statement or the section.

I would suggest starting with the section and commenting it out, then try the deleting the -indexes command.

BTW...  Just making sure, but you had SEF working, and you were previously using the standard Joomla .htaccess file?

GRAM

Yeah, SEF was working find with the standard .htaccess file.  I will try to do some mixing and matching.

Charles

Excellent work gram... I was hoping to look into som mod_rewrite stuff and had not to this point had the time... check your pm box.


Louis

Charles,

I would be happy to help, if appropriate.

You might want to do what I did.  I set up a subdirectory off of my root just for testing.  In order to make this work right, be sure to use the standard RewriteRule (from my post, not the orginal Joomla .htaccess) in your root .htaccess so it doesn't skew the results in that new subdirectory.  You don't need anything else from my post, just the RewriteRule for standard SEF. 

Just create a simple file in that new directory and name it 'index.php'.  I made a file that called phpinfo().  If the rewite rule applied, it would call that file.  If it didn't, I would get a 404 error (remember 'index.php' is the only file in that directory).

Put this new .htacces file into it, then try both 'index.php' and any other filename that should return a 404 (like a css,jpg,png etc... remember there is nothing else in this directory but an .htaccess file and the new index.php) to see if your getting 404's.

If you get the error you posted about, you can try changing lines in the .htaccess for that directory where it won't effect your production site.  I would really like to know what you find out.

GRAM

I got your PM Louis.  Reply sent.

If you need some tips on how to test for this, let me know.

GRAM

So what's wrong with just:

Code: Select all

RewriteEngine on

RewriteRule ^content/(.*) index.php
RewriteRule ^component/(.*) index.php

I've been using that on http://www.trunkmonkey.com (40,000+ page views per day with Joomla 1.0.7) and all my other sites with no ill effects and very low server load.

Edit: Nevermind...I followed this link from a general .htaccess question. Didn't notice that your method gave proper 404s.

Hi, developers

What .htaccess to use more preferably: regular for 1.0.7 or suggesting Gram?
I tested both - they work for 1.0.7. It would be to hear the recommendations from the developers (Jinx, Saka), who can tell?

Found a small issue, by testing the htaccess from gram on my open.sef site the bridged smf got probs:

Personal Message (Online)

The requested URL was not found on this server. The link on the referring page seems to be wrong or outdated. Please inform the author of that page about the error.

If you think this is a server error, please contact the webmaster.
Error 404
http://www.open-sef.org
Fri Jan 27 02:08:02 2006
Apache/2.0.48 (Linux/SuSE)

in the urls like:

http://www.open-sef.org/component/optio ... 01#msg1201
http://www.open-sef.org/component/optio ... 00#msg1200
http://www.open-sef.org/component/optio ... 99#msg1199

P.S. works now have restored the "old" htaccess but with gram's i got this.

Thanks for weighing in and testing this Marko.

I had to go back and play with the regex to come up with a matching pattern that was not too broad.

I changed the RewriteCond that matches to the REQUEST_URI.  Changes are made to the original post in this thread.

Please try again, and let me know.  I tested this with your posted urls and they all worked.  As you posted, they did not work with the original RewriteCond.

Thanks,

GRAM

This post is being referred to by others regarding some .htaccess issues.  I haven't had any problems with my site that I know of but I tried the attached changes.  It didn't work on my site using the standard SEF.  I do have caching turned on.  Do the caches need to be cleared after applying this?

Thanks

p.s.  When I apply the change, I get 500 errors. I can get to the home fine but no components.  [Edit]  Cleaned up my poor grammar...  :-(

Thanks for the quick fix, guilliam did found this issue just 5 minutes after i have put it on my site  is working now, will leave it on the site to see how it works with OpenSEF, i use the part for 3party so if i see the next days some issues, i will let you know

BoardJIVE:

Your the second person who has gottne 500 errors on his server after implementing this.

I think it has something to do with different versions of Apache, with different configs in them.  Please help and answer these questions for me.

1. Did you ever have SEF working before trying this?

2. Have you tried commenting out the section?

GRAM

Thanks Marko, and please let me know.

GRAM

gram wrote:BoardJIVE:

Your the second person who has gottne 500 errors on his server after implementing this.

I think it has something to do with different versions of Apache, with different configs in them.  Please help and answer these questions for me.

1. Did you ever have SEF working before trying this?

2. Have you tried commenting out the section?

GRAM

Thanks for the quick response.  Yes, the standard SEF is working fine.  I just now went back and removed the and it started returning 404's for all pages but the root.

User error.  I commented out this line: 

RewriteRule ^(content/|component/) index.php

since I don't have a content directory. If I had put my thinking hat on, I would have just removed the content/|  and moved on.  It works fine on my site but I no longer get the "styled" 404 pages, just the standard apache 404 page.  What problem is this .htaccess trying to solve if you don't mind me asking.

Regards

BoardJIVE wrote:What problem is this .htaccess trying to solve if you don't mind me asking.

See first post.

kenmcd wrote:

BoardJIVE wrote:What problem is this .htaccess trying to solve if you don't mind me asking.

See first post.

Thanks.  Answered that question.  The example he used was exactly the one I was using to test, a .gif.  Doh!

For every single non-existant page on your site, Joomla gets instantiated.  For example, a missing javascript, css, or graphics file contained in your template causes Joomla to load again.  If you are missing one or two files, Joomla ends up loading three to four time.  It drives the server utilization up.

The problem was worse on pre 1.07, as it didn't have the new 404 page, and would then load the template again causing something of a runaway loop type of problem. 

The goal is to let Apache handle true 404 issues and not to be in a situation where Joomla had to take care of it.

Thanks for the follow up!

EDIT NOTE:  You and Ken are fast!  You both got in there while I was typing a response 

GRAM

Steve,

This is good stuff.
Even in testing on my local WinXP Apache server I can see the speed difference.
Once we get the issues resolved, we can ship it with OpenSEF.


Ken

Thanks Ken!

That was the goal 

We have all been hearing the sad stories about overutlized servers, and we need SEF, so it had to get done.

It has been revised about four times since I posted.  Each time refining it a little more.

Some folks have provided some great feedback, and I am sure this is going the right direction.  All of my server loads are down since using it.  I have also installed it on quite a few other domains for friends and customers. 

Good job on RC1 btw...  OpenSEF is really turning out to be a great tool.

Steve

Using this .htaccess breaks an SMF install using SEF. I get a generic 404 error page, I had to remove it.

gram wrote:Good job on RC1 btw...  OpenSEF is really turning out to be a great tool.

All Marko.
Once he took over the J 1.0.x version - progress!
The ongoing bugs are rapidly being fixed.
An OpenSEF 2.0.0 RC2 is in the works with more fixes.
Any input you have is always welcome!

Still some issues on the open-sef.org site SMF forum.
The direct links above now work.
Personal message area works.
I tried to post, it did not work.
The links to forums/boards still throw errors.
http://www.open-sef.org/component/optio ... /board,8.0
http://www.open-sef.org/component/optio ... /board,9.0
etc., etc.
Sent PM to Marko via OpenSEF forum.
Marko had the audacity to require sleep.


Maybe the solution will be to use OpenSEF, your htaccess, and Marko's advanced SEF file for SMF.
We'll get there.

Now I need to try your new code above . . . probably late tomorrow. (fading now)

Goosemoose wrote:Using this .htaccess breaks an SMF install using SEF. I get a generic 404 error page, I had to remove it.

Thanks GM, I appreciate you taking the time.  It looks like we are going to have to work on the SMF SEF to get this all working correctly.

Are you using a bridged SMF, or standalone?  Also, wrapped or integrated?

GRAM

gram wrote:Thanks GM, I appreciate you taking the time.  It looks like we are going to have to work on the SMF SEF to get this all working correctly.

Are you using a bridged SMF, or standalone?  Also, wrapped or integrated?

GRAM

Hi Gram,
No problem. I'm using a bridged integrated version. It's actually a cvs version but it's not too different from the current public release. I just realized today that the reason I dropped google rankings from #2 to over #200 was that 1.0.5 was telling giving 404 errors in the header but still loading the rest of the site. 1.0.7 fixed that but is causing problems with xtdratings now. Your fix didn't fix that though.

I can't wait to get OpenSEF working as well. I actually run 8 forums through my one forum installation so it will be great to start improving my addresses.

Thanks for your work on the htaccess file gram!