Suggested Master .htaccess file

[sog2012]

Here is my .htaccess file. Is this OK?

##
# @version $Id: htaccess.txt 14401 2010-01-26 14:10:00Z louis $
# @package Joomla
# @copyright Copyright (C) 2005 - 2010 Open Source Matters. All rights reserved.
# @license http://www.gnu.org/copyleft/gpl.html GNU/GPL
# Joomla! is Free Software
##


#####################################################
# READ THIS COMPLETELY IF YOU CHOOSE TO USE THIS FILE
#
# The line just below this section: 'Options +FollowSymLinks' may cause problems
# with some server configurations. It is required for use of mod_rewrite, but may already
# be set by your server administrator in a way that dissallows changing it in
# your .htaccess file. If using it causes your server to error out, comment it out (add # to
# beginning of line), reload your site in your browser and test your sef url's. If they work,
# it has been set by your server administrator and you do not need it set here.
#
#####################################################

## Can be commented out if causes errors, see notes above.
Options +FollowSymLinks

#
# mod_rewrite in use

RewriteEngine On

########## Begin - Rewrite rules to block out some common exploits
## If you experience problems on your site block out the operations listed below
## This attempts to block the most common type of exploit `attempts` to Joomla!
#
## Deny access to extension xml files (uncomment out to activate)
#<Files ~ "\.xml$">
#Order allow,deny
#Deny from all
#Satisfy all
#</Files>
## End of deny access to extension xml files
RewriteCond %{QUERY_STRING} mosConfig_[a-zA-Z_]{1,21}(=|\%3D) [OR]
# Block out any script trying to base64_encode crap to send via URL
RewriteCond %{QUERY_STRING} base64_encode.*\(.*\) [OR]
# Block out any script that includes a <script> tag in URL
RewriteCond %{QUERY_STRING} (\<|%3C).*script.*(\>|%3E) [NC,OR]
# Block out any script trying to set a PHP GLOBALS variable via URL
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
# Block out any script trying to modify a _REQUEST variable via URL
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2})
# Send all blocked request to homepage with 403 Forbidden error!
RewriteRule ^(.*)$ index.php [F,L]
#
########## End - Rewrite rules to block out some common exploits

# Uncomment following line if your webserver's URL
# is not directly related to physical file paths.
# Update Your Joomla! Directory (just / for root)

# RewriteBase /


########## Begin - Joomla! core SEF Section
#
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !^/index.php
RewriteCond %{REQUEST_URI} (/|\.php|\.html|\.htm|\.feed|\.pdf|\.raw|/[^.]*)$ [NC]
RewriteRule (.*) index.php
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
#
########## End - Joomla! core SEF Section

<Files .htaccess>
order allow,deny
deny from all
</Files>

<FilesMatch "configuration.php">
Order allow,deny
Deny from all
</FilesMatch>

[sog2012]

Can I comment out this whole section? Redirect olddomain.com to www.domain.com

[sog2012]

I too keep getting 500 Internal Server Error messages.

This is the section which gives me the 500 error.

########## Begin - Automatic compression of resources
# Compress text, html, javascript, css, xml, kudos to Komra.de
# May kill access to your site for old versions of Internet Explorer
AddOutputFilterByType DEFLATE text/plain text/html text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript
########## End - Automatic compression of resources

But if I remove the section above, the given .htaccess works.

What exactly does the section above do?

So shall I keep using this The Master .htaccess file even though I had to remove the above section?

And do I need to add the following lines to this Master htaccess file?

<Files .htaccess>
order allow,deny
deny from all
</Files>

<FilesMatch "configuration.php">
Order allow,deny
Deny from all
</FilesMatch>

[g1smd]

There are a huge number of syntax errors in the original file, as well as many rules which run hundreds or thousands of times slower because of the use of inappropriate or inefficient RegEx patterns.

I have fixed about half of the problems so far, see the edit comments for each change for the reasons.

http://docs.joomla.org/index.php?title= ... on=history

[PhilD]

@g1smd
Thanks for taking on the task of making the file work and work more efficiently. Let us know when you have completed the editing.

[g1smd]

There are some small parts that I simply don't understand what was trying to be achieved. There's no way the code will do anything other than get stuck in an infinite internal rewrite loop and crash the server. When I can work out what it was intended to do, I'll edit it to do that thing.

[sog2012]

There are some small parts that I simply don't understand what was trying to be achieved. There's no way the code will do anything other than get stuck in an infinite internal rewrite loop and crash the server. When I can work out what it was intended to do, I'll edit it to do that thing.

I am confused.

Are you saying that this Master .htaccess written by Nicholas from Akeeba and others is not good to use and contains flaws? I have been using it on my server, so is my website at risk from something?

So what .htaccess should we be using? Just the original one that comes with Joomla?

And what about these following commands, should we include them in the original Joomla .htaccess?

<Files .htaccess>
order allow,deny
deny from all
</Files>

<FilesMatch "configuration.php">
Order allow,deny
Deny from all
</FilesMatch>

[g1smd]

It needs careful tuning to get it to work. Much of the functionality was very inefficiently coded.

Today's edits should see a big improvement in page load time, and less of a drain on server resources from certain malicious requests.

There's a few oddities still to fix, I'll take a look at those in coming days.

Yes, include the "deny" functionality. That keeps people out of important files.

[sog2012]

It needs careful tuning to get it to work. Much of the functionality was very inefficiently coded.

Today's edits should see a big improvement in page load time, and less of a drain on server resources from certain malicious requests.

There's a few oddities still to fix, I'll take a look at those in coming days.

Yes, include the "deny" functionality. That keeps people out of important files.

Thank you. Can you please let me know when you finished the file so we can use it?

I got the current Master htaccess file to work on my server and everything seems to work ok. All I did was replace the domain.com areas to my personal domain website, and I had to comment out the following, because if I didn't I would get server 500 Errors:

########## Begin - Automatic compression of resources
# Compress text, html, javascript, css, xml, kudos to Komra.de
# May kill access to your site for old versions of Internet Explorer
#AddOutputFilterByType DEFLATE text/plain text/html text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript #application/x-javascript
########## End - Automatic compression of resources

Other than that, it seems to work and my website seems to work properly. So can I keep using it? It doesn't seem to loop or crash my server as you hinted may happen.

[g1smd]

Only certain requests might invoke that result, and only for the user that made the request.

No other user would be aware that it was happening, as user sessions are segregated.

Check the server error log and make a note to yourself of any status 500 errors.

[sog2012]

Only certain requests might invoke that result, and only for the user that made the request.

No other user would be aware that it was happening, as user sessions are segregated.

Check the server error log and make a note to yourself of any status 500 errors.

When are going to be able to fix this Master htaccess file so we can all use it? I look forward to it

[ooffick]

Hi,

you might want to check with your hosting provider whether the optional mod_deflate apache module is enabled?

For more information about this Apache Webserver Module please have a look here:
http://httpd.apache.org/docs/2.0/mod/mod_deflate.html

Olaf

[mandville]

Note: can i remind people this is a "suggested" master htaccess file as this topic title says.
it is also a good idea to read through all the posts from leolams opening post to understand how and why it does not work in some cases.

[g1smd]

Yes, it is "suggested" but the coding and logic is far from optimal.

There are a large number of errors.

For example the part matching the pattern \.jpe[g,2]? will match filenames ending with
.jpe
.jpeg
.jpe,
.jpe2

The third one is "dot j p e comma". Really?


Elsewhere, a pattern like \.htm[l]? is more simply coded as \.html?


Slashes do not need to be escaped.


Literal periods in patterns DO need to be escaped.


The [F] flag terminates mod_rewrite processing for the current request so using [F,L] is incorrect.


Additionally ^.*(blah).* matches the entire string and then has to perform hundreds or thousands of back off and retry "trial matches" to find the start of the part that really needs to match. More processor cycles are wasted after the match by having to read to the end of the string even though that data is not going to be captured in a backreference or reused. A pattern like (blah) with no anchoring or extraneous .* fluff will parse from left to right in one go.


There are also simple typos such as this one:
^(images/stories/*\.(jpe[g2]?|jpg|png|gif|bmp|css|js|swf|ico|html?))$
where the pattern can only match a request like
www.example.com/images/stories////////////.jpg
with multiple slashes and no filename before the file extension.

This pattern fixes it, but without resorting to using the .* pattern:
^(images/stories/([^.]+)\.(jpe[g2]?|jpg|png|gif|bmp|css|js|swf|ico|html?))$


THE_REQUEST ends with HTTP/1.1 or similar, so this condition would never evaluate as true:
RewriteCond %{THE_REQUEST} ^.*/index\.php$
Also the ^.* pattern is greedy, promiscuous and ambiguous, and should be avoided.
This new code evaluates in one pass:
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.php\ HTTP/


There's a mix of Redirect and RewriteRule syntax in the file and that can lead to unexpected rule processing order. Rules are processed on a "per-module" basis, so all redirects and rewrites must use RewriteRule (must not Redirect or RedirectMatch) syntax.


Many rules return a 302 redirect, when a 301 redirect is what is actually required. The 302 redirect could literally kill your searchengine rankings.


There are many more examples. The edit history lists those fixed so far.
http://docs.joomla.org/index.php?title= ... on=history

[sog2012]

Yes, it is "suggested" but the coding and logic is far from optimal....

So what's the solution?

Can you edit the master htaccess and release it without all the problems that you have detected?

[g1smd]

I am slowly working on the example code in the docs. It will take a while. Many of the changes have already been actioned. Once complete, those changes will be communicated back to the original author(s).

A fix for the official Joomla release was submitted 6 months ago, and is still awaiting commit. That's already in process.

[PhilD]

@sog2012 and everyone else reading this thread.

The file posted by leolam is from a 3rd party source. Use it at your own risk!! You are responsible for making it work correctly on your site!!

Now let me say that the file is not plug and play, but requires a bit more advanced knowledge. You do have to know something about htaccess, the rules and how it's supposed to work.

I ask that everyone actually READ the comments contained within the file. The file is not without issues.

I also ask everyone to actually READ THE COMMENTS on the file on the developers site.
http://snipt.net/nikosdion/the-master-htaccess You will see there are/were numerous syntax errors and sections that may not work as expected, cause the file to not be fully parsed or generate a 500 error.

Here are a few of the comments from within the file that show the file is cobbled together from others files and also potentially can cause issues with your server or site function:

## Some sections are too picky and may cause problems with legitimate requests.
## You are ultimately responsible for disabling them or writing exception rules.

## Some sections - depending on your server configuration - may cause your site
## to throw 500 Internal Server Error. The only way to figure out which one is
## causing it is trial and error.

## Big thank you's to ............... for sharing their .htaccess rules

There are rules that are very inefficient, can cause infinite loops, and can quickly kill even a moderately loaded site by overloading the server with thousands of unnecessary requests. There are syntax errors that can cause the file to not be fully parsed, possibly leaving your site without htaccess protection that your trying so hard to obtain. You generally won't know if the file did not fully parse unless it is generating a 500 error. Also remember, for every single request the server has to read this htaccess file. A simple page request in Joomla makes many requests per page, the server has to parse the file and act on any rules that are true for every one of those requests before the page can be served. Errors and rule inefficiencies slow this process down.

I also ask that everyone read the comments that g1smd has posted to the thread. He has taken on improving most aspects of the file without being asked to do so. These improvements are both to make the file work as intended and to make the file work more efficiently. This is a big file and research has to be done on some rules as to what the intent of the rule is. Only then can g1smd correct the syntax or make the rule more efficient. I applaud his efforts to make the file more efficient and error free. So lets all give the process some time.

For those that are interested, you can also as g1smd has stated go to http://docs.joomla.org/index.php?title= ... on=history and read the changes he has made so far to the file. When g1smd is finished (and as he has stated) he will make a post to this thread notifying everyone that his edits are complete. The file will still not be 100% plug and play (everyone's site/server install is different), but it will work as expected and be more efficient.

[g1smd]

Yep. That about sums it up.

[sog2012]

Are there any generalized commands everyone can use in their .htaccess file that do not need any editing or personalizing, and that will work for everyone?

For example, I added these commands which I believe it's safe to use for everyone:

<Files .htaccess>
order allow,deny
deny from all
</Files>

<FilesMatch "configuration.php">
Order allow,deny
Deny from all
</FilesMatch>

Are there any others we can include?

[g1smd]

The simple answer is that you can include whichever bits you feel you need.

Just be aware that every bit of code you add slows down the servicing of requests for every page, every image and every stylesheet by another tiny amount. All those tiny amounts add up.


The file will never be a simple cut and paste solution, not least because there are multiple places in the file where you have to change example.com to be whatever your domain name is.

See also PhilD's post above.

[sog2012]

The simple answer is that you can include whichever bits you feel you need.

Just be aware that every bit of code you add slows down the servicing of requests for every page, every image and every stylesheet by another tiny amount. All those tiny amounts add up.


The file will never be a simple cut and paste solution, not least because there are multiple places in the file where you have to change example.com to be whatever your domain name is.

See also PhilD's post above.

I actually did a copy and paste pretty much. All I did was change domain.com to my personal domain and comment out "AddOutputFilterByType DEFLATE text/plain text/html text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript" as that was giving me Server Error 500.

But everything else seems to work and my website seemed to work properly.

But after reading g1smd's comments, I remove the htaccess and went back to using the Joomla htaccess.

I will wait for g1smd's new version of htaccess. Even though I do not know him, I really trust in g1smd's knowledge in this area.

[kurchania]

@sog2012,
you need to go through your server logs of hosting provider to give you list of error log.
some hosting dont like this syntax of htaccess

Code: Select all

RewriteRule ^(.*)$ - [R=404,L]

that time you need to replace it by

Code: Select all

RewriteRule ^(.*)$ - [F,L]

also don't copy the whole code at once.copy it by line by line and try to diagnose on which syntax you get that error.
also if you ask this thing to your hosting provider he will surely help you much faster way than us.
Regards

[sog2012]

@sog2012,
you need to go through your server logs/b] of hosting provider to give you list of error log.
some hosting dont like this syntax of htaccess

Code: Select all

RewriteRule ^(.*)$ - [R=404,L]

that time you need to replace it by

Code: Select all

RewriteRule ^(.*)$ - [F,L]

also don't copy the whole code at once.copy it by line by line and try to diagnose on which syntax you get that error.
also if you ask this thing to your hosting provider he will surely help you much faster way than us.
Regards

The whole htaccess runs and works properly on my host. Except for "AddOutputFilterByType DEFLATE text/plain text/html text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript". That part gives Server Error 500. So I just commented that part out.

But everything else works and my website loads rather fast too.

[kurchania]

@sog2012,
may be mod_deflate is not installed in your server.
may be your apache version is 2.2.17. AddOutputFilterByType is deprecated method in apache 2.1.
right now stable version of apache is 2.2.17.
people use mod_filter instead of this.we create filter chain for that.
but this is off topic thing i guess.
and without looking at server log its hard for me to tell the exact problem.

But everything else works and my website loads rather fast too.

good for you .

[g1smd]

you need to replace it by

Code: Select all

RewriteRule ^(.*)$ - [F,L]

No. I specifically pointed out that the correct code is

Code: Select all

RewriteRule .* - [F]

[F] terminates mod_rewrite processing for the current request. [L] is not required.
No need to create any backreference. It is not going to be used.

[g1smd]

But after reading g1smd's comments, I remove the htaccess and went back to using the Joomla htaccess.
I will wait for g1smd's new version of htaccess.

The new version is already visible at: http://docs.joomla.org/Htaccess_examples_%28security%29
It incorporates all of the changes listed at: http://docs.joomla.org/index.php?title= ... on=history

There are a few more changes yet to be made.

It would be useful for people to report any problems that occur with the code as it exists right now.

[kurchania]

@g1smd

[F] terminates mod_rewrite processing for the current request. [L] is not required.

if forbidden page is custom joomla page or php page than we have to mention that this is the last rule for htaccess.L Tells Apache to not process any more RewriteRules if this one was successful.thats why sometime we can write L also.
is this a wrong practice?
and if your logic is correct that also mean

[R=301,L]

should be replaced with
[R=301]

[g1smd]

For [R] redirects, the [L] is required and makes the redirect happen immediately. No further rules are processed for the current request as soon as the [L] is processed.

For [F], sending the Forbidden response automatically terminates all further processing for the current request. That is, [L] is implied and it is not necessary to state it in the rule. Stating it changes nothing in the way the rule is handled, just wastes processor cycles doing nothing.

[kurchania]

@g1smd
now that's pretty frustrating.joomla inbuilt htaccess.txt write the same code.

Code: Select all

# Send all blocked request to homepage with 403 Forbidden error!
RewriteRule ^(.*)$ index.php [F,L]

you are the first person of my life telling me this code is not for good practice.
but yes i will learn from my mistakes and will try to implement your way of forbidden redirection.

[g1smd]

Simply the [L] is not required. The [F] automatically does the [L] internally without having to be told.

The new code:

Code: Select all

# Return 403 Forbidden header and show the content of the root homepage.
RewriteRule .* index.php [F]

error redirection

This is not a redirect. The error is returned at the currently requested URL (exactly as it should).