Suggested Master .htaccess file

PierreB · Post by **PierreB** » Wed May 11, 2011 10:56 pm

Thanks g1smd...analytical as always...

I will be checking the diffs for changes in code I am interested in...

If the proposed htaccess file is a convergence at present of your own and Nikosdion's thinking, do you keep a public space for your very own brand of code concerning the htaccess file?

In one of your links concerning one version of the htaccess file, I read
"- Version 3.2 wasn't tested and killed some sites"
...so I guess it's not universally and necessarily true that newer code will always be an improvement over the previous one...
It is true, however that in general it will work better...

Thank you very much

g1smd · Post by **g1smd** » Thu May 12, 2011 12:03 am

Nikosdion published version 2.3 (which identified as version 2.2 at the top of the file and as 2.3 further down the file) in November 2010. That file was at snipt as well as on a part of the Joomla Docs page.

I made about 80 changes to that Joomla Docs file over a period of several weeks. Those changes are documented on the Joomla Docs page in the edit comments. Once I was sure the code was fairly robust and almost complete, I kicked off a discussion at: http://snipt.net/nikosdion/the-master-htaccess/ but Nikosdion immediately rejected most of the proposed changes. That was very surprising, given the detailed explanations I had noted against each change.

After that I presented another updated version tagged as 2.4.1 to Nikosdion for review. That file is on the Joomla Docs page, is also found at snipt, and is in my SVN repository (linked from the post a few posts back). His response to that was to produce the 3.0 file on snipt (now copied at:) http://snipt.net/g1smd/joomla-master-ht ... 2011-03-28 but it still contained very many errors. Discussion continued at http://snipt.net/nikosdion/the-master-htaccess/ and briefly moved to http://snipt.net/g1smd/joomla-patch/ and over the next few weeks he slowly added many of the changes I had suggested but also introduced a few more typos on the way. All of that code is in his GIT repository linked from the post a few posts back. The new code traverses versions 3.0, 3.1, 3.2 and the beginnings of 3.3.

Over that same time period, I progressed my version of the code from 2.4.0 to 2.4.9 after taking comments in this thread into account as well as also incorporating some of the changes that Nikosdion was making at his end. Those new versions are on the Joomla Docs site.

My SVN repository at code.google.com contains both codesets listed in roughly date order. My updates are tagged as version 2.4.x and the Nikosdion code is tagged as version 3.x.x but with the respective GIT commit IDs also added.

If the proposed htaccess file is a convergence at present of your own and Nikosdion's thinking, do you keep a public space for your very own brand of code concerning the htaccess file?

Yes, it's at code.google.com in SVN, but it is usually at least one version behind what I have on my local PC. That's certainly the case at the moment, as I have more changes to send to Nikosdion once he has actioned the previous suggestions. There are also several pages over at codereview.appspot.com where I have set up a commented DIFF comparing my code with his at various stages.

In one of your links concerning one version of the htaccess file, I read
"- Version 3.2 wasn't tested and killed some sites"

Version 3.2 went through 6 iterations (3.1 went through 12) and contained a number of errors. Some of them were new errors that Nikosdion had introduced as typos and then it took him a while to spot. There were some errors that he said were not errors and refused to fix. The file also contained some patterns that were too restrictive and some of that was my fault. As the exact URL patterns they were supposed to match were not documented, I failed to allow for a filename with multiple periods within. I have since fixed that error in my code using efficient RegEx patterns, but Nikosdion took the "easy but inefficient route" in 3.3 by going back to using the reviled (.*) pattern in the middle of several of those regular expressions.

In SVN my code is tagged as version 2.4.x and the Nikosdion code is tagged as 3.x.x with the respective GIT commit IDs appended.

PierreB · Post by **PierreB** » Thu May 12, 2011 2:24 am

Very explicative and informative, @g1smd

The home page for the joomla-master-htaccess also helped me understand the relations between the files and the various editions
http://code.google.com/p/joomla-master-htaccess/

PierreB · Post by **PierreB** » Thu May 12, 2011 11:29 am

Just a quick question or remark, if you will

- Question:
If one wishes to use the "Block bad user agents" and "Other useful settings" code, where should one insert it within the htaccess file?

- Remark:
The documentation page does not specify where the above code should be placed and if it should be placed in some specific order at all with regard to the master htaccess file code.

Cheers

Joe Crawford · Post by **Joe Crawford** » Thu May 12, 2011 3:08 pm

Perhaps this is the right direction for the file. From a security standpoint maybe the starting point should be a file that closes as many holes a possible and also shuts down all functions other than basic site administration and (unregistered user) page display. This could then be followed with a detailed description of each section of the file, what operational functions it may be disabling and what the exposures are if that section is removed. This way the default (recommended) state of the system is that of maximum security rather than one of maximum function with minimum security. It will then be up to the user to chose those functions he wants his/her website to provide but also to know the security exposures of enabling those functions.

Webdongle · Post by **Webdongle** » Thu May 12, 2011 3:19 pm

Joe Crawford wrote:.... This could then be followed with a detailed description of each section of the file, what operational functions it may be disabling and what the exposures are if that section is removed. .. It will then be up to the user to chose those functions he wants his/her website to provide but also to know the security exposures of enabling those functions.

Agreed but perhaps not in the .htaccess file itself ? It is well notated for those who understand it. Perhaps a separate Tutorial ?

Webdongle wrote:.... What a fantastic resource it would make if Tutorials were written that described in depth exactly what each part of the code did. Newbies could read the Tutorial and have an understanding of which parts of the code would enhance the security of their site.

Joe Crawford · Post by **Joe Crawford** » Thu May 12, 2011 3:32 pm

Webdongle,
I saw your previous recommendation on the Tutorial and basically agree. I was merely suggesting that the approach be changed from a default state of minimum security and maximum function that requires the user to overtly "enhance the security of their site", to one of maximum security where the user knows what the exposures are for each website function he enables and each section of the file he chooses (or is required by his host config) to remove/disable.

g1smd · Post by **g1smd** » Thu May 12, 2011 6:38 pm

The blocking code would go somewhere near the beginning of the .htaccess file.

The general order of logic for mod_rewrite code found in the .htaccess file should be:
- stuff that blocks access is listed first (and before any redirects, as it is pointless redirecting a request only to then block it),
- external redirects are listed next, these are listed in order from most selective to most general (this ensures that any non-canonical request reaches final destination in one and only one hop, avoiding any and all redirection chains),
- internal rewrites are listed last, as these map the externally requested URL to an internal server filepath (internal rewrites must be listed after external redirects to avoid exposing the rewritten filepath back out on to the web as a new URL).

If you use RewriteRule anywhere in your site, do NOT use Redirect or RedirectMatch at all. The mix of code from the two different Apache modules can cause problems.

PhilD · Post by **PhilD** » Sat May 14, 2011 12:00 am

Joe Crawford wrote:Webdongle,
.......... I was merely suggesting that the approach be changed from a default state of minimum security and maximum function that requires the user to overtly "enhance the security of their site", to one of maximum security where the user knows what the exposures are for each website function he enables and each section of the file he chooses (or is required by his host config) to remove/disable.

You can not have a default 'maximum security' using htaccess for Joomla installations. It is just not practical. To do so would prevent many from having successful installations. To complicate matters, throwing a list of 'foreign' options to decide on during initial installation at someone not familiar with these options, would quickly make Joomla unpopular when something is enabled breaks the installation. Go read some of the earlier posts here in this thread and you will see the difficulty people have with understanding very very basic htaccess configuration. There is also the consideration of an existing default htaccess within the public_html directory that if altered may cause issues with existing site setups and existing software's. These are some reasons why ( I imagine) the included Joomla htaccess file is not enabled by default and that the file contains only a few relatively 'safe' rules.

The htaccess file is specific to each server/site setup and should be considered as only a part of an overall security setup. In fact, htaccess is going to do nothing to prevent many common attacks through a vulnerable extension, out of date Joomla core, or an install that is vulnerable in certain other ways.

g1smd · Post by **g1smd** » Sat May 14, 2011 9:43 am

Re: https://www.akeebabackup.com/support/fo ... tml#p45656

sog wrote:2011-04-07: Thank you Nicholas. I was rather use YOUR .htaccess as I trust you more than others.

Thanks for your support, especially after bugging me over and over again in this thread and via PM.

You watched more than 100 changes go through here. What do you think those were all for?

PhilD · Post by **PhilD** » Sat May 14, 2011 5:05 pm

The blocking code would go somewhere near the beginning of the .htaccess file.

The general order of logic for mod_rewrite code found in the .htaccess file should be:.......

Maybe as a suggestion, you could put comments in the file such as:
########## Add optional bad user agent blocking code in this area
#
########## End add optional bad user agent blocking code

and elsewhere provide similar comments to denote where other code should be placed for those wishing to add their own code.

g1smd · Post by **g1smd** » Sat May 14, 2011 8:05 pm

That's a good idea; and while looking into doing that I have found there's more errors with the overall logic.

Joe Crawford · Post by **Joe Crawford** » Sat May 14, 2011 10:39 pm

PhilD - Well, I guess I’m just ‘old school’. I’d rather have a new user complain about a few functions not working out-of-the-box than have their brandi-new on-line business site get hacked every so often, orders lost and their customer data stolen until they learn about security the hard way.

PhilD · Post by **PhilD** » Sat May 14, 2011 11:19 pm

Joe I actually agree with what you are saying, just stating it won't happen and some reasons as to why.
One issue is "...and their customer data stolen until they learn about security the hard way.".

If you collect customer data you are now in a whole different realm with a whole different set of federal rules and card issuer rules to comply with. This is beyond the scope of what most have experience with. Anyone collecting certain customer data should hire a competent firm to make sure their business website and server environment complies with the laws and regulations currently in effect; both federal and card issued.

ugcman · Post by **ugcman** » Sun May 15, 2011 5:29 pm

I think this is a very important discussion and improvements for Joomla and I hope/suggest that one or a group of Joomla core devs would be responsible for updating this certified "Joomla master .htaccess" file continuously when needed.

Its so important for securing Joomla and should be part of all old and new Joomla installations.

PhilD · Post by **PhilD** » Sun May 15, 2011 6:33 pm

There is nothing "certified" or "official" about the Suggested Master .htaccess file. It is only a suggestion by a small group of forum users to use this file (or any part of it) on your site. Use this file only if you want to and at your own risk. Improper application will break your site.

A small group of knowledgeable forum users decided to make a patchwork file that was put together by others actually work as intended, made it more useful and easier to understand. To my knowledge it will not ever be incorporated into Joomla due to a variety of reasons, some of which have already been posted.

As for security, the file is but a very small part of your overall site security and will do little or nothing to stop certain exploits that are due to vulnerable extensions.

g1smd · Post by **g1smd** » Wed May 18, 2011 8:35 am

PhilD wrote:Maybe as a suggestion, you could put comments in the file such as:
########## Add optional bad user agent blocking code in this area
#
########## End add optional bad user agent blocking code

That was a great idea and is now done.

While thinking about exactly where to put such comments, I noticed more errors in the file. These are to do with the order the rules are processed. It turns out that some blocking rules never get to run for certain requested URLs.

I have started re-arranging the order of the rules so that the most specific rules are first and the more general rules are last. Additionally, rules that use RewriteRule - [L] for exceptions now appear as far down the page as possible and will likely be converted in the near future to instead use a negative match RewriteCond.

As ever, the code needs testing, especially requesting various URLs and query strings that should be blocked to make sure they really are blocked. I have updated my SVN repository, the Joomla Docs page and informed (by way of patches submitted to Git) the original file author.

C0nw0nk · Post by **C0nw0nk** » Sat May 21, 2011 2:15 pm

Ive noticed that even with the master .htaccess in place you can still access joomla's log files. for example.

www.domain.com/error_log

Post by **mandville** » Sat May 21, 2011 2:33 pm

C0nw0nk wrote:Ive noticed that even with the master .htaccess in place you can still access joomla's log files. for example.

http://www.domain.com/error_log

two points and let us be precise in the terms we use

[*] this topic is about a suggested master htaccess file and not the master htaccess.txt file provided with joomla
[*] it would be helpful if you quote the code in the file that prevents this action

I also suggest you read the last posts by phild and g1smd

orev · Post by **orev** » Wed Jun 01, 2011 7:33 pm

Using the R40 release from code.google. I am dismayed that this file does not protect the configuration.php file from such a simple attack. I am far from a security expert, but it was trivial to realize that accessing the files via "configuration.php/123", as the regex is anchored to end of line. This can easily be fixed by changing:

Code: Select all

-RewriteRule ^(htaccess\.txt|configuration\.php(-dist)?|php\.ini)$ - [F]
+RewriteRule ^(htaccess\.txt|configuration\.php(-dist)?|php\.ini) - [F]

True this "attack" does not actually allow one to download the config file, but if that is the logic then it wouldn't need to be protected at all as the php code would always execute and be blank anyway.

So far I am not impressed at all with Joomla security. Non-web related files should not be inside the web directory at all. Logs, tmp, etc... all of this stuff should be outside web accessible directories. Thanks to you guys who are at least making an attempt to mitigate it.

PhilD · Post by **PhilD** » Wed Jun 01, 2011 10:05 pm

While I'd prefer to defer this discussion to g1smd, as he is the lead on this project, I see there are some other lines with the end of line anchor. should some of those also be looked at?

g1smd · Post by **g1smd** » Wed Jun 01, 2011 11:35 pm

I am at a loss to know how a request for example.com/configuration.php/123 will result in the server mapping that request to the configuration file; unless mod_speling or accept_path_info or similar is inadvertently enabled.

The r40 code is the latest available at present and I'll certainly revisit the end anchoring problems again when I find the time. Rule order is the biggest source of flaws in the code at present.

orev · Post by **orev** » Thu Jun 02, 2011 2:39 am

This is a result of the default Apache/PHP behavior of accepting pathinfo for scripts. http://httpd.apache.org/docs/2.2/mod/co ... ptpathinfo

PhilD · Post by **PhilD** » Thu Jun 02, 2011 12:45 pm

Ok. Interesting. I see how the change works and makes the rule operate as intended.

C0nw0nk · Post by **C0nw0nk** » Sat Jul 02, 2011 10:40 pm

Just to help out I've found some new rules that can be used to block in XSS attacks.

Code: Select all

RewriteEngine On  
RewriteCond %{QUERY_STRING} ("|%22).*(>|%3E|<|%3C).* [NC]  
RewriteRule ^(.*)$ log.php [NC]
RewriteCond %{QUERY_STRING} (<|%3C).*script.*(>|%3E) [NC]
RewriteRule ^(.*)$ log.php [NC]  
RewriteCond %{QUERY_STRING} (javascript:).*(;).* [NC]  
RewriteRule ^(.*)$ log.php [NC]  
RewriteCond %{QUERY_STRING} (;|'|"|%22).*(union|select|insert|drop|update|md5|benchmark|or|and|if).* [NC]
RewriteRule ^(.*)$ log.php [NC]
RewriteRule (,|;|<|>|'|`) /log.php [NC]

g1smd · Post by **g1smd** » Mon Jul 04, 2011 1:39 am

The multiple .* patterns will force tens of thousands of "back off and retry" trial match attempts in the pattern matching for every request hitting the server. The .* construct is greedy, promiscuous and ambiguous.

Unless the .* group is the final item before the $ end anchor and is being captured for re-use, the .* group is the WRONG thing to use. That is, don't use .* at the beginning or in the middle of a RegEx pattern.

I'm thinking that ("|%22).*(>|%3E|<|%3C).* should be replaced by something like ("|%22)[^>%<]*(>|%3E|<|%3C) or similar.

Likewise (javascript:).*(;).* should be replaced with javascript:[^;]+; or similar.

C0nw0nk · Post by **C0nw0nk** » Mon Jul 04, 2011 9:20 am

I suppose it would also help g1smd if it was not depending on a log.php file and would just go straight to a 404 error.

g1smd · Post by **g1smd** » Mon Jul 04, 2011 9:56 am

Rewriting to a script returns a "200 OK" response, unless the log.php script is designed to send a 4XX header.

I would use RewriteRule ^pattern$ - [F] for most requests. The [L] flag is specifically not required when the [F] flag is used.

C0nw0nk · Post by **C0nw0nk** » Mon Jul 04, 2011 10:44 am

So something like this maybe.

Code: Select all

RewriteCond %{QUERY_STRING} javascript:[^;]+; - [F]

PhilD · Post by **PhilD** » Mon Jul 04, 2011 11:15 pm

"The .* construct is greedy, promiscuous and ambiguous."
Can we have it arrested?

I would rather not see anything changed/added that presents a 200 ok when it should present a 4xx error.

The Joomla! Forum™

Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file

Re: Suggested Master .htaccess file