Joomla! Discussion Forums

That is definitely something we're going to look at in the following days, both on an organizational and technical level.

It would also be great if the Joomla! team could share the log findings so those with scan tools can add these attack sigs to rule sets. 
It is very possible the attack sigs are all ready in many of these products as many of these attack scripts use well known methods.

I read this thread as it unfolded, yesterday. Now that the problem is behind us, I'll post my thoughts on this.

I appreciate the Joomla! team *not* sending fear inducing security alerts about this issue without first fully understanding the cause and resolution. It had to be difficult to stay focused on a sensible course with all the pressure being placed on the team to overreact. Sending out a note that states "Joomla.org was defaced and you might be next!" is analogous to yelling "Fire!" in a crowded theater. The team took the responsible, appropriate and logical path.

Mention was made of Drupal's email lists, suggesting that they communicate this type of situation immediately. I use Drupal, too, and I follow their development mailing list closely. Both projects deploy similar approaches to issues, such as this. Those responsible to develop solutions for the broader community work together with sincere interest, they consider the problem, diagnose the cause, devise solutions, and then they communicate what must be done. We must be sensitive to not pit one open source project against the other, even unintentionally, most especially in times of challenge.

As it turns out, the panic people were feeling was unjustified. Thank goodness pressure did not cause the Joomla! team to spread panic to thousands and thousands of others. It wasn't core and it wasn't a third party extension anyone else uses. The vulnerable component was never distributed. It was right to move this topic to Sites and Infrastructure because it was a Joomla! org problem, exclusively. 

We can only respond to the information we have at the time. Let's all let the process work, next time, and stay out of the way. Question our own suspicions. Why would Joomla! org do things to harm the project? They are smart. They care. They are certainly doing the best they can. Force yourself to think about the other perspective. Consider if your post is helpful or just another voice of an angry, hostile mob.

The way things turned out demonstrates Joomla! org *does* know what they are doing. Perhaps we could extend a bit more trust and appreciation for the excellent response and commitment the team has given us, time and time and time, again.
Amy

If you have a library of attack signatures, the one that compromised the shop is definitely in there.  It is very basic which is part of the reason that it is so unfortunate.

Noted.
Can you share the exploit they used?

I don't think this optimism and flower-handing for bringing the official sites back online, as well as blaming one setting and a non-distributed component, will fix and explain the defacing of 200+ other Joomla sites defaced in the last hours by the same folks...
Sorry to not see this as such a great day as the rest here, and still waiting to see if my site will get attacked too or not (I'll be SO happy to provide logs if it happens)...

I told you it was basic.

On that we totally agree Joe. Getting hacked (and recovering) is not fun at all, a necessity, and frustrating. In a situation like this, it's being glad to have found the cause and just deal with the consequences.

It's a blind systematic scan on a given site for the inclusion of php code, basically using the same method that was used in last summer's flow  of (mostly abandoned) extensions.

While I appreciate your perspective on the situation, I see it differently. When I visited the "mothership" sites - store, help site, or main site - and saw they had been compromised, I was concerned that the same thing might happen to sites I own and/or manage one of which is regularly accessed by tens of thousands and has been a long-term paying client of mine. I checked them all and found they were, at least at that point, and thankfully still, operating as they should. I also checked to ensure I had done everything I could think of to ensure these were as secure as possible as well as informing my ISP - also a Joomla user - of the announcement that was eventually posted here.

I wouldn't characterize my concerns as either a) panic, or b) unjustified, nor was my intent - I made the original Drupal comment - to pit one CMS against another, rather to set out an alternative approach to disseminating information.

I appreciate the fact that members of the Joomla team didn't overreact, did their due diligence and let us know,ultimately, what happened. I personally think it would also be helpful to get an advice, when there's a possibility of a problem, of that possibility. I'm confident that at least for my sake, I can handle that without "panicking."

And, as I said before, I sincerely appreciate the efforts of everyone involved in coming up with the solution, and letting us know in the end.

Cheers
Chris Hutcheson

Did we ever say that it would fix or explain the other sites?  We investigated our problem because that is the problem that we were capable of investigating.  Joomla! powered websites get cracked all the time, I'm sure you have seen the security forums.  Sometimes, they come in waves like this, sometimes they are isolated events.  I have been dealing with Joomla! Security for a long time and I have seen the waves come and go.  This is the world we live in.

I think it is quite likely that these crackers are using some sort of automated scanning/probing scripts to find vulnerable websites.  It is not hard to find information about known vulnerabilities and assembling them into a catalog would be child's play.  The reason that we got cracked is likely because the shop was vulnerable to the exact same attack vector that OpenSEF was.  The exploit is exactly the same.

I have seen a lot of exploits, a lot of bad code, a lot of interesting tools and a lot of crackers during my involvement in this project.  That is how I first became involved in this project.  We cannot be responsible for securing all of the Joomla! installations that are out there in the wild.  We cannot be responsible for ensuring that the owners of those sites follow best practices.  We cannot be responsible for ensuring they keep their software up to date.  That is their responsibility and their burden.  It is the price of using a powerful piece of software.  That is the trade off; you except it or you don't.

I see nothing optimistic or flower-handing about this situation.  I didn't sleep last night Joe.  I got some sleep this morning after a ridiculously long day of driving and working on getting servers back in order and making sure this was not a core exploit.  No one is handing me flowers, no one is handing any of us flowers ... there is really no reason for you to even say that.  We all take this very seriously.



Did you even read the announcement I posted?  I stated plainly for everyone reading that the blame rested wholly and squarely on the shoulders of the entire Joomla! core team.  This was human error.  It was us slipping up on security in one instance on one site.  It should serve as an example for everyone to maintain diligence.

While we are at it ... what about all the Joomla! sites that were defaced last month ... why stop at the last hours.  What about the ones that were defaced last year?  Are you going to blame us for that too?  Are we to fix those sites as well?  This is yet another wave in the childish and vain world of site cracking/defacing.  If you want to vent your anger and frustration I suggest you vent it in a direction where it might do some good.



I don't see this as a great day at all.  I see this as a day that I would rather not have even woken up ... just like yesterday.  The very last thing I ever wanted to do was send out an announcement like the one I sent out at 5am this morning after not sleeping for nearly 24 hours. 

As for logs, it is the same as every other site where these people are scanning.  There is nothing unique or interesting about it.  They used a very simple yet in this case effective exploit on the $mosConfig_absolute_path variable.

Louis

This I understand, Chris, and you were here receiving the most up-to-date information possible. But, how many hundreds of thousands of Joomla! site owners were not here? How many did not see the "mothership" site cracked? Why does it make sense to alert them before it is clear what happened and how it can be fixed?


Chris - again, I do *not* believe Drupal would send out a note without understanding the problem and resolution. I do not believe you have given an alternative.


What would that note say? "Joomla org has been defaced and you might be next, but we don't know what happened or how to fix it?" Give that a bit of thought.

If we have a good backup and recovery system in place, then, we don't have to be overly worried. We have done what can be done.

I do understand your concerns, Chris. I just don't think you are fully considering your proposal and the unintended consequences. Anyway, enough, already. I trust you meant no harm.

Amy

Louis, I'm neither angry nor frustrated, just worried about my site being subject to the same kind of attack, as this has happened before, so I know what it's like from personal experience.
I am also not looking to blame anyone, I just do not think it is siple coincidence joomla.org AND all those other sites get defaced at the same time, but as implied, using different vulnerabilities. Generally those guys use one script, which when successfully tested gets thrown at lots of sites.
So my question was (and I know you obviously can't have the answer): were all those sites using the same vulnerable extension? Did all of them have the same unsecure setting? If there IS a simple exploit on $mosConfig_.., what's the fix or countermeasure?
All the replies to the announcement I read were flower handing...
I don't mean to diminish the work you guys did on fixing this here, only it won't help any of the other Joomla users out there, some of them I am sure without any clue about what is posted in the security advisor thread who had their site set up by someone else and just add their content... Of course you are also not responsible for their sites, but it just looks bad due to the official sites (more than one, on more than one server as far as i remember..?!) getting attacked right at the same time. This is also the "why stop at the last hours": publicity.

Me too... and it always happens at the worst possible time.



Joe, the Joomla.org sites are under constant attack every day all day.  This time they just so happened to hit a site when it was vulnerable.

The same attack vector is used on LOTS of different vulnerabilities.  I am quite aware of what their processes are and we in fact mentioned that already in this thread.  The vulnerability that existed in our shop code can exist in any number of extensions.  I don't think its coincidence that all of these sites are being hacked at the same time, but they vulnerabilities are different.  The exploits are potentially the same, but the vulnerabilities are different.



The htaccess file included with the core has measures that stop the use of mosConfig variables in the request.  Unfortunately for us it was not used on the shop site in recent days, thus the vulnerability.  Also, turning register globals off also keeps things locked down much better.  That was also not done on the shop site ... I stated this in the forum announcement.

Other than that, due diligence is the solution to the problem.  There is no security silver bullet, you just do the best you can.



You are incredibly cynical Joe ... expressing thanks translates into flower handing?  *shakes head* 



One server ... and all the sites on that server were gotten to because the file system was compromised.  Don't make this into more than it was, it doesn't do any good for anyone.  I feel for anyone who has their site defaced, but they are responsible for the care and feeding of that site.  That is the unfortunate reality of owning a website, either you know how to deal with issues or you have someone on payroll that does.  If you know that we are not responsible for that then why do you bring it up?  To point out something negative?

I don't know what you mean by "why stop at the last hours" publicity but we have done all we can do.  If you want to help others then please do so, but we have told you how our sites were compromised and have apologized for it as well as alerted everyone via the announcements forum.

It appears that you are not going to be happy with that, and that is unfortunate ... but that is what we have done.  We will get the rest of our stuff online as time allows.  If you just want to continue bickering about it then I leave you to it... its not a good use of my or anyone else's time.

Louis

Hi Rob,



I am not trying to start a flame war, or spread fud, but I would liek to make an observation.

Being able to set $mos* from Register Globals/Reg Emulation is security bug in the core.

are Register Globas evil? not necessarily.  I can bit a bad programmer, but not a good one (yes, everyone makes mistakes).

How do you avoid being hurt by RG? pre-initialization of every used global solves the problem.

If a badly written component needs to reset a global variable to expose a bug, this shouldnt matter if this global had being pre-initialized.

As previous said, this was 1.0.4 and this RG might not be a issue anymore, but in anycase, the component cant be blamed alone for the security hole, the core had its part in it too.

-rsd