Log in

No account? Create an account
To The Side

Improving Web Security: Six Ways the Apache.org JIRA Attack Could Have Been Prevented by Better Code

Today it was revealed that servers at Apache.org and Atlassian were successfully attacked, leading to thousands of stolen passwords. The attack on apache.org's servers was via JIRA, and since the attack on Atlassian came from the same source, it probably was also through JIRA.

I'm sure that JIRA's programmers feel embarrassed enough about all of this--I don't want to berate them, or insult them. Everybody makes mistakes; almost all software has some pretty bad security vulnerabilities at at times. Overall, the JIRA guys seem to do good work, and seem to be generally nice people. And on top of all of that, I understand how they feel! Whenever there's a reported security issue in Bugzilla, I freak out. Thankfully there hasn't been an attack on Bugzilla like this JIRA one in recent memory. But if there was an attack like this, I'd be absolutely mortified, and the last thing I'd need would be somebody trying to insult or attack me for simply having made a mistake.

Instead, I want to use this opportunity as a reminder to all web application developers for why web application security is so important, and talk about some of the things that we do in Bugzilla that would have prevented or mitigated an attack like this one, and that web applications should probably all do as standard practice:

  • Lock down the on-disk permissions of files and directories. When you install Bugzilla, the actual installation script makes sure that the permissions on Bugzilla's files and directories are as secure as possible. That way, even if there is a security compromise in Bugzilla, the attackers can't upload programs and run them, modify existing scripts, or generally do anything nasty to the machine. It's particularly important that web applications never allow anything to be uploaded into a location where the web server could execute it.

    This is actually something that I rarely see web applications stress, in any of their documentation. Some web applications recommend that system administrators fix permissions themselves, but the chance is that the vast majority of people installing your software are going to skip the optional security recommendations, and just go for whatever's easiest. The only way to guarantee that security happens right on every installation is to have the actual installer do the setting of the permissions.

    The attackers configured the Apache JIRA to allow uploads into a location where the webserver would execute files, which is what let them compromise Apache's servers and steal the passwords of every JIRA user who logged in to the system. If, like Bugzilla, it was impossible to configure JIRA in that way, that part of the attack would have been impossible.

  • Httponly: Never allow Javascript to read the login cookie. This is one of the simplest and most effective protections you can make in a web application. Seriously--for Bugzilla, it was just a few lines of code, and eliminated a whole set of possible attacks. All you have to do is to set an extra attribute on cookies when you send them, and you gain a lot of security. If Javascript must read some of your cookie data, that's fine, just don't let it read the login cookie.

    If Httponly had been set on the Apache JIRA session cookie, then the cross-site scripting attack that the attackers used could not have stolen administrators' login privileges.

  • Open-source your software. Okay, look, I know that that's not practical or possible for everybody. But I will tell you, a lot of the security bugs that are found in Bugzilla are found by people we've never met who just happened to be reading the code. These users find all our security issues before they're ever exploited, and so we can release fixes before systems are harmed. In particular, I don't think there has ever been a successful Cross-site scripting attack performed on a Bugzilla--at least not any publicly discussed in the six years I've been working on Bugzilla.

    The "many eyes make all bugs shallow" maxim may not always be true, but for security issues, in my experience, it has absolutely held up.

    If the cross-site scripting vulnerability in JIRA had been found by an outside user before it was exploited, the Apache JIRA administrators would have been safe from it.

  • Have automated tests scan your code for potential security issues. There are lots of ways to do this. In Bugzilla, we have an automated test that makes sure that we properly "filter" any data that we got from the user or the database before displaying it on a web page, so that people can't inject malicious HTML or JavaScript into our system. The automated tests don't always catch our security issues, but the number of times that I've fixed a security issue in my code thanks to the tests is uncountable--probably in the thousands, at this point. And those are fixes that happen before the code even gets checked in, so that's a security vulnerability that gets fixed before it even becomes a part of the product.

    There are lots of other ways to do automated security testing of code, these days. Static code analysis, Fuzz testing, and automated security scanners seem to be the most popular, from what I've seen.

    If the cross-site scripting vulnerability in JIRA had been found by automated tests before it was exploited, the Apache JIRA administrators would have been safe from it.

  • Lock out users who fail to guess their password too many times. There are lots of approaches to account security, but this is one of the simplest and most failsafe. If an attacker can only guess five passwords every 30 minutes, and then they get locked out, the statistical probability that they will ever guess anybody's password is pretty slim. Starting with Bugzilla 3.6, we implement exactly that policy, and we even notify the Bugzilla administrators whenever somebody gets locked out, so that if there's a large brute-force password attack, the admins will know immediately.

    Some people say that the answer to password security is to have people change their passwords every three months. This is probably sensible on some systems, but on a web application, it's mostly pretty ridiculous. If you only change your password every three months, then that gives an attacker three months to guess your password. I can promise you that almost any normal user password could be guessed in that time, particularly if the system doesn't prevent brute force attacks. Then once the user has your password, they can usually do everything damaging that they want to do within a few minutes. So, almost any forced-rotation period is pretty silly, in a pure web application. (In other systems it can make sense--it all depends on the context.)

    Other people suggest that passwords need to be a certain level of complexity or a certain length, and up to a point, that's true. If your password is one of the 100 most-common passwords, then even with a sensible lockout policy, the attacker will eventually guess it, if they keep up over a few days. (Of course, in Bugzilla, the system administrators would see all of these lockout notices and probably stop the attack pretty quickly. Still, it's better to be safe than sorry.) So your application should probably enforce a level of password complexity that's sufficient to make it impossible to guess passwords when combined with your lockout policy.

    If the Apache JIRA had had brute-force password-guessing protection like Bugzilla's lockout method, the attackers would not have been able to discover administrators' passwords using that method. (My understanding is that newer versions of JIRA do have this protection.)

  • Store passwords securely. If you're going to store a password in the database or anywhere, store it using some standard, secure method. Don't just hash the password--you have to at least salt them. Preferably, don't even invent your own password-storage scheme--just use some library that already exists.

    Never store passwords as plain text. You might be saying to yourself, "Oh, nobody will ever break into the system and steal them." That sounds pretty good until somebody does break in and steal them, and then you'll really be wishing that you stored them properly.

    If the Apache JIRA had been storing passwords properly, then Apache.org's users would be at far less risk of the attackers now knowing all the passwords in JIRA.

And finally, on top of all those points, if you're a system administrator, upgrade your software regularly. Some of the security issues that let the Apache JIRA be compromised were supposedly fixed in newer versions of JIRA, before the attack ever happened. Almost every time I hear of an attack like this, it uses old, known problems to compromise the system.

Nobody likes getting attacked. Everybody feels bad about it, when it happens--system administrators, programmers, and most especially users. So let's just design secure applications to start with, and never have any of our system administrators or users have to bear the burden of compromised systems and stolen data.



Speaking as a Bugzilla developer, Bugzilla has been around for 12 years now. Most of these things Max points out above actually got learned by the Bugzilla developers the hard way over the last decade, and the knowledge has been passed along over the years. Hopefully sharing some of these things can help other projects not run into the same problems in the future.



Everybody makes mistakes... thanks, Max, we are definitely taking our lumps right now (and deservedly so). We will be releasing more information about the attack on Atlassian, soon. fyi

- Jon Silvers, Atlassian

Re: jira

Hey Jon. You're welcome. I'm looking forward to hearing more about what happened at Atlassian--the educative process of disclosure in situations like this can sometimes be beneficial enough to the world at large to almost make up for the damage done.



One more thing ...

Nice article. You missed one extremely important thing though: Always educate your users not to use any of 'love', 'sex', 'secrete' or 'god' as their passwords. Everybody knows that Hackers (http://www.imdb.com/title/tt0113243/quotes?qt0448608) know these are the most common passwords ;)

Re: One more thing ...

Hahahaha. Yay Hackers! :-D



Re: One more thing ...

Hmm... secrete... kinky!


My Gmail account attempted

Interestingly at 4:43AM AEST time I got an SMS and an email to my secondary email account asking for my password to be reset. Its sounds like the hackers got my gmail address and tried to access my gmail.. Thanks for the article.

Re: My Gmail account attempted

Hmm. Not sure if that was related or not, but you're welcome.


HttpOnly is weak

I appreciate the constructive tone of this article! But I gotta take issue with one of the recommendations here.

You write: "[HttpOnly] is one of the simplest and most effective protections". Baloney. It's simple, sure. But effective it is not. In most cases, anything an attacker can do without the HttpOnly flag, the attacker can do with the HttpOnly flag, at the cost of more work. It's a speedbump, not a robust protection.

Re: HttpOnly is weak

It's not a useless protection, at all, though, which is what saying that it's "weak" implies. I'd like to see somebody steal a Bugzilla login cookie with an XSS (though there aren't any of that I know of), with Httponly on.



Re: HttpOnly is weak

I'd just like to know how to steal a HttpOnly cookie by using XSS only, is it possible?


JIRA source is not hidden

All commercial licenses ($10 and up) get access to the source code. It's pretty open already.

Re: JIRA source is not hidden

But the thing is, that doesn't encourage people to look over the code--it doesn't get enough people looking at it to get the sort of effect that we have with Bugzilla. Anybody can look over the code of Bugzilla, even if they haven't bought it, so there's a lot more opportunity to get reports about security issues from people.


Replace password auth with information cards/OpenID

The best recommendation for protecting passwords in web apps is not to use them. Use information cards and/or OpenID instead.

The fact that even super-users at Apache can't use a password system securely shows (once again) that we should be moving away from passwords as fast as we can.

Would be great if this incident spurred high-profile open source people like Bugzilla and Apache to get serious about Identity 2.0.

Re: Replace password auth with information cards/OpenID

Don't know why I wrote "and/or OpenID." OpenID doesn't get rid of passwords.

I should have noted this since my OpenID provider (myopenid.com) requires a password. But they do have the "nice" feature that if I just click the "forgot password" link and put in my email address, I can log in without needing to know my password. I can just use the link sent in an email. So can anyone who sniffs that email, but, hey, at least it's convenient. Sigh....