To The Side

HTTP Response Splitting: A Common, But Frequently Unknown, Security Issue

Recently, there were two security issues discovered in Bugzilla that would fall under the category of HTTP Response Splitting attacks. Although this is a common issue in web applications, many developers are unaware of it, its consequences, or how to protect their applications from it.

In short, here's what you need to know:

Never allow unsanitized user data into HTTP headers.

Most of the time, if you're using some HTTP framework, it should be handling this for you. So mostly you just need to make sure that you have a good HTTP library that security-checks its header output properly. But if you're printing headers directly to the user, as plain text (as I've seen many PHP apps do), you do need to be concerned about this.

In particular, it's most important that you never allow any character that might be considered a "newline" into your HTTP headers. To understand why, and what could be dangerous about this, you need to know a little bit about HTTP. Let's say that we're sending a response to the user that looks like this:

200 OK
X-My-Header: some_user_data
Content-Type: text/html; charset=UTF-8

<html><head><title<Hello World!</title><body></body></html>

Okay, pretty simple. This is a page that has "Hello World!" as the title, and no content. Harmless. However, what's that X-My-Header: some_user_data there? Let's imagine that some_user_data can be anything that the user input, and he inputs a string that looks like:

some data;
Content-Type: text/html; charset=UTF-8

<script type="text/javascript">all_your_base();</script>

Now, your response looks like:

200 OK
X-My-Header: some data;
Content-Type: text/html; charset=UTF-8

<script type="text/javascript">all_your_base();</script>

Content-Type: text/html; charset=UTF-8

<html><head><title<Hello World!</title><body></body></html>

And I could have inserted anything! It's not just like a normal Cross-Site Scripting vulnerability, where I can only insert a small dangerous piece of HTML into an already-existing page. I could insert any HTML, any script, or any content at all. That content could be an ActiveX control for Internet Explorer, it could be some other content that your system is already configured to open automatically--anything. I'm not limited just to HTML or JavaScript. There's also a good chance that any Set-Cookie headers will become a part of the body instead of being part of the headers, too, so I can steal cookies even if they're marked "httponly".

For the most part, this vulnerability is prevented if you never allow people to insert a CR or LF character into headers. There's no good reason to allow inserting those characters directly into a single header, so you should be pretty safe to just completely deny those characters in HTTP headers generated by your application.

To The Side

VCI 0.7.1

Thanks to the Bugzilla VCS Extension, interest in VCI has really been picking up lately, and so has development. I just released VCI 0.7.1. The biggest feature of 0.7.1 is that every Commit object now has a "uuid" accessor--a way of uniquely identifying a Commit, universally (across all Repositories and Projects), even if the VCS does not normally provide such a mechanism. This makes it easy to tell if two different Commit objects are actually the "same" commit. (For example, if you have two branches with very similar histories, the uuid should help tell you which commits in those two branches are in fact identical.)

To The Side

VCI 0.6.1

Well, I just uploaded VCI 0.6.1 to CPAN! It's been a long time since I've done a major VCI release, but I'm working on some code that uses it heavily now, so there's a good chance that will be many more improvements to VCI coming along soon. :-)

As of this release, I feel confident in saying that VCI is no longer "alpha-quality software", and it should be pretty much ready for prime-time production use.

There are some important API changes in this release, most notably that the "moved" accessor of a Commit now contains objects instead of strings. (See the Changes file for details.)

VCI now differentiates between "revision numbers" and "revision identifiers" for Commits, where the first one is a human-readable number for a commit, and the latter is a unique string identifying the Commit. As part of this change, the "revision" accessor for a Commit in the Bzr driver now returns a revision ID instead of a revision number.

There are a few other nice new features and improvements, too!

For the next major release, I'd like to implement a "parents" accessor for Commits, that gives you all of a Commit's "parent commits". In centralized VCSes like CVS and Subversion, this is just the commit that came immediately before this one. But in decentralized VCSes like Git, Mercurial, and Bazaar, one commit could have multiple "parents", some of which might not be accessible via the normal revision history otherwise.

After that, there might be some performance improvements to be made, but I'll feel that the read-only side of VCI is pretty much done. Then comes support for working directories and the ability to commit to a repository!

Right now what I'm thinking is that you'll get a WorkingDirectory object via a call to VCI->wd(type => 'Foo', path => 'path/to/something'). This may involve some re-working of how the internals of work. Also, I'm going to make it possible to call "checkout" on a Project, providing a path, and create a WorkingDirectory that way.

To The Side


For many years, I had a very hard time sleeping--a harder time than anybody else that I knew. However, I discovered the "secrets" of how to sleep, and ever since then, as long as I apply them consistently, I sleep well. The "secrets" of sleep are:

  • One must generally eat well. In simple terms, this means complete proteins at every meal and a decent amount of vegetables. One should research which foods or combinations of foods provide complete proteins. As far as vegetables go, if I don't have a salad every other day, my sleep schedule can start to get wonky. The fresher the vegetables, the better they work. (I've always had a particularly good result with the salad bar at Whole Foods, although one doesn't have to be that extreme to have the vegetables "work".)
  • It's important never to let your body reach the point of starvation for a long period, during the day. In my experience, this seems to modify the body's systems in a way that makes it difficult to sleep at night, even after you've handled the starvation.
  • I often have trouble sleeping if I don't get off the computer three hours before bedtime.
  • If it is getting close to bed time and I'm still on the computer or my attention still feels "fixed", then I have to do the process described in the chapter "Exhaustion" in The Problems of Work by L. Ron Hubbard. If I don't do that process and I was on the computer later than I should have been, or my attention was "fixed" on something, then I often don't sleep well or at all.
  • I take a set of vitamins every day that helps with my sleep and general energy. It's the set described in Clear Body, Clear Mind by L. Ron Hubbard. In particular, once a day I personally take:
    • 10,000 IU Vitamin A
    • 250mg Vitamin B1
    • 500mg Vitamin C
    • 400 IU Vitamin D
    • 800 IU Vitamin E
    • A 100mg B Complex tablet (It's particularly important that it contain folic acid)
    • Two multi-mineral tablets
    • Sometimes, I then also take an additional 100mg or 200mg of Niacin, and it seems to help a lot with sleep. But if it's working, it will almost certainly cause you to turn red and itch for some period of time, so you want to take it well before you're actually going to go to bed. (The turning red and itching is shorter if you exercise right after taking the vitamins, to get your metabolism up.) Also, some people get a sort of "stomach burn" from Niacin, but I think that's because they weren't well-fed (or didn't eat enough vegetables) before taking the Niacin.

    I've also found that it's important to take them before 6pm, if I plan to fall asleep around 12am. The vitamins can sometimes increase my energy levels and make it hard to sleep if I just took them. However, it's also important not to take it too early, because then the effects will be gone by the time I go to sleep. So taking them with a meal between 3pm and 6pm is usually the best.

  • Every night, about an hour before bed, I drink a Cal-Mag Solution. The particular one I linked to there tastes decent, if you're concerned about the taste.
  • If I have "racing thoughts" or nightmares, I take extra Vitamin B1 (between 100mg and 500mg) and that handles it pretty quickly.
  • If I am lying in bed and I'm not tired, I get out of bed. One of the worst things you can do is try to stay in bed and sleep when you're not tired enough to sleep. What happens, at least for me, is that I then never get tired enough to sleep, because I'm lying in bed.
  • If you are on the computer and it's very late at night, you might think that you aren't tired when actually you are. There's something about the computer that causes people to not be aware of how tired or hungry they are, as long as they are on it. Try walking away from the computer for about 10 minutes or so, if you're concerned about this.
  • In order to sleep, the body needs to produce a hormone called melatonin. To produce melatonin, your body needs at least iron and B vitamins. It also needs tryptophan, but that's just a part of protein, so if you're eating enough protein, you should have enough tryptophan. For me, if I take tryptophan supplements, I'm tired all through the next day, so I don't take them.

    I generally don't recommend taking melatonin supplements, because that's an attempt to replace the body's normal melatonin production. This means that for a little while, the supplement will work excellently, but it will work less and less well over time. Also, once you stop taking the melatonin, you will have an even harder time falling asleep than you used to, for a few days. (For me, I have a really hard time falling asleep on the third night after I took the melatonin.)

  • It's somewhat important to drink enough water throughout the day. I have a hard time sleeping if I'm dehydrated. This point is less important than some of the other, above points, though. Note that one must have adequate salt and potassium in one's system in order for the body to absorb water properly.
  • If you can, try not to fall asleep in the middle of the day. Try to only sleep at night. Sleeping during the middle of the day can throw off your sleep schedule.

Many people have also told me that exercise is important to sleeping, but I haven't found it necessary. Exercise is certainly important to living, though. And I have found that it helps with sleep, to have actually used one's body a fair amount during the day, as long as one doesn't strenuously exercise in the last few hours before bed.

If one does all of the above, one should generally be able to sleep at night. There are a few other things that can prevent sleep, though. If you are doing all of the above and still can't sleep at night, tell me what's going on in the comments and I'll see if I can help.


To The Side

The Room

Bob: Hey Andy.

Andy: Hey Bob.

Bob: So, can I come in the room?

Andy: Oh, well, were you born in the room?

Bob: Born in it?

Andy: Yeah.

Bob: Um, no, I was born in L.A.

Andy: Oh, I'm sorry, you can't come in the room then.

Bob: Oh, but my girlfriend's in the room.

Andy: Well, you can come in and visit her for three hours.

Bob: And then what?

Andy: And then you have to leave.

Bob: Okay...can I come back after those three hours?

Andy: Maybe.

Bob: Maybe? Maybe depending on what?

Andy: Well, depending on how we feel about it.

Bob: "We?" Who's "we"?

Andy: Well, you know, us, the door guards.

Bob: I only see you.

Andy: Well, there's a lot of us, believe me.

Bob: Okay. So there's no way I can come in the room at all?

Andy: Well, there's a raffle. You could sign up for the raffle and maybe you'll win.

Bob: Um...okay...what are my chances of winning?

Andy: Well, there's only one winner.

Bob: Okay. But how many total people enter?

Andy: About 10 thousand.

(Bob looks somewhat flabbergasted.)

(A man in a doctor's coat comes up.)

Andy: Hello.

Doctor: Oh, hello. Can I come in the room?

Andy: Well, were you born in the room?

Doctor: Well, no, but I'm a doctor.

Andy: Oh, okay, you can come in.

(Doctor goes into the room.)

Bob: Wait, why could he go in there?

Andy: Well, he was a doctor.

Bob: But I can't go in there, because I'm not a doctor?

Andy: That's right. And you weren't born there. But you could win the raffle.

Bob: I have a 1 in 10,000 chance of winning.

Andy: Well, yes. But if you win, you get to come in the room. I mean, after some paperwork.

Bob: Paperwork?

Andy: Well, there's about a month's worth of paperwork to do to get in the room if you win the raffle. But the good news is that you only have to pay $5000 to get in, then.

Bob: $5000?? What for?

Andy: Well, for us to read the paperwork, of course.

(Bob is once again flabbergasted.)

(A very serious man in a business suit comes in.)

Andy: Hello.

Businessman: I need to get into that room right away.

Andy: Well, I'm sorry, but you can't go into this room unless you were born here.

Businessman: I'll give you $100,000 to let me into that room right now.

Andy: Okay, go ahead.

(Businessman pulls a pre-written check out of his pants and gives it to Andy.)

Andy: Thanks! Have a nice time in the room!

(Businessman enters the room.)

Bob: What the fuck. Did that guy just bribe you to let him in?

Andy: Oh, no, it's part of the rules. If you give me $100,000, I can let you into the room. Do you have $100,000?

Bob: No.

Andy: Ah, well, then I don't think I can let you into the room.

Bob: So, being born in there, winning the raffle, being a doctor, or bribing--

Andy: Gifting.

Bob: --gifting you $100,000 are the only ways that I can come into that room?

Andy: Well, if you want to stay for more than three hours, yes.

Bob: Those are the only ways?

Andy: Well, you could marry somebody who lives in the room.

Bob: Marry somebody?

Andy: Well, yes, that's what I said. Perhaps you should marry your girlfriend. Then you could come into the room.

Bob: But I've just met her!

Andy: Well, why don't you go visit her for a little bit and see if you want to marry her.

Bob: For just three hours?

Andy: Well, no, if you're going to marry her you can visit her for six hours. But you have to propose in the first three hours.

Bob: How could anybody possibly know that they want to marry somebody if they can only visit them for three hours?

Andy: Well, clearly that is not my problem.

Bob: Well, whatever. (pause) Can I send my girlfriend flowers?

Andy: Well, certainly, you just have to buy them inside of the room.

Bob: But I can't go in the room!

Andy: Well, you could send a letter into the room, or write an email. Or you could call a flower seller.

Bob: Okay, fine, whatever. I can't buy flowers outside of the room and then bring them in?

Andy: Well, yes, but there's a door charge.

Bob: A door charge?

Andy: Sure, about 15%. You're going to sell the flowers, right?

Bob: No, I'm just going to give them to my girlfriend!!

Andy: You're bringing flowers into the room, but you're not selling them?

Bob: No!!

Andy: Well, I'll have to inspect the flowers.

Bob: Inspect the flowers?

Andy: Yes, to make sure that you're not bringing in any illegal flowers.

Bob: Illegal flowers?

Andy: Well, the room has certain rules about what you can bring in if you're not selling it.

Bob: (frustrated noise). So what if I was selling flowers?

Andy: Well, then there would be different rules. And you'd have to pay the door charge.

Bob: The door charge?

Andy: Yes, about 15%.

Bob: 15% of what?

Andy: The value of the flowers.

Bob: How do you know the value of flowers?

Andy: Well, there's rules. We have a lot of rules in the room.

Bob: But my girlfriend is in there!

Andy: Well, you could propose by text message.

Bob: (blank stare) (pause) And then what, I have to do a month's worth of paperwork and pay you $5000 to come into the room?

Andy: Exactly! You're a smart person. See how it all makes sense?

Bob: Fuck you. I'm going home.

Andy: Bye! We hope to see you again some time. It's always nice here, in the room.

To The Side

Why Is Mozilla Legitimizing Your Mom?

This is a very serious question, and it seems a number (at least 0) of people agree:

Why is Mozilla legitimizing your mom?

Mozilla clearly relies on your mom for services like doing the occasional laundry and making dinner when you go over to her house, and yet clearly your mom stands for everything that Mozilla doesn't. I mean, let's look at the points:

  • Your mom is clearly closed-source.
  • Your mom only has proprietary, biological interfaces.
  • The traditional, uniform, standards-based lifeform is the monocell. Your mom is practically the opposite of a monocell.
  • Mozilla provides most of its services over the Internet, but your mom's services are not available over the Internet. In fact, your mom's services aren't even publicly available.

I wonder what the Mozilla Foundation could do with the funds that your mom has received?

So, the question is: why is Mozilla legitimizing your mom?


P.S. This is a joke that, for the most part, only people subscribed to mozilla.governance will fully get.

To The Side

RPC::Any - Simple, Effective XML-RPC and JSON-RPC for Perl

I just uploaded the first stable release of a new library to CPAN, called RPC::Any. It's a simple server framework that presents an object-oriented, unified interface for both XML-RPC and JSON-RPC. It has the following features:

  • You can use both JSON-RPC and XML-RPC with the same backend code. (That is, you only have to write your functions once--you never have to do anything special for each individual protocol in your code.)
  • It fully supports Unicode, proven with extensive Unicode tests in the test suite.
  • If it gets tainted input, it will send your functions tainted parameters.
  • It can take input from STDIN, from a string, or from an HTTP::Request.
  • Everything is written as a series of methods whose names and parameters will stay stable, so it's very easy to subclass for your own application, and make it behave completely differently.
  • It gives your functions the ability to call a type() method to explicitly type return values correctly for the protocol in use. (So you can do $class->type('int', $some_value) within your WebService methods and have that value properly returned as an int instead of a string.)
  • It doesn't try to start its own server process or send its own output, so you can control exactly how it is used and how the output is sent.
  • It supports operation in a CGI environment (in addition to almost any other environment you can come up with).
  • The test suite of RPC::Any has nearly 100% (about 98%) code coverage over the entire library.

More or less, it's everything that I ever wanted in an RPC library, and had hacked into Bugzilla using some very complex code (more on that below). It's also extremely simple to use, for most cases:

use RPC::Any::Server::JSONRPC::HTTP;
my $server = RPC::Any::Server::JSONRPC::HTTP->new(
  dispatch => { 'Foo' => 'My::Methods' });
print $server->handle_input();

package My::Methods;
use strict;

sub hello {
  my ($class) = @_;
  return $class->type('string', 'Hello World!');

The server above will read from STDIN, parse HTTP headers and JSON-RPC input, and return proper HTTP headers and JSON-RPC output (for any version of JSON-RPC, no less! 1.0, 1.1, or 2.0). Sending JSON-RPC input with Foo.hello as the method name would call My::Methods->hello() and return the string "Hello World!", explicitly typed as a string. You'll notice that My::Methods calls $class->type even though it never defines a "type" method--that's a magic method that RPC::Any gives the class immediately before it calls the method (and removes after the method is done being called).

To get XML-RPC support instead of JSON-RPC support, just replace every instance of the string "XMLRPC" with "JSONRPC" in the above code (so, in two places) and you have an XML-RPC server instead! No changes whatsoever required to My::Methods or any other code.

A Little Background

So, you might be wondering--why did I write a whole new WebServices module when there are so many options available on CPAN? Well, in fact, I'm actually using two of them as the backends for RPC::Any--RPC::XML and JSON::RPC::Common. So I'm not re-inventing the wheel for RPC--I'm just giving it all a clean, consistent interface.

Here's the whole story behind what motivated me to start writing RPC::Any:

Many years ago, for Bugzilla 3.0, we added a WebServices interface to Bugzilla. When we started off, it was pretty simple. We were using SOAP::Lite for XML-RPC, and everything seemed to be working okay.

Then, we realized that Unicode support was broken in SOAP::Lite (at the time), and so we had to write a workaround for that. Then we wanted to change how we dealt with null values, and we had to write a complex workaround for that. Then we changed how we dealt with input values--more complex, hackish workaround code. Oh, and then we realized that we wanted to preserve taint on incoming values, for security in Perl's taint mode (because otherwise we might accidentally send unchecked input to the database).

All of this was really tough, because SOAP::Lite was basically a SOAP module, with XML-RPC support tacked on. So all our workarounds involved modifying very convoluted SOAP code, when all we really wanted to do was modify how our XML-RPC support worked. Our XML-RPC Server module became quite a monster. (You might ask--okay, so why not switch to a pure XML-RPC module from CPAN? Ah, I tried--but more on that later.)

And then, we added JSON-RPC support, and we wanted to do it with the same backend code--that is, I didn't want to maintain one whole Bug.get function for JSON-RPC and another, separate Bug.get function for JSON-RPC. I just wanted to have one Bug.get function, with two separate frontends.

Well, this was far, far harder than I anticipated. The biggest problem was that SOAP::Lite called methods like Bugzilla::WebService::Bug->get(@params), but JSON::RPC called methods like $server->Bugzilla::WebService::Bug::get(@params);. (So under XML-RPC, the first argument to our methods was the name of our class, like "Bugzilla::WebService::Bug", but under JSON-RPC the first argument was an instance of JSON::RPC::Server.) So in order to be able to explicitly type return values using $self->type within our WebService code, I had to have one "type" implementation in Bugzilla::WebService::Server (which was the base class for both our XML-RPC and JSON-RPC servers) and another "type" implementation in Bugzilla::WebService (which is the base class for our WebService itself--that is, the base of things like Bugzilla::WebService::Bug and so on).

Then, there were numerous bugs in JSON::RPC--so many that I practically ended up writing my own JSON-RPC implementation on top of JSON::RPC, just to get it working properly for Bugzilla. Oh, and also, remember, I had to do all the same things I did for XML-RPC above, like get taint mode working right and testing/fixing Unicode. Oh, and then we added HTTP GET support to our JSON-RPC interface (which JSON::RPC didn't really support natively) and then we added JSONP support to it. It was crazy--just look at the old Bugzilla::WebSevice::Server::JSONRPC.

So along then comes the day when I have a bit of free time, and I'm looking for a project. I think, "Oh, I know, I've always wanted to move away from SOAP::Lite for our XML-RPC service. Let's see what else I can find on CPAN!" So I looked around at all of the available options, and I found RPC::XML, which, though not exactly what I needed, it was well-maintained, XML-RPC specific, and the maintainer had been really helpful with my requests.

So, I started to write a patch to move Bugzilla from SOAP::Lite to RPC::XML...and started to write a whole other series of workarounds--this time not for bugs (because the RPC::XML maintainer was very responsive about fixing those) but just because RPC::XML worked in a different way than we needed it to work, for Bugzilla.

Finally, at this point, I decided it was time to take the engineer attitude and solve the problem the right way--that is, not just fix the problem with our XML-RPC server, but fix all architectural problems of our WebServices by refactoring.

I looked around CPAN to see if there were any good, unified RPC frameworks that supported both the XML-RPC and JSON-RPC protocols, and didn't find anything. So, I rolled up my virtual sleeves and put the time in to do it really right--to write a module that would be the best, simplest, and most powerful RPC framework available for XML-RPC and JSON-RPC. One that fit the (very complex) needs of the Bugzilla Project, but could be used by anybody. A few weeks later, and I had RPC::Any.

It's really been a surprising amount of work, particularly in the test suite. But I think it was worth it--the new Bugzilla WebServices code is way cleaner, simpler, and actually much more powerful than it was before the RPC::Any.


To The Side

Improving Web Security: Six Ways the JIRA Attack Could Have Been Prevented by Better Code

Today it was revealed that servers at and Atlassian were successfully attacked, leading to thousands of stolen passwords. The attack on's servers was via JIRA, and since the attack on Atlassian came from the same source, it probably was also through JIRA.

I'm sure that JIRA's programmers feel embarrassed enough about all of this--I don't want to berate them, or insult them. Everybody makes mistakes; almost all software has some pretty bad security vulnerabilities at at times. Overall, the JIRA guys seem to do good work, and seem to be generally nice people. And on top of all of that, I understand how they feel! Whenever there's a reported security issue in Bugzilla, I freak out. Thankfully there hasn't been an attack on Bugzilla like this JIRA one in recent memory. But if there was an attack like this, I'd be absolutely mortified, and the last thing I'd need would be somebody trying to insult or attack me for simply having made a mistake.

Instead, I want to use this opportunity as a reminder to all web application developers for why web application security is so important, and talk about some of the things that we do in Bugzilla that would have prevented or mitigated an attack like this one, and that web applications should probably all do as standard practice:

  • Lock down the on-disk permissions of files and directories. When you install Bugzilla, the actual installation script makes sure that the permissions on Bugzilla's files and directories are as secure as possible. That way, even if there is a security compromise in Bugzilla, the attackers can't upload programs and run them, modify existing scripts, or generally do anything nasty to the machine. It's particularly important that web applications never allow anything to be uploaded into a location where the web server could execute it.

    This is actually something that I rarely see web applications stress, in any of their documentation. Some web applications recommend that system administrators fix permissions themselves, but the chance is that the vast majority of people installing your software are going to skip the optional security recommendations, and just go for whatever's easiest. The only way to guarantee that security happens right on every installation is to have the actual installer do the setting of the permissions.

    The attackers configured the Apache JIRA to allow uploads into a location where the webserver would execute files, which is what let them compromise Apache's servers and steal the passwords of every JIRA user who logged in to the system. If, like Bugzilla, it was impossible to configure JIRA in that way, that part of the attack would have been impossible.

  • Httponly: Never allow Javascript to read the login cookie. This is one of the simplest and most effective protections you can make in a web application. Seriously--for Bugzilla, it was just a few lines of code, and eliminated a whole set of possible attacks. All you have to do is to set an extra attribute on cookies when you send them, and you gain a lot of security. If Javascript must read some of your cookie data, that's fine, just don't let it read the login cookie.

    If Httponly had been set on the Apache JIRA session cookie, then the cross-site scripting attack that the attackers used could not have stolen administrators' login privileges.

  • Open-source your software. Okay, look, I know that that's not practical or possible for everybody. But I will tell you, a lot of the security bugs that are found in Bugzilla are found by people we've never met who just happened to be reading the code. These users find all our security issues before they're ever exploited, and so we can release fixes before systems are harmed. In particular, I don't think there has ever been a successful Cross-site scripting attack performed on a Bugzilla--at least not any publicly discussed in the six years I've been working on Bugzilla.

    The "many eyes make all bugs shallow" maxim may not always be true, but for security issues, in my experience, it has absolutely held up.

    If the cross-site scripting vulnerability in JIRA had been found by an outside user before it was exploited, the Apache JIRA administrators would have been safe from it.

  • Have automated tests scan your code for potential security issues. There are lots of ways to do this. In Bugzilla, we have an automated test that makes sure that we properly "filter" any data that we got from the user or the database before displaying it on a web page, so that people can't inject malicious HTML or JavaScript into our system. The automated tests don't always catch our security issues, but the number of times that I've fixed a security issue in my code thanks to the tests is uncountable--probably in the thousands, at this point. And those are fixes that happen before the code even gets checked in, so that's a security vulnerability that gets fixed before it even becomes a part of the product.

    There are lots of other ways to do automated security testing of code, these days. Static code analysis, Fuzz testing, and automated security scanners seem to be the most popular, from what I've seen.

    If the cross-site scripting vulnerability in JIRA had been found by automated tests before it was exploited, the Apache JIRA administrators would have been safe from it.

  • Lock out users who fail to guess their password too many times. There are lots of approaches to account security, but this is one of the simplest and most failsafe. If an attacker can only guess five passwords every 30 minutes, and then they get locked out, the statistical probability that they will ever guess anybody's password is pretty slim. Starting with Bugzilla 3.6, we implement exactly that policy, and we even notify the Bugzilla administrators whenever somebody gets locked out, so that if there's a large brute-force password attack, the admins will know immediately.

    Some people say that the answer to password security is to have people change their passwords every three months. This is probably sensible on some systems, but on a web application, it's mostly pretty ridiculous. If you only change your password every three months, then that gives an attacker three months to guess your password. I can promise you that almost any normal user password could be guessed in that time, particularly if the system doesn't prevent brute force attacks. Then once the user has your password, they can usually do everything damaging that they want to do within a few minutes. So, almost any forced-rotation period is pretty silly, in a pure web application. (In other systems it can make sense--it all depends on the context.)

    Other people suggest that passwords need to be a certain level of complexity or a certain length, and up to a point, that's true. If your password is one of the 100 most-common passwords, then even with a sensible lockout policy, the attacker will eventually guess it, if they keep up over a few days. (Of course, in Bugzilla, the system administrators would see all of these lockout notices and probably stop the attack pretty quickly. Still, it's better to be safe than sorry.) So your application should probably enforce a level of password complexity that's sufficient to make it impossible to guess passwords when combined with your lockout policy.

    If the Apache JIRA had had brute-force password-guessing protection like Bugzilla's lockout method, the attackers would not have been able to discover administrators' passwords using that method. (My understanding is that newer versions of JIRA do have this protection.)

  • Store passwords securely. If you're going to store a password in the database or anywhere, store it using some standard, secure method. Don't just hash the password--you have to at least salt them. Preferably, don't even invent your own password-storage scheme--just use some library that already exists.

    Never store passwords as plain text. You might be saying to yourself, "Oh, nobody will ever break into the system and steal them." That sounds pretty good until somebody does break in and steal them, and then you'll really be wishing that you stored them properly.

    If the Apache JIRA had been storing passwords properly, then's users would be at far less risk of the attackers now knowing all the passwords in JIRA.

And finally, on top of all those points, if you're a system administrator, upgrade your software regularly. Some of the security issues that let the Apache JIRA be compromised were supposedly fixed in newer versions of JIRA, before the attack ever happened. Almost every time I hear of an attack like this, it uses old, known problems to compromise the system.

Nobody likes getting attacked. Everybody feels bad about it, when it happens--system administrators, programmers, and most especially users. So let's just design secure applications to start with, and never have any of our system administrators or users have to bear the burden of compromised systems and stolen data.


To The Side

Dear CSS,

I love you, man. I really do. You're like a brother to me. But you and me and these web developers here need to sit down and have a little intervention with you.

I know that you've go a lot on your mind right now, but there's some things that we need to talk about with you, dude.

First I just want to say that I love your styles. That's not what we're here to talk about, but it's something I really wanted to give you some kudos for. font, font-style, border, text-decoration, color, background, you are awesome. Those things are so good, I just wanna hug them, most of the time. text-decoration and I have had our differences, but I've learned to work around them, and I'm pretty good now.

And the cascade? The cascade is pretty sweet.

But CSS...your positioning sucks. I know it, these web developers here know it, and it's something that we really need to talk about. It's been happening for a long time, and all of us here really would like you to think pretty hard about bringing some changes into your life.

It's not a big overhaul or anything. It's just a few things that we'd really love to see. There's just a few things. Everybody here really loves you man. We mean it. There's just a few small, little things that would really help make all of us happier, and maybe they're things that could help you feel better about yourself, too:

  • All I want for Christmas is the ability to position a box relative to another box that is not the container. Like, say, something like: position: relative-to #other_box top-right; top: 0; left: 0;, and then the top-left of my box would be aligned to the top-right of #other_box. Then I would never have to wrangle with float ever again, except when I wanted to have some text spill around a box.
  • Then while we're at it, I'd really love to be able to specify the size of a box (either horizontal or vertical) as being identical to another box. Maybe something like: width: #other_box and height: #other_box. Tables do this. They did it in 1996. Now it's 2010, and I think that this is something we all would love, but without the tables. So it's something for you to think about; it would really help us out.
  • Since we're talking about height, it would also be awesome to have a way to say "this should be the height of its container but no larger". You'd think height: 100% would do that, but no, that seems to make the box the height of its content, at least the last time I tried it (yesterday). That's OK, though, we love you even though you're eccentric, CSS. I think we just need a little something different, here.
  • I can't say how much I'd love to be able to vertically center something in a box. Some of the less-informed members of this little sit-down here might think that vertical-align has something to do with vertically aligning things in boxes. But no, no, it doesn't. It was a bit of a trick that CSS pulled on us, and we understand that there were some reasons, so we're annoyed, but understanding.
  • While we're at it, wouldn't it be nice to have a single, simple way to center anything horizontally in its containing box? margin: 0 auto usually works. I mean, if you wrap the contents in a table so that they auto-size. (That's a whole other issue, though, possibly solved by display: table?) Sometimes you have to use text-align: center, though. That's fun, but not really what we meant. Then sometimes there's no way at all, which isn't so fun.
  • If you felt really nice, letting me specify z-index on non-positioned elements and having it actually work would be really awesome. That's just if you feel like it, though. I know that there's a lot of other stuff to think about, here, and and I want you to take your own time to do it, and think about it for yourself.

Those are all the things that we'd need. I know that life is pretty difficult for you right now, what with CSS3, HTML5, quirks mode, standards mode, specs, working groups, modules, errata, and everything. So this isn't something I expect overnight. But it's time for us to say that there's millions of people that you're affecting, with your behavior, and we've given you so much love. Couldn't you just give us a little?

For Serious,