Max Kanat-Alexander

HTTP Response Splitting: A Common, But Frequently Unknown, Security Issue

Thu, 11 Nov 2010 04:19:14 GMT

Recently, there were two security issues discovered in Bugzilla that would fall under the category of HTTP Response Splitting attacks. Although this is a common issue in web applications, many developers are unaware of it, its consequences, or how to protect their applications from it.

In short, here's what you need to know:

Never allow unsanitized user data into HTTP headers.

Most of the time, if you're using some HTTP framework, it should be handling this for you. So mostly you just need to make sure that you have a good HTTP library that security-checks its header output properly. But if you're printing headers directly to the user, as plain text (as I've seen many PHP apps do), you do need to be concerned about this.

In particular, it's most important that you never allow any character that might be considered a "newline" into your HTTP headers. To understand why, and what could be dangerous about this, you need to know a little bit about HTTP. Let's say that we're sending a response to the user that looks like this:

200 OK
X-My-Header: some_user_data
Content-Type: text/html; charset=UTF-8

<html><head><title<Hello World!</title><body></body></html>

Okay, pretty simple. This is a page that has "Hello World!" as the title, and no content. Harmless. However, what's that X-My-Header: some_user_data there? Let's imagine that some_user_data can be anything that the user input, and he inputs a string that looks like:

some data;
Content-Type: text/html; charset=UTF-8

<script type="text/javascript">all_your_base();</script>

Now, your response looks like:

200 OK
X-My-Header: some data;
Content-Type: text/html; charset=UTF-8

<script type="text/javascript">all_your_base();</script>

Content-Type: text/html; charset=UTF-8

<html><head><title<Hello World!</title><body></body></html>

And I could have inserted anything! It's not just like a normal Cross-Site Scripting vulnerability, where I can only insert a small dangerous piece of HTML into an already-existing page. I could insert any HTML, any script, or any content at all. That content could be an ActiveX control for Internet Explorer, it could be some other content that your system is already configured to open automatically--anything. I'm not limited just to HTML or JavaScript. There's also a good chance that any Set-Cookie headers will become a part of the body instead of being part of the headers, too, so I can steal cookies even if they're marked "httponly".

For the most part, this vulnerability is prevented if you never allow people to insert a CR or LF character into headers. There's no good reason to allow inserting those characters directly into a single header, so you should be pretty safe to just completely deny those characters in HTTP headers generated by your application.

-Max

VCI 0.7.1

Sun, 31 Oct 2010 23:58:55 GMT

Thanks to the Bugzilla VCS Extension, interest in VCI has really been picking up lately, and so has development. I just released VCI 0.7.1. The biggest feature of 0.7.1 is that every Commit object now has a "uuid" accessor--a way of uniquely identifying a Commit, universally (across all Repositories and Projects), even if the VCS does not normally provide such a mechanism. This makes it easy to tell if two different Commit objects are actually the "same" commit. (For example, if you have two branches with very similar histories, the uuid should help tell you which commits in those two branches are in fact identical.)

-Max

Twitter Account For Bugzilla

Tue, 19 Oct 2010 21:45:00 GMT

Hey folks! I just wanted to let everybody know that there's now an official Twitter account for the Bugzilla Project. This will be a really good way to stay up to date on the very latest news about Bugzilla--we plan to keep it updated frequently about all our new feature development, releases, and things going on in the Bugzilla community.

-Max

VCI 0.6.1

Tue, 24 Aug 2010 19:36:48 GMT

Well, I just uploaded VCI 0.6.1 to CPAN! It's been a long time since I've done a major VCI release, but I'm working on some code that uses it heavily now, so there's a good chance that will be many more improvements to VCI coming along soon. :-)

As of this release, I feel confident in saying that VCI is no longer "alpha-quality software", and it should be pretty much ready for prime-time production use.

There are some important API changes in this release, most notably that the "moved" accessor of a Commit now contains objects instead of strings. (See the Changes file for details.)

VCI now differentiates between "revision numbers" and "revision identifiers" for Commits, where the first one is a human-readable number for a commit, and the latter is a unique string identifying the Commit. As part of this change, the "revision" accessor for a Commit in the Bzr driver now returns a revision ID instead of a revision number.

There are a few other nice new features and improvements, too!

For the next major release, I'd like to implement a "parents" accessor for Commits, that gives you all of a Commit's "parent commits". In centralized VCSes like CVS and Subversion, this is just the commit that came immediately before this one. But in decentralized VCSes like Git, Mercurial, and Bazaar, one commit could have multiple "parents", some of which might not be accessible via the normal revision history otherwise.

After that, there might be some performance improvements to be made, but I'll feel that the read-only side of VCI is pretty much done. Then comes support for working directories and the ability to commit to a repository!

Right now what I'm thinking is that you'll get a WorkingDirectory object via a call to VCI->wd(type => 'Foo', path => 'path/to/something'). This may involve some re-working of how the internals of VCI.pm work. Also, I'm going to make it possible to call "checkout" on a Project, providing a path, and create a WorkingDirectory that way.

-Max

RPC::Any - Simple, Effective XML-RPC and JSON-RPC for Perl

Sat, 19 Jun 2010 18:14:11 GMT

I just uploaded the first stable release of a new library to CPAN, called RPC::Any. It's a simple server framework that presents an object-oriented, unified interface for both XML-RPC and JSON-RPC. It has the following features:

You can use both JSON-RPC and XML-RPC with the same backend code. (That is, you only have to write your functions once--you never have to do anything special for each individual protocol in your code.)
It fully supports Unicode, proven with extensive Unicode tests in the test suite.
If it gets tainted input, it will send your functions tainted parameters.
It can take input from STDIN, from a string, or from an HTTP::Request.
Everything is written as a series of methods whose names and parameters will stay stable, so it's very easy to subclass for your own application, and make it behave completely differently.
It gives your functions the ability to call a type() method to explicitly type return values correctly for the protocol in use. (So you can do $class->type('int', $some_value) within your WebService methods and have that value properly returned as an int instead of a string.)
It doesn't try to start its own server process or send its own output, so you can control exactly how it is used and how the output is sent.
It supports operation in a CGI environment (in addition to almost any other environment you can come up with).
The test suite of RPC::Any has nearly 100% (about 98%) code coverage over the entire library.

More or less, it's everything that I ever wanted in an RPC library, and had hacked into Bugzilla using some very complex code (more on that below). It's also extremely simple to use, for most cases:

use RPC::Any::Server::JSONRPC::HTTP;
my $server = RPC::Any::Server::JSONRPC::HTTP->new(
  dispatch => { 'Foo' => 'My::Methods' });
print $server->handle_input();

package My::Methods;
use strict;

sub hello {
  my ($class) = @_;
  return $class->type('string', 'Hello World!');
}

The server above will read from STDIN, parse HTTP headers and JSON-RPC input, and return proper HTTP headers and JSON-RPC output (for any version of JSON-RPC, no less! 1.0, 1.1, or 2.0). Sending JSON-RPC input with Foo.hello as the method name would call My::Methods->hello() and return the string "Hello World!", explicitly typed as a string. You'll notice that My::Methods calls $class->type even though it never defines a "type" method--that's a magic method that RPC::Any gives the class immediately before it calls the method (and removes after the method is done being called).

To get XML-RPC support instead of JSON-RPC support, just replace every instance of the string "XMLRPC" with "JSONRPC" in the above code (so, in two places) and you have an XML-RPC server instead! No changes whatsoever required to My::Methods or any other code.

A Little Background

So, you might be wondering--why did I write a whole new WebServices module when there are so many options available on CPAN? Well, in fact, I'm actually using two of them as the backends for RPC::Any--RPC::XML and JSON::RPC::Common. So I'm not re-inventing the wheel for RPC--I'm just giving it all a clean, consistent interface.

Here's the whole story behind what motivated me to start writing RPC::Any:

Many years ago, for Bugzilla 3.0, we added a WebServices interface to Bugzilla. When we started off, it was pretty simple. We were using SOAP::Lite for XML-RPC, and everything seemed to be working okay.

Then, we realized that Unicode support was broken in SOAP::Lite (at the time), and so we had to write a workaround for that. Then we wanted to change how we dealt with null values, and we had to write a complex workaround for that. Then we changed how we dealt with input values--more complex, hackish workaround code. Oh, and then we realized that we wanted to preserve taint on incoming values, for security in Perl's taint mode (because otherwise we might accidentally send unchecked input to the database).

All of this was really tough, because SOAP::Lite was basically a SOAP module, with XML-RPC support tacked on. So all our workarounds involved modifying very convoluted SOAP code, when all we really wanted to do was modify how our XML-RPC support worked. Our XML-RPC Server module became quite a monster. (You might ask--okay, so why not switch to a pure XML-RPC module from CPAN? Ah, I tried--but more on that later.)

And then, we added JSON-RPC support, and we wanted to do it with the same backend code--that is, I didn't want to maintain one whole Bug.get function for JSON-RPC and another, separate Bug.get function for JSON-RPC. I just wanted to have one Bug.get function, with two separate frontends.

Well, this was far, far harder than I anticipated. The biggest problem was that SOAP::Lite called methods like Bugzilla::WebService::Bug->get(@params), but JSON::RPC called methods like $server->Bugzilla::WebService::Bug::get(@params);. (So under XML-RPC, the first argument to our methods was the name of our class, like "Bugzilla::WebService::Bug", but under JSON-RPC the first argument was an instance of JSON::RPC::Server.) So in order to be able to explicitly type return values using $self->type within our WebService code, I had to have one "type" implementation in Bugzilla::WebService::Server (which was the base class for both our XML-RPC and JSON-RPC servers) and another "type" implementation in Bugzilla::WebService (which is the base class for our WebService itself--that is, the base of things like Bugzilla::WebService::Bug and so on).

Then, there were numerous bugs in JSON::RPC--so many that I practically ended up writing my own JSON-RPC implementation on top of JSON::RPC, just to get it working properly for Bugzilla. Oh, and also, remember, I had to do all the same things I did for XML-RPC above, like get taint mode working right and testing/fixing Unicode. Oh, and then we added HTTP GET support to our JSON-RPC interface (which JSON::RPC didn't really support natively) and then we added JSONP support to it. It was crazy--just look at the old Bugzilla::WebSevice::Server::JSONRPC.

So along then comes the day when I have a bit of free time, and I'm looking for a project. I think, "Oh, I know, I've always wanted to move away from SOAP::Lite for our XML-RPC service. Let's see what else I can find on CPAN!" So I looked around at all of the available options, and I found RPC::XML, which, though not exactly what I needed, it was well-maintained, XML-RPC specific, and the maintainer had been really helpful with my requests.

So, I started to write a patch to move Bugzilla from SOAP::Lite to RPC::XML...and started to write a whole other series of workarounds--this time not for bugs (because the RPC::XML maintainer was very responsive about fixing those) but just because RPC::XML worked in a different way than we needed it to work, for Bugzilla.

Finally, at this point, I decided it was time to take the engineer attitude and solve the problem the right way--that is, not just fix the problem with our XML-RPC server, but fix all architectural problems of our WebServices by refactoring.

I looked around CPAN to see if there were any good, unified RPC frameworks that supported both the XML-RPC and JSON-RPC protocols, and didn't find anything. So, I rolled up my virtual sleeves and put the time in to do it really right--to write a module that would be the best, simplest, and most powerful RPC framework available for XML-RPC and JSON-RPC. One that fit the (very complex) needs of the Bugzilla Project, but could be used by anybody. A few weeks later, and I had RPC::Any.

It's really been a surprising amount of work, particularly in the test suite. But I think it was worth it--the new Bugzilla WebServices code is way cleaner, simpler, and actually much more powerful than it was before the RPC::Any.

-Max

Improving Web Security: Six Ways the Apache.org JIRA Attack Could Have Been Prevented by Better Code

Tue, 13 Apr 2010 11:14:14 GMT

Today it was revealed that servers at Apache.org and Atlassian were successfully attacked, leading to thousands of stolen passwords. The attack on apache.org's servers was via JIRA, and since the attack on Atlassian came from the same source, it probably was also through JIRA.

I'm sure that JIRA's programmers feel embarrassed enough about all of this--I don't want to berate them, or insult them. Everybody makes mistakes; almost all software has some pretty bad security vulnerabilities at at times. Overall, the JIRA guys seem to do good work, and seem to be generally nice people. And on top of all of that, I understand how they feel! Whenever there's a reported security issue in Bugzilla, I freak out. Thankfully there hasn't been an attack on Bugzilla like this JIRA one in recent memory. But if there was an attack like this, I'd be absolutely mortified, and the last thing I'd need would be somebody trying to insult or attack me for simply having made a mistake.

Instead, I want to use this opportunity as a reminder to all web application developers for why web application security is so important, and talk about some of the things that we do in Bugzilla that would have prevented or mitigated an attack like this one, and that web applications should probably all do as standard practice:

Lock down the on-disk permissions of files and directories. When you install Bugzilla, the actual installation script makes sure that the permissions on Bugzilla's files and directories are as secure as possible. That way, even if there is a security compromise in Bugzilla, the attackers can't upload programs and run them, modify existing scripts, or generally do anything nasty to the machine. It's particularly important that web applications never allow anything to be uploaded into a location where the web server could execute it.
This is actually something that I rarely see web applications stress, in any of their documentation. Some web applications recommend that system administrators fix permissions themselves, but the chance is that the vast majority of people installing your software are going to skip the optional security recommendations, and just go for whatever's easiest. The only way to guarantee that security happens right on every installation is to have the actual installer do the setting of the permissions.

The attackers configured the Apache JIRA to allow uploads into a location where the webserver would execute files, which is what let them compromise Apache's servers and steal the passwords of every JIRA user who logged in to the system. If, like Bugzilla, it was impossible to configure JIRA in that way, that part of the attack would have been impossible.
Httponly: Never allow Javascript to read the login cookie. This is one of the simplest and most effective protections you can make in a web application. Seriously--for Bugzilla, it was just a few lines of code, and eliminated a whole set of possible attacks. All you have to do is to set an extra attribute on cookies when you send them, and you gain a lot of security. If Javascript must read some of your cookie data, that's fine, just don't let it read the login cookie.
If Httponly had been set on the Apache JIRA session cookie, then the cross-site scripting attack that the attackers used could not have stolen administrators' login privileges.
Open-source your software. Okay, look, I know that that's not practical or possible for everybody. But I will tell you, a lot of the security bugs that are found in Bugzilla are found by people we've never met who just happened to be reading the code. These users find all our security issues before they're ever exploited, and so we can release fixes before systems are harmed. In particular, I don't think there has ever been a successful Cross-site scripting attack performed on a Bugzilla--at least not any publicly discussed in the six years I've been working on Bugzilla.
The "many eyes make all bugs shallow" maxim may not always be true, but for security issues, in my experience, it has absolutely held up.

If the cross-site scripting vulnerability in JIRA had been found by an outside user before it was exploited, the Apache JIRA administrators would have been safe from it.
Have automated tests scan your code for potential security issues. There are lots of ways to do this. In Bugzilla, we have an automated test that makes sure that we properly "filter" any data that we got from the user or the database before displaying it on a web page, so that people can't inject malicious HTML or JavaScript into our system. The automated tests don't always catch our security issues, but the number of times that I've fixed a security issue in my code thanks to the tests is uncountable--probably in the thousands, at this point. And those are fixes that happen before the code even gets checked in, so that's a security vulnerability that gets fixed before it even becomes a part of the product.
There are lots of other ways to do automated security testing of code, these days. Static code analysis, Fuzz testing, and automated security scanners seem to be the most popular, from what I've seen.

If the cross-site scripting vulnerability in JIRA had been found by automated tests before it was exploited, the Apache JIRA administrators would have been safe from it.
Lock out users who fail to guess their password too many times. There are lots of approaches to account security, but this is one of the simplest and most failsafe. If an attacker can only guess five passwords every 30 minutes, and then they get locked out, the statistical probability that they will ever guess anybody's password is pretty slim. Starting with Bugzilla 3.6, we implement exactly that policy, and we even notify the Bugzilla administrators whenever somebody gets locked out, so that if there's a large brute-force password attack, the admins will know immediately.
Some people say that the answer to password security is to have people change their passwords every three months. This is probably sensible on some systems, but on a web application, it's mostly pretty ridiculous. If you only change your password every three months, then that gives an attacker three months to guess your password. I can promise you that almost any normal user password could be guessed in that time, particularly if the system doesn't prevent brute force attacks. Then once the user has your password, they can usually do everything damaging that they want to do within a few minutes. So, almost any forced-rotation period is pretty silly, in a pure web application. (In other systems it can make sense--it all depends on the context.)

Other people suggest that passwords need to be a certain level of complexity or a certain length, and up to a point, that's true. If your password is one of the 100 most-common passwords, then even with a sensible lockout policy, the attacker will eventually guess it, if they keep up over a few days. (Of course, in Bugzilla, the system administrators would see all of these lockout notices and probably stop the attack pretty quickly. Still, it's better to be safe than sorry.) So your application should probably enforce a level of password complexity that's sufficient to make it impossible to guess passwords when combined with your lockout policy.

If the Apache JIRA had had brute-force password-guessing protection like Bugzilla's lockout method, the attackers would not have been able to discover administrators' passwords using that method. (My understanding is that newer versions of JIRA do have this protection.)
Store passwords securely. If you're going to store a password in the database or anywhere, store it using some standard, secure method. Don't just hash the password--you have to at least salt them. Preferably, don't even invent your own password-storage scheme--just use some library that already exists.
Never store passwords as plain text. You might be saying to yourself, "Oh, nobody will ever break into the system and steal them." That sounds pretty good until somebody does break in and steal them, and then you'll really be wishing that you stored them properly.

If the Apache JIRA had been storing passwords properly, then Apache.org's users would be at far less risk of the attackers now knowing all the passwords in JIRA.

And finally, on top of all those points, if you're a system administrator, upgrade your software regularly. Some of the security issues that let the Apache JIRA be compromised were supposedly fixed in newer versions of JIRA, before the attack ever happened. Almost every time I hear of an attack like this, it uses old, known problems to compromise the system.

Nobody likes getting attacked. Everybody feels bad about it, when it happens--system administrators, programmers, and most especially users. So let's just design secure applications to start with, and never have any of our system administrators or users have to bear the burden of compromised systems and stolen data.

-Max

Dear CSS,

Wed, 17 Mar 2010 12:46:26 GMT

I love you, man. I really do. You're like a brother to me. But you and me and these web developers here need to sit down and have a little intervention with you.

I know that you've go a lot on your mind right now, but there's some things that we need to talk about with you, dude.

First I just want to say that I love your styles. That's not what we're here to talk about, but it's something I really wanted to give you some kudos for. font, font-style, border, text-decoration, color, background, you are awesome. Those things are so good, I just wanna hug them, most of the time. text-decoration and I have had our differences, but I've learned to work around them, and I'm pretty good now.

And the cascade? The cascade is pretty sweet.

But CSS...your positioning sucks. I know it, these web developers here know it, and it's something that we really need to talk about. It's been happening for a long time, and all of us here really would like you to think pretty hard about bringing some changes into your life.

It's not a big overhaul or anything. It's just a few things that we'd really love to see. There's just a few things. Everybody here really loves you man. We mean it. There's just a few small, little things that would really help make all of us happier, and maybe they're things that could help you feel better about yourself, too:

All I want for Christmas is the ability to position a box relative to another box that is not the container. Like, say, something like: position: relative-to #other_box top-right; top: 0; left: 0;, and then the top-left of my box would be aligned to the top-right of #other_box. Then I would never have to wrangle with float ever again, except when I wanted to have some text spill around a box.
Then while we're at it, I'd really love to be able to specify the size of a box (either horizontal or vertical) as being identical to another box. Maybe something like: width: #other_box and height: #other_box. Tables do this. They did it in 1996. Now it's 2010, and I think that this is something we all would love, but without the tables. So it's something for you to think about; it would really help us out.
Since we're talking about height, it would also be awesome to have a way to say "this should be the height of its container but no larger". You'd think height: 100% would do that, but no, that seems to make the box the height of its content, at least the last time I tried it (yesterday). That's OK, though, we love you even though you're eccentric, CSS. I think we just need a little something different, here.
I can't say how much I'd love to be able to vertically center something in a box. Some of the less-informed members of this little sit-down here might think that vertical-align has something to do with vertically aligning things in boxes. But no, no, it doesn't. It was a bit of a trick that CSS pulled on us, and we understand that there were some reasons, so we're annoyed, but understanding.
While we're at it, wouldn't it be nice to have a single, simple way to center anything horizontally in its containing box? margin: 0 auto usually works. I mean, if you wrap the contents in a table so that they auto-size. (That's a whole other issue, though, possibly solved by display: table?) Sometimes you have to use text-align: center, though. That's fun, but not really what we meant. Then sometimes there's no way at all, which isn't so fun.
If you felt really nice, letting me specify z-index on non-positioned elements and having it actually work would be really awesome. That's just if you feel like it, though. I know that there's a lot of other stuff to think about, here, and and I want you to take your own time to do it, and think about it for yourself.

Those are all the things that we'd need. I know that life is pretty difficult for you right now, what with CSS3, HTML5, quirks mode, standards mode, specs, working groups, modules, errata, and everything. So this isn't something I expect overnight. But it's time for us to say that there's millions of people that you're affecting, with your behavior, and we've given you so much love. Couldn't you just give us a little?

For Serious,

Max

VCI 0.5.3

Sun, 14 Feb 2010 19:54:27 GMT

Well, I just did a new release of VCI! This is version 0.5.3, and it fixes all the test failures and issues that I found with recent Perl module upgrades and the latest versions of all the VCSes. So, VCI should be fully usable with all the modern tools, now!

-Max

The Bugzilla Update Has Moved!

Thu, 05 Nov 2009 16:25:23 GMT

Hey folks! The Bugzilla Update has moved to its own blog:

http://bugzillaupdate.wordpress.com/

If you'd like to subscribe in your news reader, the feed is here:

http://bugzillaupdate.wordpress.com/feed/atom/

There's a new post today that has a LOT of news from the Bugzilla Project, over there. :-)

-Max

Let's Talk About My SSD (Solid State Drive)

Wed, 16 Sep 2009 01:56:15 GMT

So, about a week ago I ordered an 80GB Intel X25-M G2 SSD, and as of a few days ago, it's running as my main system disk. It's a lot smaller than my old main disk (which was 320GB), but I just kept all my sound and pictures on the old main disk and moved the operating system over to the new disk, and that handled the size difference.

Now, having used it for a few days, I think that we are right at the beginning of the "SSD revolution"--that the curve where SSDs will start to replace hard drives for most common use is very close. Here's why:

My SSD was 80GB and about $350 (and that's only because demand is so high that they increased the price from $270, right now). Although that's way more expensive than a hard drive of the same size, it's not unreasonably expensive, or beyond the purchasing power of a normal consumer.
For normal dekstop use, the drive is about 5-10x faster than my old hard drive:
- One part of the boot process on my machine that used to take 30 seconds now takes 5 seconds or less.
- After I log in, it takes only about 2-3 seconds to present me with a fully-functional desktop with all startup programs running.
- Most programs (including Thunderbird) start under 2 seconds.
- Opening a new window or tab (including loading and fully rendering my iGoogle home page) in Firefox is instantaneous--under 1 second.
The drive generates almost no heat--I've had it on for hours and it's cooler than my hand.
The drive consumes almost no power compared to a hard drive.
Physical dimensions don't matter--a solid state drive can be any physical size, and it's still the same speed (unlike hard drives, where small laptop drives are almost always slower than big desktop drives).
Solid state drives produce no vibration and make no sound. For all intents and purposes (because I designed the rest of my machine to be very quiet) my machine is now totally silent during operation.
The drive's physical orientation doesn't matter, and it can withstand being dropped or thrown around even when operating (obviously to a limited degree--if you hit it with a hammer, it will stop functioning). Because there are no moving parts, you can pick the drive up and turn it sideways, upside down, jiggle it--whatever--while it's running, and it will make no difference in performance or safety.
All of the old problems that SSDs used to have (that they were slow, that they were unreasonably expensive, that they became slower over time, that they "stuttered" and locked up systems briefly) have now been resolved, and SSDs "just work".

Overall, it's the most significant performance upgrade I've ever made to a computer (other than giving enough RAM to a machine that was woefully low on RAM), and I'd strongly recommend it to anybody who can afford it, knows how to install it and copy over their OS, and who uses a computer regularly.

Now, if you're going to buy an SSD, it's important that you buy the right one, so read the latest AnandTech SSD Overview to understand how SSDs work and which ones are the best to buy.

Warning: Major Bugzilla Security Release Coming Soon

Thu, 10 Sep 2009 00:53:10 GMT

A major security issue has been discovered in versions of Bugzilla back to 3.0. We will be releasing a version of Bugzilla which fixes the issue within 48 hours (possibly within 24 hours), and all administrators should be ready to perform the upgrade (which does not require any database changes) shortly after the new version is released.

If you do not wish to do a full upgrade, patches for just the security issue will be available. The patches are relatively small and do not modify very much of Bugzilla.

Release of Bugzilla 3.4! (Bugzilla Update: July 28, 2009)

Tue, 28 Jul 2009 10:40:43 GMT

I have just posted the tarballs and done the website updates for Bugzilla 3.4! This means that we're out, released, ready to download, install, and go!

Bugzilla 3.4 is the best release of Bugzilla we've ever made. It has tons of great new features, the most exciting of which are listed in the release announcement, so I won't repeat them here. But you should go download it!

The Story of Bugzilla 3.4

As you look through the New Features list of Bugzilla 3.4, you may notice that it fixes tons of major issues that Bugzilla has had since its beginning. For example, we fixed the biggest performance problem in Bugzilla--sending emails when a bug is updated--and we finally hide email addresses from logged out users, to prevent spam. And that's just a tiny taste of what's new. Really, check out the New Features list to see everything.

But you may be asking yourself, why the sudden fixing of all these issues, and why didn't we do it before?

Well, that's an interesting story! From about 2003 to 2008, we spent nearly all of our time fixing up the code of Bugzilla. It needed a lot of refactoring, and we really did it--five years of it! We added new features at the same time as we refactored (remember, Bugzilla 3.0 had the largest number of major new features of any release we've ever done, and we were still refactoring), but the refactoring was our main focus. But finally, finally, with the release of Bugzilla 3.2, we fixed up one of the last major code issues in Bugzilla--we changed process_bug.cgi into a nice, simple series of steps that use Bug objects to do all their work.

After all this was done, we could finally take the time to look around and say, "What next?"

Well, what happened next was what led to such a great Bugzilla 3.4 release. First, I declared a new method of prioritizing work on the Bugzilla Project that put major issues of our current users as higher priority than adding new features for our prospective users. This led to us looking at the major survey items from our 2008 Bugzilla Survey and doing something about all the major requests that we could address immediately. Then we went through and looked at the bugs with the most votes on them, and did something about a lot of them.

And that, pretty simply, led to us addressing the things that people most wanted, and that we could actually prove that they wanted (because we had great survey feedback, or a lot of votes from individuals on our bugs).

Now that we've addressed so many of the individual things that users wanted, look to Bugzilla 3.6 and later for some big user interface and usability improvements--we have the results of extensive usability research that was done on Bugzilla, thanks to students from Carnegie-Mellon University, and we are already addressing the list of issues that that research generated.

Warning for WebService Clients: Changes Since Bugzilla 3.4rc1

Anybody who has been writing WebService clients against the 3.3.x or 3.4rc1 releases should know that we changed a few things in the API between 3.4rc1 and 3.4:

Bug.comments now takes an "ids" parameter instead of a "bug_ids" parameter (we just renamed the parameter to be consistent with out other WebService functions). Also, it will now throw an error if you try to add a private comment and you don't have the permissions to do so. (Previously it just added a public comment if you didn't have the permissions to make a private comment.)

Bug.history now returns its result in a completely different format, one which is more consistent with the format that Bug.comments and Bug.get use.

Progress on Bugzilla 3.6

Since our last Bugzilla Update just a few weeks ago, we've fixed several usability issues, sped up quicksearch, and added the ability to disable field values in global drop-down fields (without deleting the value).

Coming up soon, expect to see a lot of new WebService methods--there's been a lot of activity in adding WebService code, lately.

The End of Bugzilla 2.x

With this release, we EOL'ed Bugzilla 2.22, the last remaining supported 2.x release. That means that only 3.x releases are supported now. It's kind of wild to think that Bugzilla 2.x is "dead", after nearly ten years, and so much of my personal time spent on it. I started working on Bugzilla back in the 2.18 days, and I was pretty much the release manager for three major 2.x releases--2.18, 2.20, and 2.22. It's amazing to think that those releases were so long ago that now the very last one has reached the end of its support life. It's all Bugzilla 3.x (and hopefully 4.x soon) from here on out, my friends! :-)

Subscribe to the Bugzilla Update

There is an Atom feed that you can subscribe to and read in your RSS reader, for just the Bugzilla Update.

Parse::StackTrace

Tue, 14 Jul 2009 03:58:13 GMT

Today, as part of a larger project that I'm working on at Everything Solved, I published to CPAN a Perl module that I've been working on, called Parse::StackTrace. It parses the text representation of a stack trace from GDB (the GNU Debugger) or Python, and converts it into an object that will tell you all about that stack trace (or at least, as much as can be known from the text representation). This might be useful for anybody who receives text stack traces and wants to get information about them in some programmatic way.

-Max

Bugzilla Update: Wednesday, July 8, 2009 (Release of Bugzilla 3.4rc1 and Bugzilla 3.2.4)

Wed, 08 Jul 2009 14:57:04 GMT

Well, it's time for another Bugzilla update! And today I just did two releases, Bugzilla 3.4rc1 and Bugzilla 3.2.4.

Bugzilla 3.4rc1

Bugzilla 3.4rc1 is particularly exciting, because it's our first Release Candidate for 3.4. We did a really good job on this Release Candidate, I think--there's only one 3.4 blocker remaining (and it's only still there because we're waiting on an external party to do something). In other words, there are no known issues with the Release Candidate that are so bad that we couldn't just call it 3.4 next week if all goes well, and we've never actually been in that state for a Release Candidate, at least not as long as I've been around the Bugzilla Project.

One of the particularly exciting thing about a Release Candidate is that it has release notes! That means that all the new features are listed. There's a lot of really exciting stuff in 3.4, and you should take a look. There are some gems in the "Other Enhancements and Changes" section, too, so make sure you read that too. :-)

WebService Changes Since 3.3.4

Anybody who was writing WebService clients against 3.3.x development releases should know: we renamed the Bug.get_history method to Bug.history. You can still call it as Bug.get_history if you want, but that's undocumented and not recommended.

Also, we don't send <nil> for NULL items anymore--too many clients didn't support it. Now we just remove items from the returned result if they are undefined. (This is documented in the Bugzilla::WebService documentation.)

Progress Toward Bugzilla 3.6

There's been some activity on HEAD since our last update. We got a new WebService method to get attachment information, Bug.attachments. I've been working on making Quicksearch (the search box in the header and footer) even faster. Greg Hendricks (of Testopia fame) has been working on the ability for administrators to "disable" certain field values (so that they don't show up as options anymore, but remain set on existing bugs). And Bradley Baetz has been adding new hooks and working on improving performance in some important areas.

There's no ETA for Bugzilla 3.6, but if it works anything like how Bugzilla 3.4 works, we will have open development on it until two months after Bugzilla 3.4 is released, and then we will branch for 3.6 and the 3.6 branch will be "frozen" to only bug-fixes.

Bugzilla Meeting

We have a Bugzilla Meeting next week, on Tuesday, July 14. Just read the page if you want more information! Anybody is welcome to attend.

Bugzilla Update: Thursday, June 4, 2009

Thu, 04 Jun 2009 23:30:00 GMT

Well, it's time for another Bugzilla update!

Bugzilla 3.4

In the Bugzilla 3.4 area, we just made some more changes to how the login form in the header and footer work. Now it's easy again for users to discover how to reset their password--when we moved the login forms into the header/footer, we at first didn't have any way for people to discover how to reset their password, but now there's a link and everything works really nicely. You can see how it all works on the Bugzilla 3.4 Test Installation.

We're getting somewhat closer to Bugzilla 3.4rc1. We found a few more blockers, so those have to be resolved, and there's also release notes that need to be written before we can have a Release Candidate.

One new feature of Bugzilla 3.4 that we haven't talked much about is the "See Also" field. This is a field where you can put a URL to a bug in another Bugzilla installation or to a Launchpad bug. The "See Also" feature isn't quite complete--in the future, we also want to make it update the other installation so that the other installation knows that you're referring to it. We also want to fix up the display, and get summary/status/resolution information on the remote bug, etc. But for now it does check that you've entered a valid bug URL, and at least you can somehow record that bugs in different Bugzilla installations are related to each other, and there's a WebService interface for updating the field.

For installations that don't need the "See Also" field, you can turn it off by disabling the "use_see_also" Parameter.

Bugzilla 3.6 (HEAD)

We're working on various interesting things for Bugzilla 3.6, though our focus recently has been on 3.4rc1, so there are a lot of patches awaiting review for HEAD that haven't had the time to be reviewed. People are working on the ability to disable field values and some cool WebService enhancements, but of course our main focus is fixing up the HCI issues that the Carnegie-Mellon research team discovered in their 2008 study.

-Max

Bugbot's Code Gets a Home

Fri, 22 May 2009 22:55:01 GMT

I just created a Google Code page for the Supybot Bugzilla plugin (what bugbot is using to report changes):

http://code.google.com/p/supybot-bugzilla/

You can use that to get the code and to report issues about it.

I'd love to put the plugin's documentation on there, but the supybot-plugin-doc script throws a traceback when I try to generate documentation for the Bugzilla plugin, unfortunately.

There aren't currently any release tarballs on the page, but if anybody wants one, just ask me and I'll create one.

-Max

Bugzilla Update: Wednesday, May 20, 2009

Wed, 20 May 2009 22:06:24 GMT

Hey hey. So, I was thinking that I'd do a regular (or semi-regular) post on the status of the Bugzilla Project, for anybody interested. This is the first one.

Bugzilla 3.4

We're getting pretty close to releasing Bugzilla 3.4rc1, now. There are only a few blockers left. Mostly they're just awaiting review. I'll also need some help with the release process for Bugzilla 3.4rc1, if anybody wants to help out.

The only significant changes since 3.3.4 will be a lot of bug fixes, a change to the Bug.search WebServices API, and the ability to hide the "See Also" field. The bug fixes are pretty important, though, so if you're using 3.3.4 you definitely want to update to the most recent BUGZILLA-3_4-BRANCH code regularly or update to 3.4rc1 when it comes out.

There are a lot of significant changes in 3.4 compared to Bugzilla 3.2, though. Those will all be listed in the release notes for 3.4rc1. The difference between 3.2 and 3.4 is not as great as the difference between 3.0 and 3.2 though. We're working on having smaller releases more often (starting with 3.4), and it seems to be working pretty well so far.

HEAD (Bugzilla 3.6)

On trunk (which will be Bugzilla 3.6), we've done a fair bit. There's a JSON-RPC interface, support for suexec environments in checksetup, and a lot of HCI improvements. We've decided that for Bugzilla 3.6, our focus isn't going to be adding major new features, but fixing up the features we already have. I wrote a message to the Bugzilla Developers List about it, a week ago or so, and I got a lot of positive responses (mostly on IRC or by private email). If you're interested in helping out, feel free to check out the list of bugs we'd like to fix for Bugzilla 3.6.

-Max

Fixing HCI Bugs in Bugzilla

Thu, 30 Apr 2009 08:53:33 GMT

Guy Pyrzak led a bunch of Carnegie-Mellon students in conducting some HCI research on Bugzilla, and they came up with extensive and detailed results.

I have been working on filing bugs in response to their results. I haven't finished filing all the bugs yet, but if you want to follow along with the issues as they are filed and fixed, you can watch the tracking bug I filed to keep track of all the usability issues discovered during the research. Some of them are minor issues that now already have patches awaiting review. Some of them are larger issues that will require work over time to fix. But I am committed to seeing them all fixed as we move forward.

We've fixed Bugzilla's backend pretty well, now. It's time to focus some on the front end.

Anybody who wants to help, please do. Any of the bugs that block the tracking bug and are assigned to "Nobody", you are welcome to take and work on yourself.

-Max

Switched to Thunderbird

Thu, 02 Apr 2009 06:14:43 GMT

I switched to Thunderbird today and I am really loving it. So far, it's the best email client I've ever used. :-) It has so many nice things about it--but then again, if you're reading this, you probably already know that. :-)

-Max

New Default Bugzilla Workflow?

Wed, 01 Apr 2009 11:14:33 GMT

I have proposed that Bugzilla have a new default status workflow. I wrote my reasoning in a message on the mozilla.dev.planning newsgroup, originally as an argument for a workflow that Mozilla should move to, but I think that it covers the basic bug-fixing process sufficiently well as to apply to all organizations, and thus should be the default.

I'd welcome constructive feedback, even just statements of agreement. Note that I said "constructive" feedback, not insults or rudeness, which I will most likely just ignore. :-)

-Max

When does form autocomplete happen?

Tue, 24 Feb 2009 03:55:07 GMT

Today for fixing a bug in Bugzilla I needed an event to fire only after the browser had automatically autocompleted the login form (what Firefox, Chrome, and Safari do when you have saved exactly one set of login credentials for a site). Most people would think that window.onload would work, and it does, in one browser--Chrome. Other people might try to use YUI's onDOMReady event, which fires when all the content is available in the DOM. That works in exactly one engine--Gecko.

The problem is that autocomplete happens at different times in different browsers, and you can't even rely on knowing what engine the browser is using--in WebKit-based browsers, it happens at different times depending on what browser is being used. Here's when autocomplete happens in different browsers:

Gecko: Before onDOMReady, but after window.onload (so use onDOMReady).
Chrome: After onDOMReady, but before window.onload (so use window.onload).
Safari: After both onDOMReady and window.onload (so do a 200 millisecond window.setTimeout on window.onload).
Opera: Never autocompletes forms without user interaction.
IE: Never autocompletes forms without user interaction.

So the only reliable solution is to fallback to the Safari solution, which is to do something 200 milliseconds after window.onload. I tried 100ms, and I was getting into race conditions where sometimes my event would fire before autocomplete, sometimes it would fire after. I upped it to 200ms as a safe amount.

This is a little annoying if you want something to happen instantly after the form is autocompleted, so what I actually did for Bugzilla was I fired the event after onDOMReady for Gecko, and used the Safari solution for all other browsers.

Anyhow, it would be nice if this was all standardized some day.

-Max

Fushigi Yugi

Sat, 21 Feb 2009 12:23:02 GMT

So, I just finished watching Fushigi Yugi. It's definitely an older anime, with a more traditional style--almost a more serious and delicately-crafted Dragonball Z with more of an appeal for girls (but still basically an action show despite all the emotional content). I thought the show did best when it was focusing on action, and whenever it started to get weepy I'd kind of roll my eyes and wait for that part to pass. :-) And it's not that I'm opposed to emotion in shows--on the contrary, how else I could I have loved Honey and Clover so much? No, the weepy bits of Fushigi Yugi were just not that great. :-)

Overall I'd give it like a 5.5 out of 10. Almost everybody has seen it, I think, so I think in a way that makes it worth watching just so you can bond over it with other people. And it's not bad. I thought the ending was pretty decent, and the beginning was pretty strong. It got a little dull about 3/4 of the way through, but picked up pretty well in the last 1/8th or so.

One great thing about the series is that you get like 6 to 8 episodes per disc! They didn't fill the discs up with silly extra features or anything, they just put more actual episodes on them, which is soooo much better than the discs for any other series I've watched. I mean, I think I've even heard of series where they only put 1 or 2 episodes per disc...Fushigi Yugi is positive proof that nobody actually needs to do that.

Still, even with all those episodes per disc, there were something like 8 discs. It's definitely not a short series! It didn't feel all that long, though, somehow. Perhaps it's that since there were so many episodes per disc, I would watch a lot of episodes at once! So it didn't take all that many sessions of watching to finish it. :-)

Anyhow, glad to have seen it, but really glad to be going on to something else now--maybe even some normal movies or TV series for a little while, taking a little break from anime. Well, except for the Azumanga Daoih that justdave lent me--I do want to watch that! :-) So I guess it won't be a total break. :-)

-Max

The iPhone SDK

Thu, 05 Feb 2009 14:10:22 GMT

So, if Apple was wondering about me, and perhaps picking the petals off flowers going "He loves me, he loves me not," I think this time we ended up on "he loves me not."

The iPhone SDK only works on Mac OS X.

I do not own a Mac. In fact, I'm pretty sure I don't own any machine on which OS X will run.

The explanation for this is, "It's all built around XCode and Objective C which don't work on other platforms."

Let's back up.

Why isn't XCode based on Eclipse? Then it would run on every platform.

Objective C works on Linux. Linux is an OS that developers use. Wouldn't it be nice to have an SDK that developers could use?

If the simulator only ran on a Mac, well okay, that wouldn't make sense since Macs are now Intel machines, but maybe there's some graphics interfaces that don't exist on other platforms. Fine.

But can't I at least get the ability to develop an iPhone app using the platform of my choice, instead of being ASSIMILATED INTO THE BORG COLLECTIVE just to write a program?

I expect to get approximately 1 billion comments now, since I'm complaining about something. It seems that happens whenever I complain about something. :-) Ah well. :-)

-Max

Bugzilla 3.2.1, etc.

Tue, 03 Feb 2009 01:50:46 GMT

So, today we released Bugzilla 3.2.1, which fixes the longest-standing security bugs in Bugzilla, in addition to a few other security issues. These long-standing security issues were actually public for many years, but it required a lot of re-architecture of Bugzilla before we could fix them.

We also released 3.3.2, which has a lot of cool new features, not the least of which is hiding email addresses from logged-out users.

And we put out Bugzilla 3.0.6 and Bugzilla 2.22.7 as security fixes for people still using those older branches.

Anyhow, you can read the news announcement for more details, and the Security Advisory if you want to read up on the security issues that were fixed.

It feels really great to have these releases out. Although I haven't ever heard of a successful exploit of these security issues, they've been around for so long that it was naggingly worrying to have them there, and it's a huge relief to have them fixed and see the fixes released!

Anyhow, here's a bit of news about Bugzilla trunk, which will be Bugzilla 3.4:

- Our current estimated release date is sometime in May, but that's a very rough estimate.

- We're now in a soft freeze (since Jan 29), which means that enhancements that had patches before Jan 29 can still go in, but any enhancements that didn't have patches at that time can't go in. This allows existing patches some time to pass review and to clean up any feature work that wasn't quite done before the freeze.

- There are a few nice enhancements that should still be coming, including a simplified bug entry page.

- If there are any other long-term major problems that you see in Bugzilla that we haven't fixed in 3.4, please let me know. Point me at a bug, or anything you want. And I don't mean minor things, like the positioning of a button or some text on a page, but big things like how emails used to be displayed to logged-out users. :-)

-Max

The Dusk skin and bugzilla.mozilla.org

Fri, 23 Jan 2009 12:49:10 GMT

For those using bugzilla.mozilla.org, I wanted to let you know that when we upgraded to 3.2, you got access to a new Bugzilla skin called "Dusk." Though I've been attempting to convince the admins of the site to switch bugzilla.mozilla.org over to usnig Dusk by default, it hasn't happened (yet?). So, I wanted to let you know that you can change Bugzilla's skin to Dusk yourself, using the General Preferences page.

Dusk is much nicer than the Classic skin, in my opinion, and I think Bugzilla actually becomes more usable with it. If you want to see what it looks like before you switch over to it, you can either use the "View > Page Style" menu in Firefox, or you can look at our test installation, which defaults to Dusk.

-Max