You are viewing avatraxiom

To The Side

The Problems of Perl: The Future of Bugzilla

Once upon a time, Bugzilla was an internal application at Netscape, written in TCL. When it was open-sourced in 1998, Terry (the original programmer), decided to re-write Bugzilla in Perl. My understanding is that he re-wrote it in Perl because a lot of system administrators know Perl, so that would make it easier to get contributors.

In 1998, there were few advanced, object-oriented web scripting languages. In fact, Perl was pretty much it. PHP was at version 3.0, python was at version 1.5, Java was just starting to become well-known, ruby was almost unheard of, and some people were still writing their CGI scripts in C or C++.

Perl has many great features, most of all the number of libraries available and the extreme flexibility of the language.

However, Perl would not be my first choice for writing or maintaining a large project, such as Bugzilla. The same flexibility that makes Perl so powerful makes it very difficult to enforce code quality standards or to implement modern object-oriented designs. Here are the problems:

Read more...Collapse )

Comments

Page 2 of 2
<<[1] [2] >>

(Anonymous)

Great work!

Hi Max,

Thanks for writing a thoughtful article on the possibility of moving Bugzilla to another language. Your experience with Perl gives your assessments more weight.

When contemplating a language switch, also consider the likely future popularity of the candidate languages. The more popular a language, the more cool tools and libraries it will have. Popular languages offer a bigger pool of developers to contribute to Bugzilla. The use of Bugzilla would increase too, because developers are more likely to use a tool, such as Bugzilla, when it's written in a language they know.

It looks like Ruby is the scripting language of the future. In terms of industry dominance, Ruby is the next Java. Perl is in decline. Python, while a good solid language, has a quirky block and OO syntax that bother developers who prefer C-like languages. In any case, Python's growth is slowing and will likely peak in the next two years as more people discover Ruby. Although Ruby is not yet mature, the tremendous amount of effort now going into it now will bring it to parity with Perl and Python in the next 12-18 months.

D is also an interesting possibility. The vast majority of programmers prefer C-like languages. D is a C-like language that adds OO and garbage collection to C, but without the mind-numbing complexity of C++. C/C++/Java/C# programmers can easily become productive in D. It's possible that if Bugzilla chose to rewrite their app in D, that choice could put D on a path to become the world's preeminent compiled language. That's because Bugzilla is one of the world's preeminent open-source applications.

I think Bugzilla would be best served by being rewritten in Ruby or D. Python wouldn't be a bad choice, but it wouldn't be as good in the long-term as D or Ruby.

Re: Great work!

I agree with you about the future of Ruby. I'm not sure that Python will peak and die out, though.

I don't see any future in D yet. It doesn't have any momentum with non-CS-types.

Thanks for the comment, I really do appreciate it. :-)

-Max

(Anonymous)

We done did this (40,000 lines of perl -> ruby).

The Metasploit Project spent the last two years turning 40,000 lines of Perl code into Ruby. Ruby is easy to learn for Perl developers, is powerful as hell, is easy to read, and its not Python. Perl is a dead duck at this point - not many new programmers are learning it and all of the Web 2.0 shiny is happening with Ruby, Python, and .NET these days. We accomplished this with a parallel rewrite and a 6-month alpha/beta period. Once the beta period started, we stopped maintaining the old Perl version.

The new version can be found at:
http://framework.metasploit.com/

Re: We done did this (40,000 lines of perl -&gt; ruby).

Wow, cool! I was wondering if anybody did it.

What advantages did you see in general? And how did you convince people to do the rewrite? :-) Also, how many developers are on the whole team?

-Max

(Anonymous)

Hmm what about C?

I am not very good programmer, but you sound to me like - "I will do my best to not end with C++/C#"? So what about pure, simple C? With one plus = using Visual Studio for Development platform? You will have a lot of pros and only one cons - MS :):)But you can use MS V Studio like good tool with security templates, UML integration, etc. and by this eliminating (:):):)) this con!

Re: Hmm what about C?

Well, no, C would make all our problems ten times worse, and using Visual Studio would definitely be a bad idea--most of our developers run Linux.

-Max

(Anonymous)

Other bug trackers? Worth _having_?

> Nowadays, almost all of our competitors have one advantage: they
> are not written in perl. They can actually develop features more
> quickly than we can, not because of the number of contributors they
> have, but because the language they're using allows it. There are
> at least two bug-trackers that I can think of off the top of my
> head that didn't even exist in 1998 and were developed rapidly up
> to a point where they could compete with Bugzilla.

Care to name them?

I have seen a lot of bug trackers, and I would generally divide them into two categories. The first category consists of Bugzilla and forks thereof (e.g., Issuezilla), and the second category consists of lame half-baked immitations that are painful to work with, poorly designed, feature-impoverished, and terrible in terms of usability.

Particularly bad are the ones written in C++, PHP, and ASPX. Those languages should certainly not be considered.

The ones written in Python tend to suffer from the same malady that afflicts Mailman, to wit, if there's One Right Way for the programmer to do something in the language (Guido's Way, presumably), then the programmers tend to adopt the position that there's One Right Way for the user to want to use the software, the result being software that is painful to have to use if you don't happen to think exactly like the developers. I suppose it would be possible for the developers to overcome this and write good bug tracking software in Python, but I haven't seen an example yet. (Note that I'm not saying no such example exists, only that I have not seen it. Feel free to point one out, if you know of one that I've missed.)

Ruby would be interesting. Five years ago I'd have said it'd be impossible to find enough developers who know it, but Ruby has been receiving a lot of positive press lately (including in the Perl community -- a lot of the big names in Perl say nice things about Ruby), so its popularity has really grown. These days writing something like Bugzilla in Ruby might be quite feasible and worthwhile. I'd be interested in seeing that.

I wouldn't want to see development of the existing Bugzilla codebase cease, though.

And yes, Perl6 would be a great choice if a fully usable version of it existed today, but admittedly that's a pretty significant caveat. When choosing between Perl5 and Perl6, I keep finding myself picking the one that's available and widely deployed and stable, despite the philosophical allure of many of the design improvements in the new version.

Re: Other bug trackers? Worth _having_?

Hey, thanks for the very insightful contribution, there. :-) And I'm really glad that you feel so positively about Bugzilla vs. its competitors, that makes me happy. :-)

I think most people tend to prefer Bugzilla, also, but they sometimes install Trac because it's easier to install, not necessarily because they like it better. (Also because it does two other things--track SVN and it's a Wiki, so people like that. Bugzilla on the other hand will always stay focused on just being a bug tracker.)

Since we're way down here in the comments, I suppose I'll just say that the other bug-tracker I was thinking of was Jira. (Which, by the way, if you think about it a bit, is the end of "Gojira," the Japanese pronunciation of "Godzilla," and...yeah.) I haven't used it extensively, but I know there are some projects that are using it and like it, and it definitely didn't exist in 1998.

And yeah, I don't think we'll stop development of the existing codebase. There might be enough modules in CPAN available to make our development easier without having to switch languages.

Oh, and as far as Python having a One True Way, I do understand what you mean. :-) That's one of the things I like about Pylons, though--it's designed to allow you to pick the best tool for the task, so you aren't locked into just One Way.

Oh and yeah, I can't even imagine writing a bug tracker in C++ or ASPX. I could imagine writing one in PHP5, but only with extremely careful code review from highly-trained PHP developers.

-Max
Just rewrite Bugzilla on OCaml 'cause it is the fastest high-level language.
LOL.

-Max
"It relies heavily on complex regular expressions."
I heard a few times about Java/PHP/* advantage over Perl because their new lang has PCRE :)
Have you seen messy Java regex in action with tons of '\\\\'?

I think if wrote Bugzilla from scratch using different language you would end up with the same post just blaming new set of drawbacks which Perl has not.

Regading Python & Ruby. Just compare standard documentation you would have to work with daily and feel the difference:
Python: http://docs.python.org/ref/ref.html
Ruby: http://www.ruby-doc.org/core/



Python Documentation

This page is the place to go for starting to learn Python:
Beginners Guide

And this page:
tutorials.

From the (themed) list of 300+ Python tutorials and help from comp.lang.python most people should be able to find Python tutorials to suite their starting ability and interests.

- Paddy.

(Anonymous)

It's the programmers, stupid! :-)

Sorry, your post contains a bunch of very common misunderstandings. I won't go through them all, there are far too many for my time and 4300 chars. To begin with, writing good code is difficult in every language. You contradict yourself many times; as an example - you say at one place
In any language it's possible to do stupid things and not realize it. In Perl it's easy.
and then further down
Perl's lack of enforcement is a nice feature for the casual programmer, but for the design of large applications, you want the programming langugae itself to do as much error-checking for you as possible, so that you don't have to write the error-checking yourself.

I'd argue it is far easier to make stupid mistakes in PHP - bcause in fact, many PHP stupid things are in the core of the language itself... but point is, freedom is a perilous gift, and different from libertinage. Perl's freedom and "there's more than one way to do it" is much more difficult for the casual programmer than for the expert, because the latter knows what he is doing, and why. Perl's freedom is for the expert only; for the beginner, it is called forgivefulness, and that's a entirely different thing.

Perl's freedom is there so that you don't have to fight your tools, but you need to know what the impacts are to make sensible use of it. Language restrictions to enforce good behaviour? It's not the lack of prisons which gives raise to criminality, it is more the other way round.

If code review seems difficult, it's not the language that is to blame, but the programmer which wrote the code in the first place; and then it's the cluelessness of the code reviewer. A Perl code reviewer needs a deep, solid knowledge of the language (as you would demand of any code reviewer in any language), because too much "forgiveful code" would pass as "freedom code" otherwise. I would not let beginner's code of any realm into production without severest scrutinity.

Think of it - if you heap layer upon layer, plug in third party libraries without knowing what they do, even when you need only a small part of each, if you don't enforce a good design and layout at the beginning, you're lost. Switching languages won't help, you will end with the same stinkin' heap of crap once you're through, and you won't realize it until some cycles of code extension and code revision have been done.

Perl is fading, you say? *shrug* - Perl will not die any time soon, it is too nifty a language, mature and robust. And yeah... UNIX was seriously dying back in 1993, I remember.

But if you want to throw away the effort of years - go ahead. But I'd suggest the following roadmap:

  • refactor and revise first.
    Identify cruft and code parts that smell.
  • reduce the codebase
    Refactor and minimize. Every line you take out is a possible source of bugs less. I'm confident that you realize that I don't advocate densifying each line and writing obfuscated perl; strife towards succinctness, conciseness and clarity.
  • revise the libraries used
    If you need only a small part, inline that. Throw away smelly ones.
  • delay feature additions
    If new features don't plug in nicely, the plugged part is crufty.

If the code is cleared (which has to be done anyways, otherwise the same old cruft will creep into the new port), decide again about switching the language. But then, I'd port to the old and the new language if feasible, and cut down the less productive branch in the process.

I'd switch language on a project only if there are a good handful of killer demands which the language as is used absolutely cannot handle any time soon. I doubt that with Perl this would be the case.

On last point - you cannot make expertise dispensible just by switching languages.

Re: It&#39;s the programmers, stupid! :-)

Sure. I wasn't arguing in favor of PHP.

We've already been refactoring and revising for years. We don't intend to stop.

We do inline libraries if we only need a tiny part. We almost never need just a tiny part.

We did delay feature additions.

We did all those things you're suggesting. We're still doing them.

It would help if you had any understanding of the history of Bugzilla, or some understanding of the available programmer-base for it, or some understanding of the people who currently work on it (who are pretty good Perl programmers, at this point).

I do appreciate that you put some thought into your post, though. Many people have said the same, above (there's 130 or more comments now, it seems), but many didn't say it quite as articulately as you did.

I'm sorry that my answer is "I already know. We already are." :-) The post wasn't really intended for people who don't know the history of Bugzilla development.

-Max

Choosing

Disclaimer: I won't troll or flame. Just curious and open for discussion. :-)

Well, in my opinion, Perl is good candidate to be in Museum of Science these days. Still it is sometimes good for up-to-100-lines of scripts. But hell no for developing big complicated systems.

Now I am using Python just because of its simplicity and high modularity. I like simple things because what is simple is genuine. Complicated is usually ugly. In my practice I have done large projects for banking were involved all these: Perl, Java, C++ and Python. And I can say that Python is the most optimal tool along with C++ if you need something very-very special.

Usually Perl guys says: "You can do this way, you can do that way (same thing), you can do third way (same thing)". C'mon, why I need all of that? What for? There ARE rules for Perl users, but those rules are always ignored. I agree that lack of braces are cumbersome after Perl. The argument that "Python lacks variable declarations, which means that invalid variables are caught at runtime instead of compile-time." is quite weak, because you just don't need it. You should write a *pythonic* code in pythonic way, instead of thinking in Perl way just using Python syntax! :-)

Python Pros on Wiki are very weak and sometimes ridiculous. Look at IBM's articles how well Python does with Unicode. Yes, str() and unicode() are bad thing, but Python 3000 will get rid of this entirely.

Wiki says: "Sometimes has unclear error messages.". I am wondering what it does mean? As for me, Python's tracebacks are one of the best I ever met in this bundle, mentioned above. I don't think you like 6-10 screens to scroll Java traceback just to find you did syntax error. :-) And Perl's error messages like "Something wrong in line 205" sounds like small kid feels pain, but can't say where.

Wiki says: "Poor Unicode handling" — it just gets me laugh. :-) I'd like to see Perl's Unicode support and clear difference between Unicode string and 8-bit encoded string. As an example, try to convert UTF-16 narrow-width Japanese Katakana to UTF-8 wide-width Japanese Katakana in Perl and then I will show you how it is done in Python. :-)

Wiki says: "Python has no equivalent to Perl's "taint" mode." Well, yet Zope is one of the most secure websystems ever made. Especially, Zope 3. Can you point me equivalent secure system written in Perl? (Just curious)

And still my Python code will be shorter than your Perl code. Well, you may gzip your Perl code, then it will be smaller while readability of the Perl gzipped source will not suffer so much... :-)

P.S. I know Ruby and some of its features I want have in Python. Yet I prefer Python over Ruby much-much more.

Re: Choosing

Hey hey. :-)

Yeah, I agree some of the Python pros and cons aren't as clear as some of the other languages.

The Wiki page is only locked because there was some trolling and flaming. :-) (This subject is a bit of a passionate subject to some people.)

My only current objection to Python is that in some cases if you want to use Unicode and don't expect it, you have to go back and modify your program a fair bit. And that it actually throws an exception when you try to mix ASCII and Unicode, instead of just doing its best to display mangled bits. (Many terminals would actually just display it correctly, if it was "mangled bits.")

That is, it'd be easier if strings were just considered "bytes" or always utf8.

I do know Python 3000 will fix it, but I think it's too far off to serve our needs, unfortunately.

As far as the error messages, I agree that the tracebacks are better than Java's. But sometimes "syntax error" and a carrat pointing to the place is hard to debug. (Often it's a cinch, but once in a while it's tough.)

Bugzilla is almost certainly as secure as Zope, although of course they're completely different things.

My concern isn't the security of the framework, it's the security of the code written on top of it. Taint mode absolutely prevents SQL Injections, or at least points out with a hard failure every place they could possibly happen. It also prevents running system commands that could damage your system. To me, the lack of Taint is actually the thing I'd be most uncomfortable about if I were to start moving to Python.

And yes, I have seen Python code be much shorter than Perl code. I re-wrote one thing from one language to the other, and it did get shorter and easier to understand.

-Max

The webhoster support factor

In case I didn't overlook it, you forgot the webhoster support factor. Afaik any scripting-enabled package speaks PHP and Perl but not all come with Python, Ruby or other languages. (See All-Inkl.com [1] for example.) While I'm not sure all required packages can be installed on such a host without shell access, this could be an important factor for Bugzill-ability of webhosting packages and therefore number of Bugzilla installations.



Sebastian

[1] http://all-inkl.com/eng/index.php?cna=webhosting&cnb=privat

Re: The webhoster support factor

Yeah, it's true that there are a lot of webhosts that support PHP, at the least. With Perl, it's a toss-up as to whether or not they have all the modules installed and whether or not they'll install new ones for you. :-) (And whether or not they have gcc installed so you can compile any XS ones in the local directory.)

I haven't ever looked into whether or not webhosts support Python, but I can certainly imagine Ruby is less common.

-Max

(Anonymous)

Perl module dependencies

Hi Max,

I'm going to single out just one issue you bring up: Perl CPAN module dependencies. I haven't read *all* of the comments, but I saw at least one mentioning a link to a "Catalyst in a box" proof-of-concept package as a solution to dependency issues.

Similarly, it's quite possible to package a Catalyst application as a so-called PAR package, akin to Java's JAR. If I recall correctly, Catalyst has some script that can make this a breeze, but if not, you can experiment with the "pp" utility that comes with the PAR::Packer module. Given the appropriate amount of effort, it can package just about any Perl application either into a stand-alone executable or as a .par. I've successfully done this with an application of about 150k lines. Also, I know that dotReader (dotreader.com) is packaged using PAR::Packer because they've contributed back to PAR.

If I happen to stumble across an excessive piece of spare time, I'll try packaging Bugzilla for the heck of it.

In case you're interested in further details, just drop me an email. I'm the current maintainer of PAR, see http://search.cpan.org/~smueller for an address. Just note that I might not be able to answer in the first two weeks of June.

Best regards,
Steffen

Re: Perl module dependencies

Hey Steffen! Wow, good to hear from you here! :-)

Yeah, I've heard great things about PAR, but I haven't had the time to look at it. If you wanted to help us out with packaging dependencies for Bugzilla, that would be great! :-)

I'll send you an email.

-Max

(Anonymous)

Hi!

I've used Bugzilla for six years and switched to JIRA about half a year ago. I don't know whether you consider JIRA a competitor since it's commercial and you are open source. Anyway, from what I can see, out of the box, JIRA wins hands down on concept, usability, speed (I run it on the same box as Bugzilla, against the same MySQL database), convenience, extensibility etc. The funny thing is: None of that is principally out of reach for Bugzilla, but it seems that the Bugzilla team isn't really aware of it.

My point is: Think about the concepts and abstractions in Bugzilla before making Bugzilla "just a bit faster". I understand this is hard, given the user base, and I'm happy to share with anybody from the Bugzilla team being interested in what I think is missing in Bugzilla.

Karsten Silz
Yeah, I definitely consider JIRA a competitor, and I'm totally aware that it has some advantages. :-) I don't consider it a competitor in the open-source space, but it's a competitor nonetheless.

I'm not unaware of any of these things. Atlassian is a professional software company where a team of developers work full time to create products. Bugzilla is a volunteer project where people work as they can.

There is nobody concerned with making Bugzilla "just a bit faster" (except in a few situations where it's not "just a bit" but "a huge amount that makes the difference between a large corporation being able to use Bugzilla or not"). There is a team of volunteers that wants to make the best bug-tracker that we can.

I think we're all aware of what is missing from Bugzilla. It's not a matter of awareness or stupidity, but a simple matter of time and resources.

Just as an example, there is a HUGE demand for User Interface designers in our industry today, and not enough supply. So that makes it really hard to find one to work on an open-source project for free. I have a decent understand of usability and interface, but I also personally maintain nearly all of the major parts of Buzgilla, nowadays, with continuous huge projects underway.

If you'd like to help, we'd LOVE that. But if you think we don't know what's needed, you should look through the thousands of open enhancement requests against Bugzilla, many of them filed by me or other core developers.

-Max

(Anonymous)

Catalyst. No!!!

http://readlist.com/lists/lists.rawmode.org/catalyst/0/1377.html

I've found moving from mod_perl to Ruby on Rails one of the best things I've done this year. I did, however, give consideration do Django as well. Being a Perl guy, I simply found Ruby more to my liking.
Thanks for the info! Yes, Catalyst has lots of CPAN dependencies. :-)

You might want to check out Pylons as well. It was originally a port of Rails to Python, but it became a lot more full-featured than Rails as time went on, as I understand it.

-Max

(Anonymous)

java fast?

I crack up every time I hear this one. I've developed in many languages and Java is by far the slowest one. The one thing Java is great for is swing, applications like Eclipse, SQL Developer ... etc. These applications are however the only ones that max out my processor consistently.

Page 2 of 2
<<[1] [2] >>
To The Side

November 2010

S M T W T F S
 123456
78910111213
14151617181920
21222324252627
282930    
Powered by LiveJournal.com