A total rewrite: costly, time-consuming, but worth it?

i'm surprised you decided to start from scratch. Upgrading your old system might have been messy to begin with, but you would be in a much better spot much sooner.

I'm sure there are situations where both viewpoints are valid. The trick is to evaluate a situation correctly so that the correct route is taken.

I fully understand where Vidar is coming from.

I have seen other situations where a full re-write has been successfully done - the most common case being that the application grew dynamically with the business itself over a decade or so.

Design principles or lack of them - from a decade ago, spaghetti code, User Interface design no longer being relevant and being tightly coupled with business logic, subsystems being devolved, small features taking very long to take to market - these are most common reasons why a re-engineering is justified.

Sometimes it makes no sense to putter along in this way. Every case has to be evaluated specifically - so I would not agree to Joel's categorical statement.

A risk analysis - with mitigation and contingency conversions to time and budget, a testing strategy, instituting a user group for decision making, prototyping of the UI, careful planning of schedules, resource loading - these go a long way in re-engineering projects.

Mac OS X is based on the Mach kernel and BSD, relying literally on decades of Unix development before and during NeXT times:

http://en.wikipedia.org/wiki/History_of_Mac_OS_X

Joel Spolsky has no Innovation Norway money in his pocket. A rewrite could paralyze the company and restrict cash flow. And what happens when customers understand that CMS 2.0 is right around the corner.. but not quite. Suddenly CMS 1.0 is selling less. Two products could result in twice the support burden. Data migration might become an issue. Some customers might refuse to upgrade. Developers with inside information might quit.

Webnodes decision is fine because Microsoft dropped support for old school VBScript based ASP. It is logical, future compatible and apparently financially sustainable. Joel Spolsky simply believes feature creep and new bugs will kick him out of his skyscraper office in New York. His operating margins are probably thinner than yours. He has to cover health insurance, etc.

http://www.joelonsoftware.com/items/2008/12/29.html

A rewrite could imply that you already have a development process problem. You are blaming the house for being crocked. Bad house ! Or you grew too fast and added to many features too soon. If so, a rewrite will probably produce a better product.

PS
Why is there no URL field in the comment form :)
http://labs.teppefall.com

There is a rule of software engineering that all programs are constantly being re-written. I've been through three re-writes of our product in six years. Every time we started, we knew that the new version was going to be revolutionary compared to the old version. It turns out that at the end of the day, most customers viewed the improvements as evolutionary. But that's okay because the new versions did improve significantly upon the previous versions.

So how did we successfully complete this feat not once but three times? A solid customer base and strong revenues. They were critical in supporting the R&D team and their endeavor. Without it, we would have sunk ourselves (and we probably did come close but I just don't know it).

Our reasons for doing the re-writes were the same as yours - technical obsolescence, unmaintainable code, and engineering mistakes. Each re-write was fun because we got to experiment with new technology, learn new things, and apply old lessons. At the same time, it was also quite taxing because of the pressure. Every software project takes twice as long, even when you plan for the extra time.

http://bitshaq.com

Actually, Joel redacted this quite early on and in fact when talking to Jeff Atwood changed his opinion openly.

Read this post from 2008: http://www.codinghorror.com/blog/2008/10/programming-is-hard-lets-go-shopping.html

I recall Dr. Alan Kay saying the Smalltalk-80 development team would re-write the Smalltalk kernel -- from scratch -- every 6 months or so in the late 1970's. In this way, all members of the development team were "current" with all aspects of the system, and the forced re-write from scratch ensured that salient and necessary features continued on, whereas less-suitable features got dropped. Not to mention, it also allowed them to refine their infra-structure for build/deploy/test/etc.

15+ years later, and my having worked in an enterprise-sized development shop where the sheer scale of the system, coupled with an increasing pace of body-shop assignments, code rot, forced-re-engineering of existing code under pressing customer escalations, multiple-branch code streams with significant numbers of backports and other issues, etc … there is a significant attraction to the notion of re-writing from scratch, setting aside a budget for a 5 year "leap frog" team whose mission is to do exactly that … and then take over the production release for the next 5 years while another team "leapfrogs" to the future system.

Although rewriting code is essential, a FULL rewrite indicates deeper problems. It indicates an inability to rewrite offending code.

In my own development, I will rewrite any module and it's interface whenever I think of an improvement. This can break everything else that uses it, but usually only takes a few minutes to track down all those broken places and fix them to use the new interface.

The result is that the product very quickly becomes more and more streamlined until it is a piece of code I can be truly proud of. Something that is sleek, efficient, and easy to modify. It also allows you to recognize areas where more code can be reused, as often the rewritten code is shorter and sweeter, leading to fewer bugs in the end.

If you're afraid to make changes like this for fear of breaking your application along the way, I believe you're destined to actually take LONGER overall simply due to working with a more clumsy base of code.

Bottom line: rewriting code SAVES time in the long run.

Mac OS X doesn't represent a complete OS rewrite - that would have been Copland.

Instead, the development manager brought in to oversee the (hopelessly behind schedule) Copland OS development recommended "Apple look outside the company" for a solution.

Openstep would be that solution - which itself had been evolving since 1987, where it began as a 'from scratch' effort.

Mac OS is a rewrite and extension of Mach and NeXT libraries to work on newer hardware.

Have to agree. We are a financial company. When I came on, there was a critical app done in Lotus using MySQL and all the data was in first notmal form, unchecked for data integrity and redundancy all over. The developer initially had no clue about Db development and had decided to handle his own primary keys, didn't understand foreign keys, and many other problems.

The front end was a spaghetti coded mess from a prepackaged solution that was so bad it took over a minute to load pages and the entire thing would crash daily.

The only solution was to rebuild not only the database and their structure which took a redundant database in 1st normal form to third normal form and reduced the size (by introducing relationships where their were none) from 2GB to 200 MB

Did anyone thank me though? No. Everyone ended up being pissy about the entire thing even though everything worked immensely better. They had no idea what kind of mess they had created for themselves and what I had just delivered them out of.

So sometimes, you HAVE to rebuild from scratch when the original team of devs build an app was so bad that the application no longer supports the companies needs.

The transition from OS 9 to OS X was even more frightening than the one from Windows XP to Windows 7, but there wasn't a Vista to put a public face on it. I can recall seeing an early demo of Pink (the first try at a re-write) in 1990, before the earthquake, but it wasn't until 2001 that they released OS X, derived from an entirely different code base.

For more details you can check out http://lowendmac.com/orchard/05/1026.html

I have been through a similar project - but the code I had to sift through was more aged - up to 25 years old in some cases.

More can be read over at Slashdot: http://developers.slashdot.org/comments.pl?sid=1661266&cid=32306026

I'm in the process of trying to get my company to let us rewrite our software. I will not be able to convince them of a full all at once rewrite because of the time involved and the lack of progress in our current code base during this process. The good thing is logic in one language is not so different from another in most cases. The rough part for us will be 3rd party GUI components that won't work and will have to be replaced. I just wish I could convince my boss that it is worth the time investment. We spend so much time now on upkeep because of the rats nest of code written by horrible long ago fired programmers.

A couple things here:
1) Everything you said about Mac OS is wrong. Every single sentence of it.

The only feature (I can recall) Apple ported from Classic to OS X that wasn't already in NeXT was the Labels feature in Finder, and they certainly didn't improve it in the process. Dozens of Classic features were never ported, and the entire spatial philosophy of the Classic OS was thrown in the trash.

I actually switched to Windows during that period. (I gave them until 10.4, but when it was obvious that spatial was gone, I jumped ship.)

2) So let me get this straight: the only reason you stayed afloat during the rewrite was government loans? And you're telling us that Joel was *wrong*? Sounds like he was spot-on to me.

To your point about developer experience, developers (well, here in the US at least) come and go. If you're relying solely on developer experience over a 2 year period to ensure the quality of the product is better than the previous product, you're gambling. Maybe it's a safer gamble in a country where you can get government loans to bail your ass out of dumb decisions, but here in the US it's more sink-or-swim.

I do agree that Joel's article is flawed, it doesn't list exceptions. (And I believe exceptions exist.) But it's just advice, not the bible... you're not supposed to blindly obey anybody's advice, you're supposed to make your own decisions.

But this example leaves very little to the debate. From my perspective, you've done nothing but prove:
* Joel is probably right
* Your company is very, very lucky

I've faced this problem a good many times over the years, even in fairly large volume. The only time I've seen a rewrite really produce a reasonably improved project — it was indeed not really a rewrite at all. What happened is rather a new back-end system was built to supply not the same, but a fundamentally improved functionality which was similar enough in nature to be controlled by the same control interface. The back-end system launched independently and through a variety of integration points ran in parallel to the existing back end, with new distribution partners integrating with the new back end. The next major project included rebuilding the user interface. In the mix, the new system was on an entirely different platform (it moved from an ASP/ASP.NET system to Java). This large, iterative cycle resulted in a far better product. And starting with a new back end meant the most important changes happened first, allowing the business to scale in ways that weren't otherwise possible.

I agree with the author. And I agree with some commenters. Deciding when to rewrite software is a delicate choice that depends on several factors.

Firstly, the reasons why a rewrite is a good choice:

1) Old system is based on old / legacy technology that is not providing the infrastructure necessary to grow the product (I have experience in this case where we spent 1.5 years writing a large rebates system - it was developed in 4GL/Informix and we rewrote it in J2EE/PostgreSQL - it was a huge success and on time but this was because we had fixed requirements and little scope creep, and a different lead developer)
2) Bad architecture choice initially, and due to experience you can rewrite it to be more flexible, modular and extendible.
3) Unmaintainable code written by unexperienced developers.
4) Bad platform choice - I have a client where I developed a sales leads system using Java swing as a rich client app. I warned them back in 2003 that the right way is a web based system. They did not believe me at the time and was worried about performance issues. 7 years later and we have endless trouble with the client/server version as people try to run it across a WAN, the architecture does not support firewalls that well due to Java's RMI dynamic port issues, and the performance was never optimised for WAN purposes. The architecture does not lend itself to further optimisation. So 7 years later and the client is ready to invest the time and money to port the system to a web based system. In this case the language stays the same, but we are moving from a rich client/server based model to a web based platform. Some code can definitely be reused (maybe 30%) as the whole backend was properly separated in a 4 tiered system. But as you know a lot of the work is in the front end and the user experience, so this has to be rewritten.)

There are several reasons not to rewrite code. Just jumping from say .NET to J2EE or vice versa, would not necessarily help. Jumping from assembler to .NET will if there is no performance benefits, as you would gain the benefits of higher levels of abstraction, easier code maintenance, rapid development, portability etc.

Like everything in life one cannot state these things in absolutes. It depends on context.

James Schend:
1) I never said they brought over all the features from OS 9. I said the best parts. You might think they didn't bring all the best features, but do you disagree that it was their intent to bring over the features they believed were the best?

A few people have commented on the blog post and said that Apple didn't write OS X from scratch. Well, I didn't say that either. I said start from scratch. It depends on how you define "start from scratch". No code, AFAIK, was brought over from OS 9 without changes. Yes, as I mentioned, they used Darwin/NextStep as the foundation for the new OS, but do you know many operating systems that haven't used any already existing components?

You could of course debate the degree they used parts from NextStep compared to other new operating system, but that's missing my point.

My point was that changing the platform/codebase can have a huge impact on the success of a company, which is why I believe we made the right decision to rewrite our application.

2) The support from Innovation Norway was basically tax credits for research. We spent time and money to reseach specific parts of the architecture, and the following year, we would get back a certain amount of the taxes we paid the previous year.

The company would have survived, and we would have been able to complete the project without the support from Innovation Norway, but the support enabled us to spend more time on certain parts that we and Innovation Norway thought were innovative. We hope and believe that those key strenghts can be the determining factors for our future success.

As for the experience of the developers, I believe that's a key part of the whole equation. The cost of loosing a good developer is huge. I know the labour market in the US is more volatile, and people change jobs more often than here in Norway, but there are ways to keep developers happy in a company, even in the US.

Vidar: Unless you're using an extremely basic definition of "features" (like: windows, menus, cursors), then no Apple did not bring the best features from Classic to OS X. Heck, Classic's best feature was its spatial design, which has never been present in OS X.

I think it would be a hard case to make that Apple's "intent" was to satisfy Classic users, since OS X turned out so different from Classic, and most unique Classic features has never been ported to it. If that was truly Apple's intent, then they've failed completely.

Obviously their intent was to make a good product. And I'm not arguing that OS X is a bad product; obviously it's not.

But Apple didn't rewrite Mac OS from scratch; they also didn't start from scratch. There isn't a single technology in OS X that didn't already exist in NeXT... all Apple did is buy a competing product (and their developers!) and re-brand it as "Mac OS."

As for retaining developers: No, you can't retain developers if you're broke because you don't have any customers because you haven't shipped a product in a full year and your competitors are eating your lunch. That was the point.

Your company was lucky; you were able to retain customers by getting government money (regardless of whether an accountant would call it a loan or not... whatever.) Or possibly because the job market for developers in Norway sucks.

Either way, that wouldn't work in the US or, I suppose, most parts of the world.

I think Joel is mostly right, but he presents his argument as black-and-white in a world that isn't. Sometimes burning the disk packs is a good idea: especially when one is dissatisfied with the architectural status quo: and if you think about it, we're talking about the very small overlap between research and business. Research is a very different kind of investment from what business people are accustomed to making, because the point of research is to learn, not to ship a profitable product. Research tends to have a very long tail in terms of return on investment. See the 40 years we've spent in industry failing to arrive upon the architectures which were discovered at Xerox. When you burn your disk packs, you're making an investment in research with the expectation that your research will inform a successful product -- recognizing that there's really two projects involved helps to explain the risks involved.

James Schend: The job market doesn't suck at all here in Norway. I'm 100% sure I could get a dozen job offers within a week if I tried. But I like the job, and I like to develop a software product and be part of the international product launch and help the company grow. And last but not least, the pleasure of seeing a lot of people using the stuff you've developed is very rewarding.

We've tried to hire developers several times during the last year, but the number of qualified applications we get is very low.

As for government money, I understand that it might be hard to get that in the US, but it's much easier to get investors in the US than here in Norway, so I think it's fairly even..

Whether a rewrite is a good idea or not depends. It's all in the planning.

You're a growing company soon to face scalability issues, new feature demands, new customer issues, new competition, and the like.

You have a software team comprising members with various levels of expertise and temperament. You have the owners of the company that probably doesn't understand or care about all the technical details. And you have managers whose necks are on the line pending the outcome of the project.

And there is NEVER enough time. NEVER.

Having a good plan will be the determinant of whether you will be a shining star or be broken. Having vision and foresight are essential. Having a good motivated team is as well.

To say it's a non-trivial problem is a gross understatement.

Also, another issue you are up against is that the legacy code most likely is poorly documented, and the original people who wrote it are no longer around. My last project was literally plagued with this issue -- scripts running on servers whose details were unknown -- yet everything would come crashing down if you switched off those servers! The scripts themselves spaghetti code, and in my case, really horribly written PHP code being activated by other scripts using lynx or elinks browsers to pass commands along. I kid you not.

But at the same time, there's a lot of business problems that were solved and implemented by the badly-written under-documented code. All sorts of unstructured wierdo crap that was essential to the business operation.

Been there.

Sometimes you simply don't have a choice but to rewrite. But to do so without solid engineering and planning is a big mistake. One writer mentioned that there's no guarantee that the rewrite will be better than the legacy. IT HAD BETTER BE, and you had better be able to SHOW that through your plan.

Joel was right and wrong. For Netscape, it was strategically bad but for Gecko, it was the most successful decision ever.

Not only did it break the monopoly IE had on the market with the now near ubiquity of Firefox, but it also broke open chances for other engines to come to fruition such as KTML->WebKit.

Sometimes it's necessary to bit the bullet and dig deep just that most business managers are neither trained or too risk adverse for the long-term strategic planning necessary for game-changing decisions.

Note how most car companies refused to rebuild the petro-burning car platform from its century of development, and it took some others billions of dollars and a decade of work to come up with a new sustainable hybrid system that totally broke those original companies' backs when it came to the PR and economic crunch.

Long-term changes are hard, but looking only at the short-term horizon is also a strategic mistake.

I'm happy that it worked out for you because you took a big risk. But the benchmark should really be - where would you be today if you had decided to not start from scratch but instead put all that time into a serious refactoring the existing code? My firm belief is that you would have years of time to spare. But that is based on 10 years of professional personal experience and nothing else..

"There isn't a single technology in OS X that didn't already exist in NeXT..."

*cough* Quicktime? Java? Quartz? Carbon?

"all Apple did is buy a competing product (and their developers!) and re-brand it as "Mac OS."

Oh, of course. That's why it was released immediately after Apple bought it...oh, wait, it took over 3 years of development, didn't it?

In the mid 90's I worked on a software system that was written in a DBASE-style database and tied closely to the Xenix operating system. Clearly we had no choice but to rewrite from scratch, since you simply can't migrate from a flat-file data management system on Xenix to a SQL-type data management system on Linux without rewriting. The current solution by that employer is based upon Java and MySQL -- completely different from what the mid 90's solution looked like. Was the transition difficult? Sure. But you gotta do what you gotta do. Obsolete is obsolete, in the end, and there simply isn't any migration path from dBase BASIC to Java SQL other than a complete re-write.

Bjorn: I agree that to determine if we did the right thing, you need to look at the result. But do you take a look at where we are today, or where we are in two years? The answer might be different.

We signed our first partner in Portugal yesterday. If the current trend continues, we'll have the same number of international partners in a few months as we did back in 2005.

"..the single worst strategic mistake that any software company can make: They decided to rewrite the code from scratch."

another example of bullshit blanket statements. so 2000! it is not hard to imagine an article stating that "single worst mistake in software company is to hesitate to make radical changes to the code when it is absolutely required". statements like these are meaningless and potentially harmful.

Al
visualcriteria.net

As an owner and also productmanager of a small company you can think of a very good solution and have a very sound productstrategy and developmentpath. None the less, business is dynamic so sooner or later the flexibility of the strategy will be tested by scenario's you didn't think of or developments in the business going a different pace as you anticipated (by immense success or the lack of it). This means that the design becomes less fitting for the real solution, it needs a overhaul. There are 3 ways to deal with this:

1. Ignore it. The quality of the code will become a mess, maintenance becomes more a problem and also custom work on it for clustomers. It maybe more cosst effective on the short run, but customers will recognize that the product lacks quality. If you have a monopoly, it's maybe possible to pull it off, but don't expect much enthousiasm from the people in or outside the company. This kind of development gives me reasons to hate software development companies.

2. Rewrite from scratch, but with good research on standing bugs and required features, business cases, usecases and design first. It is very costly, but if you have a good design specification it makes it much easier to get it right this time (and if it wont the next time).

3. Rewrite from scratch important cores, every 6 months. This also means refining(rewriting) design specs. But after a while you will see that certain parts of the api's will be prove to become very stable and the product becomes very flexible (less complexity) and performant, making it easy to do bug fixes, to adopt features that clients really need and to weed out all the unused stuff. It may look expensive, but you earn it back in maintenance and competitiveness. Quality on the long run always wins. The short term is the same challenge to every small business.

I think this decision was not a 're-write' decision, but rather a 'change the technology' decision, and the reasoning behind it was solid. Although superficially there is a divergence from what Joel discussed back in 2000, in the end, I think both positions can be reconciled; Joel's real issue was re-writing because 'it is a mess'... which does not seem to be the case here. Well done on successfully managing such a change. I have been through at least one similar project and it was a financial disaster, and very nearly a strategic disaster for the company concerned.

I actually believe that the only way to achieve really high-quality software is to evolve it as far as you can, then rewrite it from scratch, with reference of course to the previous version. There are some things that can't be fixed by tiny evolutionary steps, you will need to redo the whole thing, perhaps several times, or at least make some massive large-scale changes to the system such as rewriting it in a different language or a different style.

I am a big fan of rewrites, but I will qualify that by saying that I mostly work on small projects. The kind of apps that start out messy in the first place, are developed in a mad rush, and then shamefully extended far beyond their means over the next few years. It is in those particular scenarios that I think a rewrite is good housekeeping, and it sounds like the situation Webnodes was in.

Ask any developer what the greatest flaw is in their biggest app, then ask them how they could fix it. Most of them will either say it's unfixable, or that it requires starting over scratch, with the wisdom they acquired over the many years of supporting and patching the old app.

If you, as a programmer, never revisit old problems with new solutions, you are stagnant. You are relegated to writing the same thing, making the same mistakes, every single time. Don't lie and tell me that, at the end of a development cycle, you didn't feel like there were things you could improve, or would have done differently to make your life easier. Most often, when I deliver a project, I feel the desire to start writing the v2.0 right away. It would be faster, easier to use, easier to integrate... if you go from start to finish without learning anything, you're doing it wrong.

Joel Spolsky's comment isn't wrong or right, it is simply valid from his perspective. He is a very vocal member of the community, but how many of you have actually used his products ? Joel talks mostly about how he is awesome and how everyone else is wrong or stupid. How often does he write about his failures ? Or should we believe he never makes any mistakes ? I personally cannot relate at all to his point of view.

I put far more stock in Jeff Atwood's words and opinions over at Coding Horror. Maybe it's because he mostly writes from a developer's perspective, not that of a boss. I don't know Jeff's stance on code rewrites, but I'd like to think it would be a more balanced approach rather than religious mantra.

It's a strategic mistake to decide to do a complete rewrite. However, sometimes we have no strategic decisions to make. If you have no choice but to rewrite and hope, then that's not a strategic decision. It's a last-ditch effort.

(Where do you folks figure "last-ditch effort" comes from, anyway? That's right -- when the enemy in a large dug-in battle crosses the field into range of your last defensive ditch, it's up to effort to save you because your strategy has already failed.)

Making the "choice" between having only an obsolete product you can't figure out how to rewrite incrementally and risking everything to rewrite it from scratch is only really a choice if you have the proverbial golden parachute. An incremental rewrite is the choice to make unless you just can't make that choice.