The Tar Pit

The editor of TSS has decided to run a series discussing The Mythical Man Month, by Frederick Brooks. Hopefully it will produce some good discussion. There are a lot of Agile advocates that hang out on TSS that really could use to (re)learn some of the lessons of software development. I make a point of rereading it every few years lest I forget the lessons learned by computing's pioneers. The first chapter - The Tar Pit - contains on of my favorite concepts, as illustrated by the graphic below (slightly changed from original): Programs Let me explain. A program is what we all have developed. It's simple piece of software that is useful to the programmer and/to some set of users who are directly involved in defining its requirements. Most bespoke departmental and a substantial portion of enterprise applications fall into this category. They are sufficiently tested and documented to be useful within their originating context, but once that context is left their usefulness breaks down quickly. In addition, they are not solidly designed to be extensible and certainly not to be used as a component in a larger system. Obviously this is a range, and I've really described a fairly well developed program - one almost bordering on a programming product. That script you wrote yesterday to scan the system log for interesting events that has to be run from your home directory using your user account in order to work is also just a program. Programming Products Moving up the graph, we hit programming product. In theory, all commercial applications and mature bespoke applications are programming products. In practice this isn't really the case - but we'll pretend because they are supposed to be and I increased the standard over what Brooks originally described. The big challenge with programming products is that, according to Brooks, they cost three times as much to develop than simple fully-debugged programs yet they contain the same amount of functionality. This is why it's so hard to get sufficient budget and schedule to do a project right. The difference between a solid piece of software and something just cobbled together is very subtle (you can't tell in a demo) yet the cost difference is quite astounding. Consequently, I think most commercial applications are released well before they hit this stage, and bespoke ones require years to mature or a highly disciplined development process to reach this point. Programming Systems Programming systems are programs intended to be reused as parts of larger systems. In modern terms, they are libraries, frameworks, middleware, and other such components that are all the rage in software development. Like programming products, programming systems are thoroughly tested, documented, and most importantly are useful outside of the context in which they were created. And, like programming products, according to Brooks they take three times as long to develop as a regular program. Developing programming systems for bespoke applications or niche use can be a tar pit all its own. For one, many programmers like building libraries and frameworks. The problems are more technically challenging, and there is no strange-minded user to consider. The programmer and his colleagues are the user. Programming systems are relatively common in groups that execute a lot of similar projects and/or that contain programmers who really want to build components. Programming System Products Amazingly, programming systems products are relatively common - even if there really aren't that many of them. As you've probably guessed, a programming system product has all the traits of both a programming product and a programming system. It is useful to a wide range of users and can be effectively extended and/or embedded for the creation of larger systems. It has complete documentation and is extensively tested. Where are these wondrous things? Well, you are using one right now (unless you printed this). Your operating system is one. It both provides useful user-level functions and a huge amount of infrastructure for creating other programs. MS Office is one as well, because it has a pretty extensive API. Most commercially developed enterprise systems should be programming system products, because:

1. They provide off-the-shelf functionality for regular users
2. Customers always customize them
3. They often must be integrated with other products
4. Simply integrating their own components would go better with a programming system

The problem is that they are not, because of: The Tar Pit Brooks didn't explicitly write this definition of The Tar Pit but I think he would agree. Put yourself in the position of a development manager at a startup or in a larger company about to launch on a new product. On on hand, you want to make the product as good as possible. You know that what you develop today will serve as the base for the company/product line for years to come. It needs to be useful. It needs to be extendable. It needs to be thoroughly tested and documented... It needs to be cheap and delivered yesterday. The differences between a programming system product and a simple programming product are far more subtle than the differences between a program and a programming product. But the programming system product costs a full NINE TIMES as much to develop as the program with essentially the same "outward functionality" - at least if you are a sales guy or a potential customer sitting in a demo. I think this is the struggle of all engineering teams. If the product is long lived, doing it right will pay major dividends down the line. But it can't be long lived if it is never released. It stands a worse chance if it comes out after the market is flooded with similar products (actually, that's debatable...). The ultimate result is a mish-mash of tightly coupled components that, as individuals, fall into one of the lesser categories but as a whole fall down. There is a documented API, but the documentation isn't really that good and the application code bypasses it all the time. The user documentation is out-of-date. Oh, and the application isn't really that general - hence why all the major customers need the API so they can actually make it work. Escaping the Tar Pit Ok, so if you develop a large system you are destined to fall into the tar pit because cheap-and-now (well, overbudget and past schedule) will override right-and-the-next-decade. You need a programming system product, but budget and schedule will never support much more than a program. So how do you escape it? Accept It Products that give the outward impression of being far more than they are often sorely lacking in conceptual integrity. If you are building an application - build it right for the users. Remember you can build it three times for the cost of building a programming system product. Maybe by the third time there will be budget and schedule for it. Or maybe, just maybe, you can evolve your current system. But pretending will just make a mess while wasting significant amounts of time and money. Partition It Some pieces of your system are probably more important than others. There are places where requirements will be volatile or highly diverse amoung customers - those are the places where you need a truly extensible system. You should also be able to reuse strong abstractions that run through your system. The code associated with those abstractions should be top-notch and well documented. Other pieces just need to be great for the users, while a few that remain need to be convenient for yourself to extend or or administrators. Open Source It Find yourself developing yet another web framework because what exists just isn't right? Open source it. This isn't really my idea. It's what David Pollak of CircleShare is doing with the lift web framework for Scala (actually, I'm guessing at David's reasoning, I could be wrong). The infrastructure for your application is essential, but it isn't your application. It is what Jeff Bezos refers to as muck. You have to get it right, but it's distracting you from your core mission. So why not recruit others with similar needs to help you for free? That way you don't have completely give up control but also don't have to do it alone. Theoretically the same could be done for applications. Many large customers of software companies have significant software development expertise - sometimes more than the software companies. I think it would be entirely feasible for a consortium of such companies to develop applications that would better serve them than commercial alternatives. But I have yet to convince someone of that...

Sphere: Related Content

Is implementation important to a startup?

I'm not entirely qualified to answer this question, but here it goes. Taken from the Scala mailing list:

Amir Michail

On 9/2/07, Erik Engbrecht > ... > All-in-all I would say Scala would be perfect for a small team of > extraordinarily smart developers, but I wouldn't turn your average Java or > C# programmer pulled off the street loose on it. Not because they couldn't > write working Scala code, they could, it would just be a more concise > Java/C#. But I wouldn't start a company based on programmers grabbed off > the street (or leased across the ocean), either. > I actually don't think implementation is much of an issue for internet startups, particularly for Facebook apps. This is off topic, but I would be interested in knowing what you think of it: http://weblog.fortnow.com/2007/08/impact-of-facebook-platform-on-cs.html http://weblog.fortnow.com/2006/07/science-and-art-of-computation.html

So here's what I wrote (off-list) in response:

Amir, I'm not really qualified to comment on the skills required to write Facebook apps because not only have I never written one, I've never even accessed Facebook. However, I think one of the most effective ways of achieving long-term success for a business is to have a product that is in demand (obviously) and others find very difficult to replicate - or at least replicate to the point where people have difficulty deciding whether to use the new product or the old. The best way I can think of to do that is for the product to have an inherent internal complexity - meaning it requires either hard-core CS skills, substantial domain knowledge, or both to grok it. For example, Google is really an AI company. They don't actively publicize it (nor do they hide it), and I bet the majority of their employees aren't working on anything AI related, but ultimately that's what the company is - and that's why they'll stay ahead. Replicating the core of their business is simply too hard. It certainly could be done, but you'd need to get a sufficiently large group of self-motivated geniuses together who would both cooperate and have enough entrepreneurial spirit to not be lured away by Google. I'm not saying you can't build a successful startup simply by creating something cool and getting a lot of people to use it. You certainly can. In fact, I would say the vast majority of successful startups were built that way. But I also think the vast majority of startups don't have long-lasting products. They have products that fill a (possibly transient) need for a larger player and get acquired. They spend hundreds of thousands for maybe a few million dollars building something that would take a larger player >$10x plus months to replicate, not because the startup is better at engineering or product design - it may or may not be - but because the startup is 1 of 100 or even 1000 so the larger player would have to fund >10 initiatives a pick the somehow pick the best result in order to replicate what the startup did, and it can't do that because it can't release >10 similar products all at the same time without looking stupid to investors and confusing its customers. From what I've read about Facebook apps they are relatively easy to build. If you build a business around one, then essentially you are either trying to be that 1 in >100 that people latch onto long enough for a larger player to be acquired, or so are going to launch yourself on a never ending cycle of re-inventing your application as people duplicate it and trends change. Of course, that's not to say you can't have something really sophisticated at the core of your Facebook app. I'm sure you can. But if you do, implementation matters a whole lot. Implementation tends to lag theory by years if not decades. Your ability to implement is your competitive discriminator. As for the blog articles...well...I think the CS should remain more pure. CS is not programming, but programming is an essential skill for computer scientists (as I actually believe it is for most scientists, but in CS is is a much stronger need). So I think adapting CS to "Web 2.0" is a rather foolhardy exercise. By the time programs adapt we'll be on Web 4.0, so the programs will be behind the times and will have lost the theoretical goodness of CS. That's not to say those programs are bad ideas, I just don't think they should be labeled CS. It dilutes the brand, so-to-speak. I would say CS programs should offer far more electives that are cross-listed with other disciplines, such as business, economics, or the sciences. But these can't replace fundamental courses or even upper level technical electives, so they would need to be framed as to replace what my alma mater called "humanities requirements." -Erik

That's just my opinion, and to tell the truth my opinion has changed over the years. But this is closely related to the ongoing debate about whether a CS backrgound is or is not important for being a software developer. I actually think it is, but I also think a good CS program is overkill (in the CS theory and programming sense) for what most software developers do and grossly inadequate from a communications/requirements analysis sense.

Sphere: Related Content