Why do software projects fail? is not a philosophical question in Descartes’ sense, that is one that can be decomposed in order to arrive at a practical solution. Rather, it is a contemplative problem in Aristotle’s sense: one that can be looked at in many ways in order to reveal something about ourselves.
Software projects fail because we are weak. We can make software projects better not by becoming stronger, but by minding our own weaknesses. The truth is that software programming has made a lot of progress over the past twenty years. It has not made it along the via positiva, however, through solutions such as OO, or Design Patterns, or Agile, or SOA, or a host of other white knight solutions that over time have turned out mostly to be hype.
Rather it has progressed through the via negativa, or by learning what not to do. The via negativa was originally a mode of theological discourse, mystical in essence, which strived to understand God not in terms of what He is, but in terms of what He is not. This mystical path serves well in pragmatic matters, also, and below I will outline some principles of the via negativa as applied to software development.
I should include the caveat that I am not sure these principles have ever worked well for me, and that I continue to underestimate the scope of a task and the amount of time it will take to finish it. On the positive side, I believe it all could have all been worse.
The Via Negativa of Software Development
1. Programming is neither an Art nor a Science
The thing I like best about Hunt and Thomas’s book The Pragmatic Programmer is their attempt in the first chapter to analogize the act of programming to the Medieval journeyman tradition and the concept of craft. I liked it so much, in fact, that I never bothered to read the rest of their book. The point of thinking of programming as a craft is that we escape the trap of thinking of it in other ways.
Many programmers think of coding as an art, in the romantic sense. Every project is new and requires a new solutions. The solutions a programmer applies constitute a set of conceits that express the programmer’s own uniqueness. Rather than applying mundane and known solutions to a set of programming problems, the artist coder prefers to come up with solutions that will express his originality.
On the other side, some programmers think of coding as a science. There is a set of best ways to accomplish any given task and their role is to apply that perfect solution.
Between these two is the notion of programming as craft. There are a set of ways of doing things, such as retrieving data from a database or displaying them for data entry. The craftsman’s goal is to accomplish his tasks in the ways he knows how to do it, whether they are the best or not, with as little waste as possible. The craftsman’s motto is also Voltaire’s: Le mieux est l’ennemi du bien.
2. Don’t Invent when you can Steal
A corollary to the craftsman’s motto is not to invent what you can steal. One of the biggest mistakes that developers make is to assume that their problems are unique to them. In fact, most problems in programming have been faced before, and developers are unusually generous about publishing their solutions on the Internet for others to use. Good design should always involve, early in the process, using Google to find how others have solved your problems. This is somewhat different from the Patterns frame-of-mind which assumes that there are common theoretical solutions to recurring problems. What you really want to find are common implementations, specific to the language and platform you are using, to your programming needs.
This was a well-known and commonly applied rule in the early days of radio. Comics would freely steal jokes from one another without attribution, and occasionally would complain about how many of their jokes were being used on other stations. What the radio comics knew, and we need to relearn, is that the success of the punchline depends not on the setup, but on the delivery.
3. Smells are More Important than Patterns
Software changes too quickly for patterns to be useful. By the time patterns are codified, they are already obsolete. With experience, however, also comes a knowledge of what doesn’t work, and it is rarely the case that something that hasn’t worked in the past will somehow magically start working in the present.
A smell is a short-hand term for things you know will not work. While Richard Feynman was known as a gifted physicist at Los Alamos during the Manhattan Project, his peculiar talent was in intuiting what not to do. He would go from project to project and, without knowing all the details, be able to tell those involved wether they were on the right path or the wrong path. Eventually, his intuition about such matters became something other physicists respected and relied upon.
Feynman had a natural gift for sniffing out bad smells, which he couldn’t completely explain himself. With experience, we all come to develop a good nose for smells, which manifest as a sense that something is wrong. The greatest trick is to trust our respective noses and take smells into account, so that smells become identified as risks which deserve to be monitored.
Here are some common smells that emanate from doomed projects:
- Taking process shortcuts
- Too many meetings in which nothing is accomplished
- Custom Frameworks
- Hysterical laughter (oddly a common phenomenon when people start realizing there is something wrong with the project but are not willing to say so)
- Secrets between development roles
- No time to document
- No project plan
- No hard deadlines
- No criteria for the success of a project
- No time for testing
- Political fights between development roles (these are typically a symptom of a bad project, rather than a cause)
- Managers who say that everything is on track
- Managers who initially were setting themselves up to take all the credit are now positioning themselves to avoid all blame
4. Shame Drives Good Code
The most successful innovations in software development over the past twenty years have been shame based. Pair-programming works because there is always a second set of eyes watching what we are doing. It is difficult to take shortcuts or ignore standards when someone else is likely to call one on it almost immediately.
Code reviews are a more scattershot approach to the same problem. Given that a code review can treat any piece of production code as a topic of analysis, it behooves the programmer to add comments, break up methods, and a host of other good coding practices in order to avoid having other developers point out obvious laziness.
QA groups are the ultimate wielders of shame as a tool to drive good development practices. QA brings obvious mistakes to light. A developer who might try to slip in bad business logic without first verifying that it works, on the self-assurance that it should work, is less likely to so if he knows his private malfeasance might be brought to public light. While QA is typically bad at catching problems with deep programming logic, they can instill a sense of shame in developers that will lead them to verify and thoroughly unit test their deep logic as they would test their shallow logic, knowing that others are watching.
5. Manage Risk, Not Progress
All the points above are programmer-centric. Manage risk, not progress is an essential rule for managers, project managers and business analyst. Because programming tasks can be shifted around, it is common to put off difficult problems till the end, simply because one can. This makes project plans, which measure all tasks with a common rule stick, notoriously bad at predicting the success level of a project at any given time.
Measuring risk is a better way to determine the success of a project. To my thinking, identifying risk and determining whether risks have been overcome is the chief role of a good project manager. If he spends the rest of his time surfing the Internet, I couldn’t care less. Unfortunately, many project managers insist on reading self-help books about leadership and interpret their role to be one of offering inspiration to others. They often view their roles in this way to such a degree that they tend to refrain from tracking risk, which is always a downer. And the best way to avoid tracking risk is to never identify it in the first place.
6. The Code Must Flow
Code must be treated as disposable. It must be treated as a commodity. It is a common feature of coding to treat every piece of code as special. On the other hand, we know that it isn’t true when we review our code months later. What is truly gained in coding through a problem is not the physical code itself, but the knowledge of how to solve the problem.
By constantly coding through every phase of a project, we gain knowledge of the problem domain. By disposing of our code when we know it doesn’t work, and starting over, we learn not to make fetishes of our code.
The second worst programming practice is when a prototype is turned into a real application. This is a problem for developers. It seems like it will save time to simply build on the same code that we started prototyping with, despite the fact that it fails to implement good practices or commenting or any of the other things we have come to expect from good code.
The worst programming practice is when management asks how long it will take to release a prototype, and demands the code be sent out as is. There is no answer to this worst of all programming practices, for while nothing can straighten out the crooked timber of humanity, poor management can always warp it further.