Software development projects are notorious for having a high failure rate. In the context of this paper, “failure” is defined as, “not meeting the project sponsor’s expectation and/or stated requirements”. This would include such things as failure to function in the intended way as defined in a requirements document, not obtaining the required performance standards, going so far over budget that the project is canceled, or incurring so many bugs that the end-users view the system as unusable.
I began programming business applications twenty-nine years ago. In that time I’ve worked as a systems support engineer, developer, solution architect, director of development, consultant, trainer, and CEO of a software company. What I’ve learned from these years of experience is that projects fail repeatedly for a very short list of reasons. This paper will identify those key points of failure and offer simple guidance on how to avoid them – I say simple because to adequately cover all of the ways to solve software development problems takes volumes of books.
1 – Requirements
Many, if not most, companies have a natural history in the migration of their data storage, workflow, and reporting processes. The typical path of transformation is to go from paper, to spreadsheet, to database, to sophisticated business application. During this transformation, which often occurs over many years, the terminology and workflow process that were used when the business operated on paper often gets carried over to the spreadsheet. Business jargon and processes are established around how the business needs to operate under a paper-based system and continues after the company migrates to a spreadsheet-based system. This repeats itself again when adopting the database-based system, and so on.
The problem with this is that once a company has finally matured to using a fully capable business application for streamlining workflow processes, expanding the businesses capabilities for analyzing and reporting on business data, that system’s full capability is rarely realized. This is not due to the inability of the technology or the programmers creating it, it is typically caused by the business not being properly analyzed when preparing the requirements.
All too often, the internal sponsors of the project, end-users, business analysts, and other domain experts, are often in too much of a time constraint to meet milestones imposed by a Project Manager or Business Manager. Thusly; the project misses a truly golden opportunity to realize a much higher ROI on the system, greater productivity increases, longer life of the system, and better suitability for the way the business currently operates.
Here’s how you might resolve the problem:
- Advise/enlighten the PM: Let the PM and the project’s stakeholders know of the consequences of not evaluating the workflow process and domain terminology sufficiently.
- Document the cost of needing to rewrite a system: A rewrite in only a couple of years, or worse, never getting the system launched at all, compared to the extra time to conduct a proper analysis needs to be reviewed during the initial planning of the project. Engage the business analyst and/or architect to help with this as early in the process as possible.
- Question conventional terminology. Create a dictionary of the domain’s “Ubiquitous Language”. Challenge each term and its meaning to each stakeholder, sponsor, or end-user. In other words, requirements gathering is more than just collecting nouns and verbs.
- Work with a Domain Expert: A domain expert – versus everyday end-users – can analyze business processes that need to improve and how the system can accommodate that. Don’t just assume the data set tells the whole story about how it is used. The business analyst, or domain expert, must have a solid understanding of your business, not the technology to be used to serve it. Again, this should be done in collaboration with the architect.
- Create simple to understand user stories: Good user stories are short, precise, and limited to single actions. They should clearly state who, what, and why for each action the end-user or the system needs to perform. Don’t create elaborate requirements documents that obscure the intent of the requirement – it’s the old adage of, “can’t see the forest through the trees”.
2 – Translation of Requirements to Technical Specifications
The biggest “hat trick” in developing software is taking business concepts, which are often rather abstract in nature, and then converting them into very literal, concrete technical specifications. Many times the context of the business processes are either not understood by the programmers or, not accurately translated into the technical specifications and ultimately into the code of the system.
The problem with this is that you have business people producing the requirements and technical people making that translation. Unless the technical person has a true understanding of your business and, its business concepts, then the translation will most often be wrong. Unlike translating two languages with Google translate, where a person can guess at the meaning of words not translated correctly given a specific context of the conversation, a computer application cannot. Concepts, processes, actions all have to be very specific in order for the computer to process it.
Many development companies assign the task of making this translation to programmers. This is inherently flawed as programmers are dealing with the finest details of coding rather than the higher level, abstractions found in business. Bridging this gap in concepts and level of detail is nearly impossible to do well and, often time produces catastrophic failure in the project.
This is witnessed by observing the code and comparing it to the user stories. Often time the code combines multiple unrelated user stories into the same file, making it all but impossible to understand, modify, extend, verify, or maintain.
Another observation is that the code will be missing complete concepts derived from the domain experts and will be accommodated by a lengthy bit of code that works around the concept rather than articulates it. Examples of this would be where there are well used, common terms of the business, which relates to either specific data or specific processes that are real-world things in that particular business domain. When reviewing the code, it is common to see none of these terms used, but instead, replaced with technical jargon, arbitrary abbreviations, or worse, single letters. This makes it difficult to impossible to know if the code is truly matching the requirements. Even if the end-user functionality seems to be there and working, it doesn’t mean the code was constructed properly. What this can lead to – and almost always does – is that there is a high probability that while the first iteration of the system might seem to work fine, when the business wants to extend a feature’s capability or, add new features, the foundation of the code just won’t support it. I cannot count the number of times either I or other technologists have had to advise the client, “A rewrite is required”.
Most companies attempt to solve this issue by including a business analyst and/or solution architect on the team.
The responsibility of the business analyst is to be a domain expert and know how to correctly document the requirements of the system in a way that technical people can understand.
The role of the architect is to take the requirements and model a system in a way that illustrates a clear understanding of the requirements to the project’s stakeholders and a clear technical framework to work within for the programmers – thus, the “hat trick”.
The problem with this is that most companies think of a Solution Architect as a person who has been a senior programmer or technical team leader who, due to longevity (experience) needs to be promoted to architect as the next logical step. However, this just means that we are back to the start of the problem with a programmer, albeit highly experienced, making the translations. Unless the business analyst did a superb job in articulating the requirements in way that is easy for programmers to understand, there’s still likely to be the gap in concepts and understanding.
Rarely do senior technologists have a deep understanding of business concepts, and equally, business analysts a deep understanding of programming. What’s even more problematic is that programmers are almost never experienced in a domain enough to offer insight or suggestions into how to improve the current business processes or way the business data is organized. They rely solely on the details of the requirements and do not typically “think outside of the box”. This is extremely common with offhshore development companies.
The one difficulty regarding this is that it still takes a unique individual to truly excel at being equally capable in understanding business concepts versus technical concepts. Many people operate with only one type of thinking pattern, either they are very detail-oriented, methodical thinkers, the type that make good software programmers, accountants, and NASA engineers, or, they are creative, abstract thinkers who make good architects, artists, financial analysts, and marketing professionals. The latter can more easily see things from different perspectives where the detailed-oriented needs to make sure all the dots are connected. It is simply the altitude at which you view a problem.
Understanding broad business processes such as customer relationship management, or supply chain management requires a far more abstract, imaginative type of thinker than does determining how to construct complex securities trading algorithms in code.
Unfortunately this is a difficult problem to solve as there are not a lot of solution architects who have both a MSC and MBA in their education or, the hand-on business experience to augment their technical experience. However, this is really what is needed.
Here’s how you might resolve the problem:
- Educate architects in business concepts: Companies with internal development departments need to adjust their definition of solution architect and provide the educational and training opportunities for gaining the missing knowledge. Architects should take business courses in order to have a better understanding of how businesses operate. If the architect is working within just one specific domain, education about that domain from a business point of view is critical.
- Allow sufficient allocation of time and budget: Teams pressured to hit the PM’s milestones will always pull out their toolbox of “shortcut tricks”. This often leads to a serious degradation of the quality of the underlying code. Since PMs don’t usually see the code, or understand it, then, “out of sight, out of mind”.
- Engage the architect earlier into the process: Allow them to work side-by-side with the project sponsors and/or the business analyst from project kickoff to final release. This cross-collaboration will help each develop the skills to bridge the gap between the abstract and the specific.
- Make Domain Driven Design (DDD) an integral part of your development process. The DDD philosophy (not really a methodology or process) provides a set of design patterns and, an approach to acquiring requirements that can help bridge the gap for the highly technical person who needs to understand the business requirements. DDD explains the best methods for communicating with the stakeholders of a project. The style in which the technologist communicates helps them discover the deeper meaning of domain terms and concepts. This greatly aids in modeling a system that accurately reflects the real-world domain.
3 – Process
There are many processes used in software development, but for this paper, let’s simplify it to the following; Phased-Based, Agile, and RAD. Phased-based are those processes, such as the “Waterfall” methodology that puts an emphasis on a thorough completion of each phase before the next phase begins. For example, all requirements are gathered and documented before it goes to the design phase.
Agile-based are those processes, such as “Scrum”, which perform all phases on small chunks of the requirements in short iterations, typically 2-4 weeks long, and then repeating through enough iterations to produce a final product. For example, requirements, design, programming, testing for a subset of the requirements would all be performed in one 2-week iteration.
RAD (Rapid Application Development) is a methodology that focuses on end-user needs, typically articulated through user-interface mockups and data schemas. There is little need for understanding the business domain or the workflows. This methodology is intended to produce working software as fast as possible.
Many technologists will profess one method is better than another; however, it really depends on the nature of the project.
Phase-based: If you are designing a guided missile system for the U.S. Department of Defense, there is typically a much greater risk due to project failure, a bigger budget, and more time flexibility given to the project. The more important aspects of the project might be quality, capability, and assurance of completion. For this type of project, the phased-based is often more suitable as it provides more details upfront so that PMs know what milestones to establish, resources required, and budgets.
The drawback to this method is that if, you don’t get one phase correct, there is a cascading effect and magnification of the problem, thrown over the fence to the next phase. It also is not very accommodating to changes mid-stream.
Because it is very difficult to know everything upfront, before coding begins, this methodology has a very high failure rate.
Agile: The various Agile methodologies have great advantage over the Phased-based when applied to the majority of business applications since most business applications have a moderate amount of complexity compared to guided missile systems. The greatest advantage is that the Agile, iterative style of process is highly accommodating to change. Typical businesses don’t have the time to go through the extensive analysis phase that the Department of Defense has; therefore, most projects start with an incomplete, or inaccurate, set of requirements. The business sponsors and/or programmers often discover missing concepts, need for additional features, changes to features, or changes to workflow processes as coding progresses.
Agile processes embrace this reality and, due to the short-cycle nature of the process, it allows for incorporating these changes into the project without having significant impact on code already generated.
RAD: RAD has been around a long time and there have been many tool development companies attempt to create a single tool that does everything very quickly. Create the user-interface, the database, and the business logic with the least amount of hand coding.
This method of software development is fraught with problems. First, it forces business requirements to be implemented within the tight confines of how the tool generates code.
Few RAD tools, if any, accommodate good coding practices such as best practices for object-oriented programming or ease of testing code. Most tools inadvertently promote the lumping of large amounts of code together, which is doing many different things – this breaks the object-oriented nature of the code. While RAD development style is very quick and can produce a working feature almost immediately, it is far more difficult to easily add new capabilities, find bugs, or be maintained later on after the code has been deployed than properly written code.
Applications that are best suited for this style of development are those applications that are mostly just database systems where data is entered and later queried for review. These systems might have little to no real business logic other than some data validation. In addition, the simpler the data structure is the more suitable this method is. In addition, applications best suited for RAD are also applications with far less need to change significantly with new, or extended, features over its lifespan.
Software development teams, or companies, that do not have a well-defined development process, that is applicable to the nature of its projects, can easily get derailed in meeting estimated budgets and timelines.
In addition, taking a well-defined process, that has been proven effective in numerous other organizations, and making changes can compromise the effectiveness of the process.
An example of this is Scrum. Many teams or companies alter this process to fit their traditional practices. In Scrum, there is no role of Project Manager. While most all companies, prior to adopting Scrum, have always had this particular role, they seem to be unwilling to abandon it. The result of this, is it works against the Scrum process. This ultimately causes many companies, who have tried it, to abandon it saying, “it doesn’t work”. What doesn’t work is using a traditional PM in a Scrum process.
Here’s how you might resolve the problem:
- Understand each process type: If you chose to do an Agile process like Scrum, don’t do what is called “ScrumBut”, where you take parts out and put other parts in. These processes have been developed over long periods of time and in environments where they received a thorough adoption. Making arbitrary changes to shoe horn the process into your current conventions can completely eliminate all of the advantages of the process.
- Adopt the appropriate process for the particular type of project: The solution to adopting a proper process is to first understand the nature of the project. It is not a good practice to adopt one process and apply it to all projects, unless all of your projects are of the same type.
- Thoroughly educate your client on the process: This is particularly true with RAD and Agile as they are the least understood by business managers, project sponsors, PMs, and end-users. Make sure that the client fully embraces the process in both theory and practice. The client must be fully aware of their role within the process and, they must be prepared to exercise that role in a timely manner. For example, in Scrum, the project’s domain expert, business analyst, or project’s sponsor might be filling the role of Product Owner. It is their responsibility to prioritize the requirements that need to be coded for each Scrum Sprint. It is also their responsibility to be available for clarification of requirements. Projects can be seriously delayed or otherwise compromised should the client not fully embrace, or exercise, their role and responsibilities. Don’t be timid in setting the ground-rules for the client – internal or external.
4 – Programmer Skills
A vast amount of coding is being produced “offshore”. The term “offshore” is a bit misleading these days. Let’s just say it refers to countries that have relatively cheap labor rates for programming when compared to major European countries, Japan, Australia, or the U.S.A.
I have worked with programmers in China, India, Egypt, Thailand, Vietnam, Philippines, Lithuania, Denmark, Holland, Germany, England, New Zealand, Australia, and the U.S.A.. What is consistent with all of the countries that fit the definition of “offshore” is that the skill set of the programmers is far lower than that of their peers in the U.S.A., England, Germany, Denmark, Holland, etc. – the non-offshore countries.
I have concluded after working with hundreds of these programmers is that the problem originates in the universities of these “offshore” countries. There is very little taught in the way of advanced programming techniques, object-oriented programming, software design, design patterns, unit testing, etc. These universities primarily focus their computer science classes on the use of development tools and programming language syntax. Little is being taught on the ways to creatively and properly use the tools and programming languages.
Most of the graduates come out of the university, even with masters degrees, knowing only the RAD style of development, and a limited set of tools.
Here’s how you might resolve the problem:
- Establish an ongoing training program: This may take away 2-4 hours every week per programmer from billable/production coding time, but will pay enormous dividends in the long run. Focus on architecture and good coding craftsmanship.
- Continually improve the level of English: English is the standard language around the world and what virtually all technology books, articles, and tutorials are based on. I the programmer is weak in English, they only get a limited amount of knowledge transferred during the learning process. If the programming team does not have a solid foundation in English, including a lack of strong accent, it is problematic at best, to assure that requirements are clearly understood. High standards must be set in this area. This is especially true for European sponsors of project since in most European countries, English is also a second language and strong accents prevail. Many projects have either been difficult to get done well, or simply fail, because a German accent has great difficulty in beinbg understood by the Indian programmer, who also has a strong accent. This issue becomes critical when the development company has adopted a development process such as Scrum where the developerment team has much greater direct communication with the client, business analyst, domain expert, and end-users.
- Elevate your coding standards: There are many different versions of what good coding standards might be, but almost everyone would agree that few practice any of the versions well. My hot-button, and a red-flag indicating there’s a programmer skills issue, is the naming convention used in the code. If I see programmers using abbreviations in their code, then I know they have learned bad programming habits in school or on the job. I also know that they have not spent time investigating what good coding standards are. In other words, they would not be considered good craftsman of their trade. While there are many other ways to make this judgment, I have found that this one item is foolproof in telling me that many other best practices are ignored. In addition, if the naming of classes, methods, and variables related to the business domain, do not reflect its specific intent within its domain, then I know the programmer is very likely to lack a solid understanding of the requirements from the point of view of the project sponsor or domain expert.
- Make Test Driven Development a Religion: TDD is a development process mostly embraced by the Agile community. It is also a catch-22 situation when it comes to implementing it. With most organizations not fully on the Agile bandwagon and, still following traditional command-n-control project management conventions, testing is one of those things most often dropped from a programmer’s daily tasks when the project is pressured to meet deadlines by the PM. Development teams that can fully embrace the Agile, particularly Scrum, methodology, know that teams can achieve high standards for both quality and output when motivated with rewards on such items as test-coverage. High test-coverage usually translates to less refactoring, less bug fixing, easier extension, and easier maintenance. All of which contribute to the ROI of the project, the thing the PM should be most concerned with. It also has an indirect benefit of improving the relationship with the client should additional phases of development occur. This results from the tests making it much easier to add the new features requested. When the amount of time and expense is lower due to good coding and thorough test-coverage, that usually impresses them and motivates them to stay with you as a client.
- Follow the Domain Driven Design approach to fully understand the domain: DDD skills provide the programmer and architect methods to best understand the business through the eyes of the business person. This greatly increases the likelihood that key domain concepts will be accurately portrayed in the code. This provides a stable foundation of code to build upon as new requirements or changes to existing features come in. DDD improves the communication techniques between the business and the engineering sides of the project.
This was a short list of project failure points and some ideas on how to solve, or avoid, them. All comments to this paper are welcome. There are dozens, even hundreds, of other points of contention in a software project; however, these have been the most consistent ones experienced over the twenty-nine years of working at developmenting software.
Last bit of advice is that unless your organization is committed to solving the above mentioned problems with bold action and firm commitment, the probability of success is greatly diminished. Changing development methodology from Waterfall to Agile can be a cultural nightmare. For those who make the transformation when needed for the nature of their projects, the end result is astonishing.
To use a metaphor from rock climbing, if a rock climber’s techniques of climbing is inadequate, and he/she falls, if the process of belaying is also inadequate and only slows the rate of descent, the climber will still hit the ground with catastrophic results. Focus on the programmer’s skills and your organization’s processes and, that should reduce the risk of falling in the first place or, not arresting the fall altogether.