Why Do IT Projects Fail? How not to.

All projects fail because of people for one reason or another. The projects start with hope, transform into a group delusion that no matter how bad it is... all will end well, and finally are deemed a failure when reality can no longer be ignored.

Having taken over a large number of projects that have failed and turned them around, each had unique characteristics but there were common themes.

1. Strategy (Schedule, Scope, Resources, Quality)
2. Requirements (Clarity, stability, consistency)
3. Communication between teams (Business, Management, Users, Technologists)
4. Technology (Architecture / Implementation)
5. Feature & Data Migration (HARD)
6. Training users / achieving adoption
7. Not finishing

Strategy (Schedule, Scope, Resources, Quality)

Expectations govern everything. There's little hope when the business has unrealistic expectations as to what will be delivered, when and by whom.

Best captured as the IRON TRIANGLE, in which scope, resources and schedule are set -- often quality is presumed to be set as well, but with nothing left to change, quality is relinquished as pressure mounts.

Big rewrites constitute the most extreme example in which an entire application is to be rewritten, revved and architecturally bettered. These are the most common class of failures I've encountered.

1. Didn't have time to do it right the first time.
2. Don't have time to or can't fix it in it's current form.
3. Rewrite the entire application, better this time, without losing functionality while increasing scope.
4. In less time than it took to write the original.

Rewrites should be avoided in lieu of fixing the existing application. In most cases this is possible, but occasionally the current application state and business need necessitate a rewrite.

Success requires the iron triangle to be broken to allow for flexibility.

1. Quality is inadvertently the MOST flexible dimension but it MUST NOT be. Low quality design / implementation is drag that will produce break-fix activity, reduce productivity and ultimately rework. Conversely high quality (getting it right) allows teams to get it DONE and move on to new things.
2. Scope is one of the least flexible dimensions of the iron triangle. Applying MVP (minimum viable product) help, but I assume this point to be relatively fixed.
3. Resources are inherently limited (mythical man month / orchestration overhead). Some limited (N) number of people can be brought to bear on a problem, but at some point the returns are diminished and often negative. Scale the team as big as helpful, but this dimension has a natural limit.
4. Schedule is the most flexible dimension though most initially view it as the most fixed.

Create as much flexibility across the dimensions as possible. Even Quality can be relaxed in certain business contexts, but be extremely careful.

Examples on how to relax dimensions: (All require explicit caveats & compensating requirements)

• Quality: Some features have more users, greater value than others. Administrative features might have degraded usability or in an extreme case, no UI, with the explicit requirement of highly trained or technical administration/users. Maybe you have a LOB (line of business) app and decide to not build any administrative capability in lieu of engineers articulating it via config files. There are degrees of freedom available (BE CAREFUL).
• Scope: 80% of the work is required to achieve the final 20% of the scope -- UI/Usability polish, power user requirements, defensive usability, auth/admin, edge cases, etc... Any and all of these will require the end user or business to accept the burden not accommodated in the implementation.
• Resources: Fewer high quality engineers can often exceed the productivity of many less skilled/experienced engineers. If the budget is tight hire right.
• Schedule: (my favorite lever) Often a project can be designed to allow for portions of the application to be written (rewritten) that provides value without the WHOLE being completed. As a project needs to be 100% before providing business value, the risk approaches 100%. Figure out ways to break it up so that incremental pieces can be completed while providing value.

Requirements (Clarity, stability, consistency)

Much can be done to create clarity -- prototypes, stage-gate, engineering involvement. Getting the requirements right in the beginning is one of the best ways to increase productivity by reducing rework. Getting users, business and engineering involved during the creation of requirements will result in better agreement / understanding.

That said, requirements are very difficult to get right. Often users do not understand what they need. Prototypes don't always illuminate issues or the path forward. Thus, though this should be given HIGH priority, realistically assume an upper limit that will still require significant evolutionary refinement.

Communication between teams (Business, Management, Users, Technologists)

Getting everyone on the same page, or Edward Deming's "Consistency of Purpose," is extremely difficult. Business leadership (often up to the CEO) must buy into the goal of the project and get agreement from all stake holders, users and architects. If the chain of communication breaks down, the project will suffer as isolated groups agree and develop features only to find a lack of acceptance elsewhere, which at best results in dissonance and rework -- at worst, failure.

Technology (Architecture / Implementation)

As a technologists, we often apply the latest "silver bullets" hoping for success, but great and terrible products have been built in terrible and great technologies. There is no silver bullet... only lead bullets (The Hard Thing About Hard Things, Ben Horowitz).

Good development / design / architecture is the ante -- the entrance fee to being successful. It's presumed. If you don't hire good technologists / designers / architects, then success will be elusive. After 20 years of tech, I've encountered a great basket of approaches, many I have believed in, and we all have our religions... Mine currently being mainly functional & immutable... But technology cannot succeed in spite of the rest... and so I'll leave it at that. Hire great engineers, trust them, then focus on the rest. Technology is the relatively easy part.

Feature & Data Migration (HARD)

Change is hard... Once something exists and over time, structures and behaviors are built around them. Users get used to features. Management creates KPI's around them. Analytics, ceremony, reports, behavior, hiring, budgets, businesses are built around them.

When features and data structures change, people must change... and nothing is harder.

Of the 2, features & data, features are far easier to change than the underlying observable data structures... Most rewrites (feature or application level) find data migrations shockingly difficult. Users, being creative human beings, find unique ways to use store data in surprising ways, which complicate data migrations immensely.

Fear the data!

Training users / achieving adoption

Once the hard work of migrating data/features is done, behavior change must be achieved through training... training... nudging... pushing... and ultimately achieving adoption. Human behavior is both extremely flexible and INFLEXIBLE.

Everyone does it differently... HIGH VARIABILITY.

People resist change... FRICTION.

People don't read user manuals. They don't read release notes. You can only succeed through: High usability, Simplifying features, adhering to existing mental models... but not everyone will be happy... be courageous, be bold, but be empathetic.

Remember & remind business, that not everyone will be happy. There will be friction, but you must manage to limit the amount so as to not create a FIRE.

Not finishing

Few consider this risk of NOT finishing, but I've found it to be one of the highest factors over time. Finishing a race is MUCH harder than quitting it. Quitters don't win.

The goal of the project must be achieved by completing it. If it is an incremental rewrite, it must be finished no matter how tired the organization becomes. The hell of failure is paved with half measures and partially implemented projects.

I have been on many rewrites that went far but not far enough. Data structures were partially migrated, applications nearly rewritten, architectures partially applied... this results in perpetual friction, work arounds and mitigating efforts.

FINISH!

Omissions:

1. Project Management (Scrum/Agile/...) - Project management can help or hurt, but for the most part they serve as a mechanism to track the project according to the goal / expectations -- the iron triangle. They codify and track to the path. At best they illuminate issues early; at worst they are antagonistic to the goal.
2. Technical architectures - I explicitly didn't get into the hairy details as tech arch is context sensitive. There is no silver bullet. The ideal technology depends on the environment, the resources, the existing code base, devops, etc... Whether you choose React/Redux ES6 immutable front ends backed by Elixir (as we are) or Angular1.x backed by Java, either might seem the right choice. I'll leave that to your technologists to determine. There are great, good, ok and terrible decisions to be made there as everywhere else.
3. Details : How to write requirements, do UAT, usability design, UI design... this is as much a high art as a science. Whether high fidelity, functional mockups tested against real users in usability labs, paper prototypes or build/UAT test live... that depends on the need, timeline and culture of your environment.