Waterscrumfall Paradox
Problem Statement
Many people face nowadays the paradox of agile development practice - it is not being accepted by project management/business leadership (besides periodically made fashion statements). This is because executive leadership typically thinks in terms of when things need to happen. So, they ask to estimate development efforts with a belief that it helps to place projects onto the timeline. Estimating effort on any kind of novel development is not a scientific process unless you can produce very detailed information about the system's composition and the tasks. The latter is not [supposed to be] known in advance in any Agile development methodology including Scrum. As a result, we end up with a hybrid approach. It is pretty obvious that any development related to customers' or business partners' commitments involves delivery dates way in advance of project maturity. It should also be realized that such delivery dates have very significant uncertainty (we are cycling back to the primary reasons for introduction of agile practices!). The essence of the Paradox is that business follows "agile" de facto even though it believes that it plans ahead. Instead of accepting the truth of fundamental uncertainty, business sponsors push the uncertainty down to the project's end. Correspondingly, commitments get weaker, because nobody wants planning to fail. Practically every PMO understands the paradox (if they truly don't, think about refreshing or training your staff).
Path Forward
We use delivery commitments to push teams, but there are only few levers that can be effective. We don't want to sacrifice product quality. Adding resources to late projects makes them later. Wearing off your teams may work for some time and in startups, but we have sworn years ago to mature away from heroic development.So, where do we go from here?
Business leaders should probably start from stating the truth: we don't know when the projects will be finished. No "but". We need to maximally decouple our R&D process from commercial commitments. The phrase "fast to market" is grossly misunderstood. Fast to market with a bad or wrong product has two very negative implications: 1) customers don't want your product or cannot use it; 2) competitors get an advantage to learn from your mistakes. Don't get me wrong - there are cases when "fast to market" is a must. But we also see proof, again and again, that delivering better products is more important than being there first. Therefore, if you want to create long-term value, discard the "fast to market" approach and focus on your ability to deliver consistent quality and function.
After business sponsors grieve and finally accept that there is no meaning in long-term commitment, it is time to define a minimal product. It may or may not be valuable, but it should be deliverable. Work on that product without making business commitments. Drive it through internal milestones. Deliver it. Then, define the scope and commit to the first "commercial" release. The predictability should be significantly higher at this point, because you are committing to an incremental change. Most major risks should be retired by that point or, at least, well understood. Collect market and internal feedback, correct the course, commit again. Repeat.
It Is All Mental
The most interesting thing about the WaterScrumFall Paradox is that it is all mental. Your PMO may think that they drive something, but they are really not driving that much. There are two major sets of factors affecting your development quality and timeline: 1) organizational capability (including people and tools); and 2) system/product complexity. Neither of these two is controlled by project management, at least, not directly. Your mission, business culture, and how you treat people determine quality of job applicants. Product design and development approach contribute to complexity. All those long discussions and meetings about product timeline is not more than some kind of collective meditation. It should not come as a surprise then that real timelines end up with some variation of the [natural] scenario described above: the first delivery is not a valuable product, but a base that can be used to deliver a valuable product.
Do we really have to do all the dancing then?
In general, if you know what you are doing and your product does not have significant unknowns, Agile methods may not give you any benefits. OS development would be one of such cases. Given. However, I was trying to address a different problem. How can we help business to be confident in its investment and yet give system developers time and resources they need to do good work? Great number of people want to know the answer "are we there yet?"
Pure Agile won't work for everybody, even for software development. If you're building a new OS, for example, you want to have long cycles, plan releases, have at least 1-yr roadmap, go through requirements-design-develop-test cycles. There also might be a huge documentation effort. Waterfall is quite good here. If you're building a web application - the Agile is perfect. What if you are building a Web App on top of your own OS? (This doesn't happen much, but a good example is Amazon's cloud platform - they have a bunch of simple services as well as things like RedShift). In this case you need to have different cultures in different development groups and you need to balance between the two.
By the way, Alex, I really like recent release of scalable Agile framework (v4, I believe). Our type of business (medical device) is not covered there in significant detail, but many things we can adjust the process. In particular, we all realize that there is really no such thing as partially built infusion pump, but there is definitely an opportunity for partially built electronic health record system. Even in the case of a pump, the design should foresee incremental improvements, so it should be based on the idea of a device platform rather than a one-off.
So lets get our business partners to read "Lean Startup" book by Eric Ries, lets explain them what Scrum really is Empirical process for developing products and lets start applying Empirisism in real life! If you are a scientist - you will get empirical process naturally, cause it is based on experiment. If you are an engineer - this is where you might struggle. At the end of the day, when business schools start teaching and widely adopt the next evolution of scientific management - business prosective will change. But so far, they have not widely adjusted their curriculums yet. And we still have to deal with "projects", "delivery", "output" instead of products, ideas/knowledge and outcomes.
Lots of good points here. I think there are a few books that help illustrate how/why added resources are not necessarily effective. The first one that comes to mind is “The Mythical Man Month”. It lives on my bookshelf at home, and has on the last few successful companies I have worked at. My old team(s) actually have awards for SCRUM/Agile. For software it is important to implement TDD/BDD to verify that actual results and quality are legitimate. You never trade off quality for making deadlines. Two quotes to remember here: “Shipped crap ALWAYS comes back”. “You always have time to do it right the second time” I encourage my teams to write tests first. It eliminates gold plating, and ensures what they are working on in meeting user stories / conditions of satisfaction / and acceptance criteria. Workflow paradigms should come from the top, but in the case they do not, results speak for themselves, always. Product quality should never be a trade for deadlines. The sad fact of life is that deadlines will be missed. Hopefully, your sales engineers have at least some magic to show off in that case. The good part about it is the whooshing sound as those deadlines pass you by. For estimating, in my experience, it is better to over quote the timeline and heroically best expectations than to under quote, and be the villain.