Why Scrum: Defined Process vs Empirical Process

There are projects that are fairly straight forward and relatively predictable, where rational tools of planning and decision making can be applied. Other projects that are more complex or unpredictable call for a different approach relying more on self-organization and innovation.

Most software development projects are considered complex and unpredictable in nature due to the convergence of three factors: people, requirements and technology. The various approaches used for delivering and managing projects could be more easily understood in the context of process control models and project complexity.

Defined Process — A Traditional Plan-Driven Approach

Traditionally, once a project starts, a requirements package is created and then is “signed off.” The project manager assumes that this sign-off results in a fixed set of requirements and that now planning can begin. The project manager estimates how long it will take to complete the requirements and creates the project plan. The plan predicts that the project will be finished by a certain date, and that date is communicated back to the customer.

The fundamental flaw in this approach is that the plan, which drives everything, is based on an assumption that the requirements are fixed and won’t change. Experience has shown us that this is never the case; requirements are never fixed — they always change. When the requirements change, the plan is affected; and as a result, the completion date needs to change too. Unfortunately, in many cases, that is impossible, and the team has to deliver by the date they committed to. This is when a major crisis occurs and the project starts to go out of control.

Empirical Process — Agile Value-Driven Approach

The value-driven agile approach is based on empiricism that switches the whole mindset. It assumes from the start that whatever requirements exist up front are not fixed and that they will change.

The agile mindset also assumes that you have to deliver by a certain date. This approach fixes the time and resources and leaves the requirements undetermined. To us, this approach more closely resembles the reality of creating software. Now the whole notion of value-driven makes perfect sense. When you have a fixed amount of time in which you aren’t sure whether you can deliver all the requirements (because they will change and hence the time needed to finish them will change), the natural reaction is to prioritize the requirements and finish first those that add the most value to the customer.

You may be thinking, “What about the requirements that aren’t finished by the delivery date?” That is the reason you use the value-driven approach. You acknowledge the fact that not all of the requirements will be completed by the delivery date. The important question to ask is whether you have delivered enough features to support a system that provides value to the customer.

Failure Rate of Software Projects

The 2015 CHAOS Report has recently been released by the Standish Group. The CHAOS Reports have been published every year since 1994 and are a snapshot of the state of the software development industry. This year the report studied 50,000 projects around the world, ranging from tiny enhancements to massive systems re-engineering implementations. This year the report includes an enhanced definition of success looking at some additional factors which were covered in previous surveys.

The results indicate that there is still work to be done around achieving successful outcomes from software development projects. This table summarizes the outcomes of projects over the last five years using the new definition of success factors (on time, on budget with a satisfactory result).

With the take up of agile development methods over recent years it was possible to compare project outcomes between agile and traditional waterfall projects. Across all project sizes agile approaches resulted in more successful projects and less outright failures, as shown in this table

What is the Problems of Traditional Approach?

The traditional plan-based approach isn’t flawed in and of itself; it just isn’t suitable for today’s software industry. The plan-based approach was originally based on traditional project management concepts, which originated from the construction industry. In the construction industry, the plan-based approach is suitable: the blueprints, which are the requirements, are fixed and probably won’t change while the building is being built. You can estimate how long it will take to build the steel pillars, pour the concrete, and so forth.

The reason why the traditional plan-based approach is suitable for the construction industry but not for the software industry comes back to the difference in the way we control empirical systems (like software development) and the way we control defined systems (like construction or manufacturing). The Table below shows the differences between the characteristics of a defined process and those of an empirical process.

After reading the table, it’s easy to see that software development is definitely an empirical process, not a defined process. The problem is that we’ve been approaching software development for years as a defined process — and that approach doesn’t work.

References

1. Standish Group 2015 Chaos Report
2. Becoming Agile in an Imperfect Word — by Greg Smith & Ahmed Sidky

Other Scrum Readings

• Scrum in 3 Minutes
• What are the 5 Scrum Values?
• What is the Evolution of Scrum?
• Classical Project Management vs Agile Project Management
• Why is Scrum Difficult to Master?
• What is Velocity in Scrum?
• What is Agile? What is Scrum?