You are currently viewing Predictable Software Engineering

Predictable Software Engineering

Software production is infamously unpredictable. Releases are frequently late and deployed software fails to work. Lead time from feature request to making it available to customers varies greatly from case to case. That makes promising and planning difficult. It precludes rapid response to change and to feature requests.  It leads to disappointment of customers, investors and other stakeholders.

There is a way to make software production more predictable: it’s a combination of two principles: Continuous Integration and Delivery Automation. 

Continuous Integration is a principle of keeping in-development software ready for production at all times. If that condition is satisfied, a release can be cut at any time and is guaranteed to work. As a result, deadlines to delivering working software can always be achieved, even if the resulting software does not offer all the planned features. Continuous Integration is achieved by automatically testing even the smallest proposed change against a mock-up of the entire system before the change is accepted.

Delivery Automation is the principle of making the route from source control to production fully automated. It reduces unpredictability by ensuring that the process is strictly defined and timed and that the risk of human error is reduced. It makes the delivery process automatically testable, which is a prerequisite for Continuous Integration. It can make consistent, reliable records of the process, which offer a wealth of easily accessible information. Finally, it ensures that in-depth specialization is not required to execute the process.

What exactly is Continuous Integration

Software development has the concept of ‘trunk’. The trunk is the main development branch of a software product. It is the place developers submit their work to to share it with other team members and make it available for release. 

Traditionally, developers refer to the trunk as being ‘broken’ or ‘working’. A broken trunk means that the software in the trunk does not work properly. The major consequence of that situation is that a release cannot be cut. During that time, releases are delayed, deadlines are missed and promises broken.

It may also be the case that the code from trunk works on developers’ workstations or their development environment, but does not work in production. That kind of problem is often not discovered until late in a project. If it happens, a significant delay in delivery may occur.

If the trunk is not ready for production, it may take months before it is fixed again. During that time, it is often impossible to run a full functional test on the software from trunk. As a result, as long as the trunk is broken, new development work is either delayed, proceeds without testing or proceeds based on an old working version of trunk. Each of those choices has negative consequences.

If new development work is delayed, the whole project is at risk of being delayed. 

If development proceeds without testing, new bugs get introduced, as there is no way to detect them. That makes the problem of broken trunk self-perpetuating. 

When new development work proceeds based on an old working version of trunk, each developer works largely in separation from other developers. When trunk if finally fixed, it is often revealed that changes done by developers cannot be easily merged together, as they modify the same files in a different way. To merge the changes together, one of the changes often needs to be re-done. However, that often takes a significant amount of time. By the time that work finished, it is often revealed that the trunk has changed further and the change is again incompatible with trunk. That further delays the inclusion of the change. That problem is self-perpetuating as well.

Continuous Integration is an answer to those problems. Continuous Integration, at its essence, is about keeping the trunk working and ready for production at all times. This is achieved through gating tests. Every time a change is proposed, a mock-up environment is stood-up and the code is tested in that environment. The tests include checking if the code would deploy to production. They also include a full functional test. No human person should have the right to bypass the gates. 

Continuous Integration is also a mindset change in that developers should work on small changes at a time and submit to trunk often. By doing so, they avoid integration problems between work done independently by different developers.

Because of the sheer volume of tests and because they are on the critical path to trunk, they need to be reasonably quick, possibly no longer than 40 minutes.

Continuous Integration tests should be expanded in the course of the project. When it is found that a certain kind of bug has slipped through the gates, it should be considered if a test can be written to prevent that situation from happening again.

Setting up Continuous Integration has its costs, however. It requires both time and expertise to do so. Furthermore, significant computing resources are required to handle the large volume of tests.

The benefit of Continuous Integration is predictability. Release can be cut at any time and deployed to production. Releases may be planned to occur on a given day weeks or months in advance and happen exactly on time. A high priority change may be added to the regular release cycle without being delayed by a broken trunk.

What exactly is Delivery Automation

Delivery Automation is the principle of automating all repetitive and operational steps from source control to production. The repetitive steps can generally be divided into the following areas of interest: building a deployable kit, deploying it, testing it and creating records. The automation may be triggered by a push button or start automatically at a given time. 

Delivery Automation does not preclude the existence of a human decision-making step between source control and production. Its goal is merely to automate the repetitive and operational work in the delivery process.

Delivery Automation has a number of advantages. It makes the process strictly defined and timed. It reduces the risk of human error. It makes consistent records, which allow to reliably retrieve information about releases and deployments, such as which features are available in which release. 

Delivery Automation also makes it possible to quickly implement changes to the delivery process. That’s because the inner workings of the automation can be changed without changing the interface used by the operators. As a result, operators do not have to be retrained when the delivery process changes internally, as it would be the case if the delivery process was manual. 

Furthermore, delivery automation itself becomes an automated test that can be used in Continuous Integration.

A disadvantage of Delivery Automation is that it requires resources and specialist knowledge to build it. However, it is usually far less risky and expensive to require specialist knowledge to build automation to perform an activity than it is to require specialist knowledge to execute that activity. That’s because it moves the requirement of the specialist knowledge away from the critical path to success. When delivery is manual, specialist knowledge is required to execute it each time.  Delivery Automation increases consistency and reliability of software delivery. It makes consistent records. It makes change easier. It makes software production more predictable.

Predicability Enables Lean Manufacturing

Once software production is predictable, waste-reduction methodologies such as Lean Manufacturing can be applied to the process with positive results. 

Lean Manufacturing is the concept that only work that adds value to the end customer should be done and every other work is a form of waste. One of the pillars of Lean Manufacturing is Just-in-Time production. It is a scheduling principle whereby a pull request received from the end customer triggers a series of activities towards the beginning of the production chain. 

The danger of introducing Just-in-Time scheduling to software production is that it delays the moment certain work starts and schedules that work on the critical path. However, if the critical path contains unpredictable processes, the time it takes to deliver full product becomes unpredictable. In other words, if work that is scheduled on a critical path takes significantly longer than originally estimated, which is often the case in software development, it delays the entire project. 

When the software development process is known to be unpredictable, it is a reasonable choice to stay away from Just-in-Time production and instead rely on demand prediction to start the work early. That often requires teams to work on many features in parallel, in the hope that at least some of them will be useful. However, the work on features that do not turn out to be useful is a form of waste.

Lean Manufacturing is not an answer to all scheduling activities in a software development team. Some unpredictability is inherent in software development and is unavoidable. That includes all creative and Research & Development activities. Those activities are unpredictable not only in their time frame, but also in their results. For that reason, pull requests should not be the only criteria for scheduling that kind of work. It requires a degree of human judgement to schedule them. 

Just-in-Time production is also not suitable for scheduling work on fixing bugs that have potential disruptive negative consequences. Human judgement is required to hedge the risk and schedule work on them before the disruptions take place.

However, Lean Manufacturing and Just-in-Time production, when applied to predictable, strictly defined and timed activities, can greatly reduce waste of making products that nobody wants.


The predictability of software production can be greatly improved by introducing Continuous Integration and Delivery Automation. The two principles enable more accurate planning and make it possible to adhere to precise release date promises. They allow software teams to react to customer feedback in a more predictable time frame. It enables the teams to rely less on predictions and more on pull requests from end customers. The end result is a more useful product created with fewer resources and delivered on time.

Leave a Reply