Long term software evolution – Taming the complexity beast

“Ideas are easy, execution is everything” – this is a well-known maxim of the influential Intel manager and technology investor John Doerr, with which he emphasises the special importance that the highest implementation competence has in technology innovations. The reason for this assessment is that an existential risk in the development of technology products is the ever-increasing complexity. Demanding functional extensions, the departure of knowledge holders or fundamental changes in production principles are complexity drivers that are often recognised too late or are not addressed resolutely enough. Complexity management is therefore a central task in the implementation of technology products.

In software development, proposals for managing complexity are not new. Different process models were discussed as early as the 1950s, and in the last 20 years agile software development has exerted great influence in many forms, including beyond the software world. However, despite undeniable methodological progress, the number of failed software projects remains high: projects are abandoned, essential requirements remain unfulfilled or fall far short of expectations, or the software can no longer be developed further to meet new requirements.

To mitigate these risks, the process model for software development at SMA is strongly characterised by the product mindset. This implies that the software’s life cycle over many years is at the centre of the production process. Exacting requirements in terms of quality, maintainability and delivery reliability are combined with the need to implement innovative and ambitious further developments.

This task has a high inherent complexity and requires the appropriate tools. A central component in this respect is the automation of activities that were originally manual.

One example is the process for creating new versions of Viriato, which today is highly automated: the correct branches are created for all relevant components and versioned in the version control system, jobs are automatically set up on a Continuous Integration server, dependency management is updated, and the issue management system is modified to enable the assignment of implemented stories to Viriato versions.

The result of this automation is significant efficiency gains, the elimination of sources of error and improved standardisation.

The production team implements such activities as part of the continuous improvement process, which is an integral part of the Scrum methodology. In this, potential process improvements are identified through a structured procedure, reflected upon and implemented where it makes sense to do so. Viewed over a long period of time, these many incremental improvements lead to impressive results.

Incremental improvements are a fundamental tool for managing complexity because they enable an evolutionary path to be taken in a complex environment that is not available in the same way with a large, riskier replacement. Replacement systems for legacy systems are often developed precisely because incremental adjustments are no longer possible or become too expensive due to unmanageable complexity in the software.

In the articles below, some representatives from sma.software provide an in-depth insight into specific aspects of software production at SMA.

Product development – Björn Glaus

Björn Glaus is Head of Production at sma.software. For Björn, the unwavering focus on product development for timetable and capacity planning is at the heart of sma.software. In his opinion, this consistency leads to a unique expertise at the interface between high-quality, modern software development, deep domain knowledge and relevant algorithmic applications. Björn sees SMA’s core competence in implementing this skill mix in a way that works with the user.

One user group of particular importance to sma.software, says Björn, are the Viriato users within SMA. The daily internal use of the software in our consulting projects enables a short feedback loop and a constant professional exchange between developers and users. SMA’s consultants operate in a highly competitive environment, which requires software that provides optimal support for their projects. Their proximity to product development creates an ideal environment for the creation of sophisticated software.

According to Björn, particularly relevant for product development are improvements that have been achieved in prototyping. There are two recent advances to be highlighted:

Firstly, thanks to investments in the technical infrastructure, a functional prototype can now be implemented more quickly and easily and used in suitable consulting projects as a pilot study. Prototype development is decoupled from product development but is based on precisely defined product and quality levels. The use of the prototype in a consultancy project is lightweight and low risk for all parties involved because the prototype is strictly used for only in a specific project and it is possible to go back to the product version at any time.

Secondly, the introduction of new roles and processes at the interface between R&D, business analysis and development has meant that results from prototyping can be transferred methodically and efficiently into the product. Thanks to prototype development, uncertainties and risks are clarified early on. This greatly simplifies the transfer into the product, especially the compliance with the high requirements regarding stability, architecture guidelines, automatic test coverage and other quality criteria.

Software Craftmanship – Benjamin Ernst

As an architect at sma.software, Beni makes a point of continuing to spend a lot of time programming or undertaking code reviews alongside technical coaching and conceptual tasks. He sees Viriato’s modular architecture as a central and effective tool in complexity management: the software is developed in the form of loosely coupled modules, each using a common, powerful domain model as well as a custom-developed application framework that provides shared services and base classes. This modularisation enables a high degree of parallelisation in the implementation, because different modules can be worked on simultaneously without developers getting in each other’s way.

According to Beni, a sustained rapid development velocity is only possible if the quality of the code and the way of working is also kept constantly high. For this reason, the architectural integrity and other quality aspects of the software are continuously monitored in an automated way. In addition to this monitoring, constant investment in the architecture, the existing source code and the technical tooling is also necessary. A large code base tends to create structural deficiencies through ongoing developments and to decrease the degree of organisation in the code. Here, Beni emphasises the particular importance of code refactoring and functional re-designs. These are technical practices in which the structure of the code or the user experience is improved without changing the functionality. Such renewal work keeps the software up to date, but it is also essential for retaining know-how in the development team. Software that is difficult to develop further because the code is not well understood is an often-underestimated risk.

For Beni, development in a product-oriented company is characterised by investing more on in-house developments than is typical in the fast-moving project business. There, to keep project times short, third-party solutions tend to be used. In contrast, because of the long-term perspective in the product business, it is worthwhile to develop the frameworks yourself. These are customised and further development is under our full control. As examples, Beni mentions the build system (for creating the software from the source code), which is specifically geared to the modular architecture, as well as the persistence layer (the database abstraction layer), which is optimised regarding the high-performance requirements of timetable planning.

Customer-Projects – Markus Ullius

As Head of Services at sma.software and leading a team of business analysts, project managers and testers, Markus and his team can draw on many years of experience in the implementation of large software projects. Markus proudly mentions that customers have repeatedly emphasised the high quality and adherence to delivery deadlines of sma.software. He sees this as the result of a mature development process that covers the entire functionality life cycle: the precise formulation of the customer’s needs as requirements is the first step and is a prerequisite for the development of solution proposals in the form of “stories”. The implementation of a solution then takes place in well-planned development “sprints” with a three-week cycle.

Delivery takes place only after a quality sprint, in which no more functional development takes place, only testing and bug fixing. After delivery, the newly developed software goes into maintenance and support. Markus emphasises the short communication channels and the organisational proximity of support to development. If necessary, a qualified developer can be called in to deal with a support case in a very short time.

Markus points out that the stories are highly standardised in terms of completeness, comprehensibility and complexity. This standardisation is a prerequisite for multiple projects to be implemented at the same time, because in some respects it becomes irrelevant for the development from which project a user story originates. The story is tailored in such a way that it can be implemented in a few days, acceptance criteria are formulated in advance, and close technical support is ensured during implementation. The business analyst, the developer and the tester form a team within a team during the implementation.

According to Markus, the goal is to protect the development as much as possible from disruptions from day-to-day project business, e.g. last-minute change requests or unexpected re-prioritisations. Such discussions are allowed to take place, but they are upstream of the development. A focused, disciplined and low stress working environment is the goal for development.

Conclusion

The domain of sma.software is the railway system. Railway is characterised by a high inherent complexity and long life and investment cycles. For innovations to be successful, they must be implementable, i.e. feasible, which is a challenging task in such an environment. sma.software is able to successfully implement innovations because it has many years of experience in mastering complexity at different levels – code, process, culture, team. The results of this consistent work are a stable, extendable software platform and a highly capable, reliable organisation. These serve as the basis for sustainable and fit-for-purpose solutions.