Data and Functions – What Else Is There Really?

I have been involved with software development in one way or another for over 30 years – firstly as a developer (Assembler, Cobol, Fortran, RPG etc. and finally Visual Objects, C, C++ and C#) and then later as a Project Manager, Systems Manager and finally Programme Manager.

Although as a Programme Manager today I am not allowed anywhere near the coding environment (and boy, it has changed over the years so that is probably just as well!) I still have my love and fascination for coding and algorithms and I am currently learning Swift4 and starting to code on the iPhone (which is, by the way, much more powerful than the first mainframe on which I coded with its miniscule 64k memory!)

So What?

“Very interesting, but so what?” I hear you ask. Well, consider this view:

Code is developed solely for the purpose of directing the computer hardware to execute the commands and functions necessary to achieve the desired outcome – whatever it may be (producing a report, controlling a database, running a SatNav, flying a drone etc.).

The languages, paradigms, methods (development, testing, change control etc.) that have evolved over the years merely improve or enhance our ability to produce faster, more efficient code that has fewer bugs, is produced more quickly, more cheaply and is easier to maintain.

Not that this is a bad thing, of course. Gone are the days where a company can afford to allow development for the sake of it. We live in a fast-paced environment where everything is driven by cost and quick turnaround to meet ever changing business needs. Without these newer, faster methods and processes the IT Departments would never keep up (and some would argue that they still don’t, but that is a subject for a different discussion!)

Consider This

My observation over 30 years is that there are only two things that are pertinent; Data and the Functions that act on that data. Everything else is in support of this.

Data is your company asset. It never goes away – you might add to it or delete from it (especially with GDPR looming), but by and large it forms the bedrock of your business by defining your customers, your inventory, billing, payroll, reporting etc.

Functions are the activities that your company performs to manage, make money and stay in business. These functions will comprise both manual and IT processes and will create, delete or refer to data in various ways – from a payroll system that is an entirely automated package solution to a printed report for the C-Suite produced on a spreadsheet detailing statistics relating to sales, revenue etc. However, the world changes and evolves and Businesses need to keep pace.

This leads to two conclusions:

1. Companies need look after their data and treat it like any other asset. 

This is not about security or backups – this is normally adequately covered. This is about the data itself. Most companies have a myriad of databases and files but I have not personally seen a company that has one fully-integrated “database of everything”. Granted, in most cases this is not possible or practical, but the consequence is that there will be data duplication, integrity problems and update anomalies occurring at regular intervals. Establishment of a “Golden Source” for certain data is about the best I have seen.

Also, how many companies have an up-to-date Enterprise Data Model and Enterprise Data Dictionary? Not many, I suspect. This is starting to emerge as companies prepare for GDPR. (I wonder how “the right to be forgotten” following GDPR implementation will surface some of these problems.)

2. It is inevitable that functions will change or be replaced over time – often quite radically. This will vary from complete changes to business processes (requiring new functions, complete re-write of software systems or purchase and implementation of new packages) through to Change Management implementing small changes, regulatory updates, expansion of services etc.

However, in all functional change, the existing data remains structurally the same – it may be updated or added to or completely new lines of data may be created, but it is still adding to your company asset. Better get your Enterprise Data Model off the shelf.

One programme that I ran involved developing a brand new “one and done” system that replaced three mainframe systems that had been running for three decades. Some of the data (which included life policies) was as old as the systems themselves as data was never deleted. There was no Enterprise Data Model and some of the “rules” that had been applied to field names and field validation were not documented – there were at least three different meanings of the field “Order Number”! It is little wonder that the data migration took a year longer than originally planned!

What To Do, What To Do

It is not my intention to trivialise the extremely complicated and integrally interdependent systems running in today’s large corporations. In addition, to “drop everything” and spend a lot of time and money sorting out data models and functional charts would be prohibitive and is not going to happen. Indeed, the disadvantages would probably outweigh the advantages in the short term – and who knows what advances in computer technology will emerge in the next decade that make this problem (and this article!) obsolete.

However, here are my suggestions for “chipping away” at the problem should you get the chance in your next major development:

Data Normalisation

Data normalisation is the process of organizing the columns (attributes) and tables (relations) of a relational database to reduce data redundancy and improve data integrity.  It is also the process of simplifying the design of a database so that it achieves the optimal structure.

The concept of normalisation was first proposed by Edgar Codd in 1970 with the familiar Third Normal Form (3NF) becoming widely used in 1971. There are a number of good articles and books available on the internet and through the BCS. Here is one: https://www.studytonight.com/dbms/database-normalization . However, the mnemonic I use is that for 3NF, all attributes in a table must be dependent on “the key, the whole key and nothing but the key”.

So, ask your Chief Data Architect to explain to you what is the status of all your databases, whether they are all in 3NF and does he/she have an Enterprise Data Model to show you!

Functional Decomposition

Functional decomposition is a set of steps which breaks down the overall function of a system, or process into its smaller constituents.  In effect, these functions actually become the Functional Requirements.

Functional decomposition is an integral part of today’s Systems Development methodologies and as such I am not trying to interfere, contradict or suggest change to the existing development lifecycle. What I am advocating is an extra use of this valuable process to enhance further the documentation and understanding of the final system.

Here is an example found on the internet that is part of the High-Level Requirement Phase and illustrates the above:

Function decomposition is usually accomplished through analysis, workshops and team discussions of new or existing functional information and the result is a chart that describes the proposed system (requirements) in great detail. As a rule, you should continue to decompose each level as long as they still describe the function and not the solution. There are many tools on the market that assist with this process and then follow through into the Design Stage and link to the proposed Data Model, but this is not the purpose of this article.

My point is that by taking this diagram - which is often discarded once used as part of the development lifecycle - it can begin a new life as a “living document”, kept up to date and used to:

1. provide a clear view of exactly what functions are performed and in what context of the overall system.
2. Can be used effectively in presentations to walk Sponsors or new team members through what the system does (or will do) and aids thinking around possible enhancements for later development.
3. Can be colour coded to show areas not yet developed or areas that form a discrete function that could be outsourced or replaced by a package.
4. Will assist in determining areas affected by potential or imminent change.

In Conclusion

As I said in the beginning (and thank you for sticking with me for so long!), there are only two things that are pertinent; Data and the Functions that act on that data.

So, in order to give you a bit of comfort around the state of your systems:

1. get a copy of your Enterprise Data Model and Enterprise Data Dictionary from your Chief Architect (if he/she has one). If not, ensure each project/programme produces a Data Model and Data Dictionary going forward as part of the required sign-off deliverables. Make this the starting point for the Enterprise versions.
2. Similarly, get a copy of the Functional Decomposition Structure Chart from each project, or if not available make this a prerequisite deliverable for sign-off.
3. Set up a project (not too long – make it timeboxed) to review and combine these artefacts into your first Enterprise view of your systems. You may be surprised what you learn!

Thank You!