I have worked in the space of data driven decision making practically all my professional (and personal) life, and seen it change - for better or worse only time can tell. Recently, OpenAI came out with a report that details how people use it's interface to models we have all come to know and love as ChatGPT. It prompted me to reflect on how decision making with data has changed in my perspective - and hopefully, this article will do the same for you.
In the first half of my career,
- data was highly specialized - generated either from complex system simulations like Computational Fluid Dynamics (CFD) software or expensive tests either component or engine level.
- Basically, data - from generation, processing and storage point of view - was expensive and hence, remained sparse.
- However, it was generated within physical constraints, so models for decision making were specialized and explainable as well.
- Decision making followed a review with a committee most of whose members could understand all the aspects of the data and the downstream effect.
- Consequently, change management was relatively easier.
In the next maybe 5 or so years,
- Data became generalized - coming from external sources like supply chains or COVID tracking or operations in field or manufacturing logs.
- Volumes shot up! Raw quality went down. Translucency sets in.
- Consequently, while data processing and storage became cheaper (and not to say, exponentially faster), the understanding of constraints that drove the data got weaker.
- Decision making still remains with a committee, however there are two groups - one of which understands the upstream better and the other is closer to the downstream.
- Change management became slower, and required multi-disciplinary discussions.
- Need for negotiation skills in the room went up!
And now, coming to the current report of last 2-3 years that prompted this thread of discussion for me
- Data has become nearly infinite for the models - already available before you even bring your own dataset.
- Easy to work with - just chat - btw, further proof of how lonely social media has left us all that we needed expensive machines to talk to! [just a cynical joke :-)]
- The usage increase for non-paid-work over time - be it assistance or companionship or validation.
- In summary, it helps us move from data to decision much faster now.
- However, as it moves some of the decision making upstream for non-work aspects (reviewing that email you were sending), it makes it much less critical.
- The system tries to maximize engagement rather than information, so (potentially) your biases are reinforced.
- Even at work, the effort and time expected to be spent on data processing and quality is going down exponentially. (Note - expected to be!)
- Here too, data to 'a' decision path is shorter.
- However, there are now further clusters in the decision making committee slowing down change management of the said decision options.
That is why you see aspects like productivity (where decision making is with individual) surge in discussion and reporting while professional decision making (or collaboration - where a committee decides) do not feature as much.
I may be completely blind sided or insightful. Maybe I should ask ChatGPT? Or you could help me!
