Lessons Learned Part 9: Validation

"The great enemy of truth is very often not the lie: deliberate, contrived, and dishonest -- but the myth: persistent, persuasive, and unrealistic." -- John F. Kennedy

Validation is a critical piece that is often left out of discussions of both organizational learning and of actual lessons learned systems. It's important to know that a lesson is in some sense "correct" and not just someone's opinion. You might imagine a frustrated employee who was part of a failed project arguing, absent any buffer or discussion, that the reason for the failure was incompetent management and so the lesson is to always make sure your management chain is competent. This is probably not true but even if it is, it's not very useful.

While it is not too hard to validate lessons that arise from events that happen frequently, it becomes more difficult the less often an event or situation occurs. But before we can even start thinking about how to validate lessons, we have to understand what we mean when we say a lesson is "valid." Sometimes we just don't have enough data to validate a lesson statistically because the event hasn't happened frequently enough, and that may actually be a good thing. If someone tells me "Never light a match near the gas tank of your car because the car will blow up" and therefore I never light a match near the gas tank, there is no way for me to personally validate that lesson. But the consequences are such that I do not care to discover whether that lesson is valid or not.

But even if lessons cannot be validated in the sense of ascertaining truth, they can often be validated in the sense of gaining organizational consensus and acceptance. Here are a few scenarios and guidelines for how you might validate organizational lessons:

• When the same type of event occurs more than once to the same person or team, a best practice or solution to a problem can be validated by looking for consistent outcomes. ("Every time we do X, we get Y."). This is essentially the basis for probability and statistical methods.
• Sometimes an event might occur only once to a single individual or team, but multiple times throughout the organization. In these cases it is helpful to have a knowledge intermediary or community of practice that can step back, look across the organization, identify the commonality of those events, compare experiences, and develop a consensus as to what the effective recommendations should be.
• In cases where it is critical to avoid repeating an event that happened only once, it is useful to have a group of people with expertise in that area (either internal or external to the organization) scrutinize the event and validate the lesson learned in the context of their expertise.
• In worst case situations, a rare event occurs in an area where no one has the appropriate expertise. In these cases the knowledge intermediary, acting as lessons learned reviewer, might make a judgment based on the significance of the lesson. Lessons such as the one about the exploding gas tank above might move quickly into the lessons learned database, while others that are less significant might be held until more data can be obtained.

In any case, there should always be some process to check lessons before they are posted to a lessons learned repository. Otherwise the repository will become increasingly useless as redundant lessons and lessons of varying quality and effectiveness are accumulated.

Continue to Part 10: Scoping



Previous posts in this series:

• Part 1: You Can Lead a Firm to Knowledge But You Can't Make It Think
• Part 2: What is a Lesson?
• Part 3: Initiation and Recognition
• Part 4: Capture
• Part 5: Capture Using After Action Reviews
• Part 6: Ten Tips for Facilitating After Action Reviews
• Part 7: Additional Ways to Capture Lessons
• Part 8: The Role of the Knowledge Intermediary in the Lessons Learned Life Cycle

References:

• David Aha and Rosina Weber: "'Preface,' AAAI Workshop: Intelligent Lessons Learned Systems"
• R. Dietrich: "'Why Don't They Get It?,' Society for Effective Lessons Learned Sharing (SELLS) Spring Workshop Lessons Learned Self-Assessment and Analysis -- Why Don't They Get It?"
• George Roth and Art Kleiner: "Developing Organizational Memory through Learning Histories"
• Paul Shrivastava: "A Typology of Organizational Learning Systems"
• U.S. Department of Energy: "DOE Standard: The DOE Corporate Lessons Learned Program"
• Rosina Weber et al: "Categorizing Intelligent Lessons Learned Systems"