Data Quality Improvement Programs

𝗗𝗮𝘁𝗮 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗶𝘀𝗻'𝘁 𝗮 𝘀𝗶𝗻𝗴𝗹𝗲 𝗰𝗵𝗲𝗰𝗸 －it's a continuous contract enforced across the various data layers to avoid breakage. Think about it. Planes don’t just fall out of the sky when they land. Crashes happen when people miss the little signals that get brushed off or ignored. Same thing with data. Bad data doesn’t shout; it just drifts quietly—until your decisions hit the ground. When you bake quality checks into every layer and, actually use observability tools, You end up with data pipelines that hold up. Even when things get messy. That’s how you get data people can trust. Why does this matters? Bad data costs money → Failed ML models, wrong decisions. Good monitoring catches 90% of issues automatically. → Raw Materials (Ingestion)  • Inspect at the dock before accepting delivery.  • Check schemas match expectations. Validate formats are correct.  • Monitor stream lag and file completeness. Catch bad data early.  • Cost of fixing? Minimal here, expensive later.  • Spot problems as close to the source as you can. → Storage (Raw Layer)  • Verify inventory matches what you ordered.  • Confirm row counts and volumes look normal.  • Detect anomalies: sudden spikes signal upstream issues.  • Track metadata: schema changes, data freshness, partition balance.  • Raw data is your backup plan when things go sideways. → Processing (Transformation)  • Quality control during assembly is critical.  • Validate business rules during transformations. Test derived calculations.  • Check for data loss in joins. Monitor deduplication effectiveness.  • Statistical profiling reveals outliers and distribution shifts.  • Most data disasters start right here. → Packaging (Cleansed Data)  • Final inspection before shipping to warehouse.  • Ensure master data consistency across all sources.  • Validate privacy rules: PII masked, anonymization works.  • Verify referential integrity and temporal logic.  • Clean doesn’t always mean correct. Keep checking. → Distribution (Published Data)  • Quality assurance for customer-facing products.  • Check SLAs: freshness, availability, schema contracts met.  • Monitor aggregation accuracy in data marts.  • ML models: detect feature drift, prediction degradation.  • Dashboards: validate calculations match source data.  • Once data is published, you’re on the hook. → Cross-Cutting Layers (Force Multipliers)  • Metadata: rules, lineage, ownership, quality scores  • Monitoring: freshness, volume, anomalies, downtime  • Orchestration: dependencies, retries, SLAs  • Logs: failures, patterns, early warning signs Honestly, logs are gold. Don’t sleep on them. What's your job? Design checkpoints, not firefight data incidents. Quality is built in, not inspected in. Pipelines just 𝗺𝗼𝘃𝗲 data. Quality 𝗽𝗿𝗼𝘁𝗲𝗰𝘁𝘀 your decisions. Image Credits: Piotr Czarnas 𝘌𝘷𝘦𝘳𝘺 𝘭𝘢𝘺𝘦𝘳 𝘯𝘦𝘦𝘥𝘴 𝘪𝘯𝘴𝘱𝘦𝘤𝘵𝘪𝘰𝘯.  𝘚𝘬𝘪𝘱 𝘰𝘯𝘦, 𝘳𝘪𝘴𝘬 𝘦𝘷𝘦𝘳𝘺𝘵𝘩𝘪𝘯𝘨 𝘥𝘰𝘸𝘯𝘴𝘵𝘳𝘦𝘢𝘮.

𝗕𝗹𝗮𝗰𝗸𝗥𝗼𝗰𝗸’𝘀 𝗔𝗜 𝗧𝗮𝗰𝗸𝗹𝗲𝘀 𝗗𝗶𝗿𝘁𝘆 𝗗𝗮𝘁𝗮 BlackRock researchers published a framework for monitoring data quality continuously across the modeling pipeline, not only at ingestion. The target is a familiar problem in finance: dirty data. In regulated systems, even small inconsistencies such as a misplaced decimal or a delayed feed can break automated processes. Yet data cleaning is still often treated as a one-time preprocessing step. “In regulated domains, the integrity of data pipelines is critical,” the researchers write, noting that cleaning is frequently an “afterthought” rather than a core system layer. Their architecture adds a governed quality layer at three points: 𝗜𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 𝗚𝗮𝘁𝗲 Standardizes vendor feeds, enforces schema rules, checks duplicates and missing identifiers like CUSIPs. 𝗠𝗼𝗱𝗲𝗹 𝗖𝗵𝗲𝗰𝗸 Monitors model behavior itself. If an asset pricing model shows implausible yield gaps, the system flags a local anomaly even when raw data appears valid. 𝗘𝘅𝗶𝘁 𝗖𝗵𝗲𝗰𝗸 Validates outputs before they reach downstream systems or decision makers. The key result is alert quality. In benchmark tests of the AI-based data completion module, using AI to fill in missing values rather than dropping incomplete records reduced the false positive rate from ~48% to ~10%, while reaching 90% recall and 90% precision. A +130% improvement in their benchmark. The framework runs in production across streaming and tabular data for multiple asset classes, according to the paper. More details below. Authors: Devender Singh Saini, Bhavika Jain, NITISH UJJWAL, Philip Sommer, Dan Romuald Mbanga, Dhagash Mehta, Ph.D.

80 % of your data problems come from 20% of your company culture. Data quality issues come from organisational behaviour, but leaders keep focusing on the infrastructure. Tooling can detect errors, but it cannot correct for misaligned incentives or the absence of responsible data stewardship. You don’t need to fix everything at once. But you do need to fix the right things first. Start with this small set of cultural shifts to address your data problems: Assign clear data ownership for key assets. Embed validation rules at every step of the data flow. Build literacy across roles so teams can question, interpret, and escalate issues. Detect anomalies in real time and route resolution to the responsible owner. Data governance programs fail because they address symptoms, not causes. Fix the cultural 20%. The rest gets easier.

Managing data quality is critical in the pharma industry because poor data quality leads to inaccurate insights, missed revenue opportunities, and compliance risks. The industry is estimated to lose between $15 million to $25 million annually per company due to poor data quality, according to various studies. To mitigate these challenges, the industry can adopt AI-driven data cleansing, enforce master data management (MDM) practices, and implement real-time monitoring systems to proactively detect and address data issues. There are several options that I have listed below: Automated Data Reconciliation: Set up an automated and AI enabled reconciliation process that compares expected vs. actual data received from syndicated data providers. By cross-referencing historical data or other data sources (such as direct sales reports or CRM systems), discrepancies, like missing accounts, can be quickly identified. Data Quality Dashboards: Create real-time dashboards that display prescription data from key accounts, highlighting any gaps or missing data as soon as it occurs. These dashboards can be designed with alerts that notify the relevant teams when an expected data point is missing. Proactive Exception Reporting: Implement exception reports that flag missing or incomplete data. By establishing business rules for prescription data based on historical trends and account importance, any deviation from the norm (like missing data from key accounts) can trigger alerts for further investigation. Data Quality Checks at the Source: Develop specific data quality checks within the data ingestion pipeline that assess the completeness of account-level prescription data from syndicated data providers. If key account data is missing, this would trigger a notification to your data management team for immediate follow-up with the data providers. Redundant Data Sources: To cross-check, leverage additional data providers or internal data sources (such as sales team reports or pharmacy-level data). By comparing datasets, missing data from syndicated data providers can be quickly identified and verified. Data Stewardship and Monitoring: Assign data stewards or a dedicated team to monitor data feeds from syndicated data providers. These stewards can track patterns in missing data and work closely with data providers to resolve any systemic issues. Regular Audits and SLA Agreements: Establish a service level agreement (SLA) with data providers that includes specific penalties or remedies for missing or delayed data from key accounts. Regularly auditing the data against these SLAs ensures timely identification and correction of missing prescription data. By addressing data quality challenges with advanced technologies and robust management practices, the industry can reduce financial losses, improve operational efficiency, and ultimately enhance patient outcomes.

How do you turn Data Quality from reactive to proactive? The answer lies in DQ Scorecard Analytics. Typical DQ Scorecards capture DQ scores for various rules and provide drill-downs into data errors. But the real value of a DQ scorecard is not in showing the current score; it’s in the ability to analyze the history of DQ failures and take actions based on that analysis. Here’s just one example: in many industries, data comes from hundreds or even thousands of sources: - Retail receives product, pricing, and inventory data from hundreds of vendors - Healthcare receives lab results from hundreds of labs - Real estate platforms collect listings from 500+ MLSs - Financial firms ingest pricing and reference data from many external providers When all these data feeds, it becomes critical to know which vendors ( products / MLSs / labs/etc) generated the most DQ rule failures over the last month? Over the last 3 months? Over the last year? By analyzing the history of failures, we can identify chronic offenders (vendors, MLSs, labs, data providers). And istead of trying to improve DQ randomly, we can re-examine contracts with just these vendors and include clear data expectations (hello, Data Contracts!). Knowing who the chronic DQ offenders are allows us to purposefully improve ingestion processes and reduce manual cleanup. Bottom line: a well-designed scorecard should not only show rule results and DQ scores, but also enable us to analyze: - Trends over time (last week, last month, last quarter) - Which rules and rule categories consistently fail - Which upstream processes need fixes (not just which records failed in our data warehouse) By analyzing the history of DQ failures, we can turn Data Quality from a reactive fire-fighting exercise into a proactive continuous improvement program where the least amount of effort produces the greatest results.

As real-world evidence (#RWE) continues to shape regulatory and reimbursement decisions, ensuring the quality and transparency of real-world data (#RWD) is more critical than ever. A growing ecosystem of tools and frameworks is helping elevate standards across study design, data selection, and reporting, as highlighted below. 🔹 SPIFD2 Framework – A structured process guiding researchers to identify fit-for-purpose data sources, ensuring relevance and validity in study design https://lnkd.in/egXXm5c6 🔹 REQueST Tool – Developed by EUnetHTA to support registry owners in maximizing data quality for health technology assessments and regulatory use https://lnkd.in/eAgzihkJ 🔹 DARWIN EU – A pan-European federated data network enabling secure exchange of RWD for healthcare delivery, policy-making, and research (EMA) https://lnkd.in/eT4XP_bJ 🔹 FDA QCARD Initiative – Offers oncology-specific guidance on data quality and study design, improving the rigor of RWE proposals in cancer research (FDA) https://lnkd.in/erssUTAh 🔹 STaRT-RWE & HARPER Templates – Provide standardized frameworks for planning and reporting RWE studies, enhancing methodological transparency and reproducibility (ISPE/ISPOR) https://osf.io/6qxpf/ 🔹 REPEAT Initiative – A non-profit program evaluating reproducibility of published RWE studies and promoting transparency in longitudinal healthcare database research https://lnkd.in/e4Q5JjCE 🔹 EU Data Quality Framework for Medicines Regulation – Provides guidance and recommendations for assessing the quality of datasets used in regulatory decision-making, with a focus on real-world data and adverse drug reactions (EMA) https://lnkd.in/eKNFtPHU Together, these efforts are helping build trust in RWE by aligning its transparency and quality standards with those of randomized controlled trials. The goal is to ensure that decisions based on RWE are robust, reproducible, and ultimately beneficial to patients. #RealWorldEvidence #RWD #DataQuality #Transparency #FDA #EMA #Biostatistics #ClinicalResearch #RegulatoryScience #EvidenceBasedMedicine #HealthTechAssessment #RWEFrameworks

Data quality is not a technology problem. I have watched organizations spend millions on data quality tools and come out the other side with the same broken data, just better documented. Here’s why: tools measure quality. They don’t create it. Quality is created upstream. In how systems are designed. In whether business owners understand and enforce data standards. In whether there’s a named human being accountable for the accuracy of each critical data element, not a team, not a committee, a person. The moment you assign accountability to a committee, you’ve assigned it to nobody. The organizations with genuinely high data quality share one characteristic: they treat data standards the same way they treat financial controls. Ownership is explicit. Exceptions are tracked. Remediation is required. Nobody gets to say “that’s an IT problem.” Data quality is a business discipline. Technology is just the measurement layer. Until your business leaders own their data the way they own their P&L, your data quality program is theater. #DataQuality #DataGovernance #CDO #DataManagement #DataStrategy

I’m obsessed with one truth: 𝗱𝗮𝘁𝗮 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 is AI’s make-or-break. And it's not that simple to get right ⬇️ ⬇️ ⬇️ Gartner estimates an average organization pays $12.9M in annual losses due to low data quality. AI and Data Engineers know the stakes. Bad data wastes time, breaks trust, and kills potential. Thinking through and implementing a Data Quality Framework helps turn chaos into precision. Here’s why it’s non-negotiable and how to design one. 𝗗𝗮𝘁𝗮 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗗𝗿𝗶𝘃𝗲𝘀 𝗔𝗜 AI’s potential hinges on data integrity. Substandard data leads to flawed predictions, biased models, and eroded trust. ⚡️ Inaccurate data undermines AI, like a healthcare model misdiagnosing due to incomplete records.   ⚡️ Engineers lose their time with short-term fixes instead of driving innovation.   ⚡️ Missing or duplicated data fuels bias, damaging credibility and outcomes. 𝗧𝗵𝗲 𝗣𝗼𝘄𝗲𝗿 𝗼𝗳 𝗮 𝗗𝗮𝘁𝗮 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 A data quality framework ensures your data is AI-ready by defining standards, enforcing rigor, and sustaining reliability. Without it, you’re risking your money and time. Core dimensions:   💡 𝗖𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝘆: Uniform data across systems, like standardized formats.   💡 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆: Data reflecting reality, like verified addresses.   💡 𝗩𝗮𝗹𝗶𝗱𝗶𝘁𝘆: Data adhering to rules, like positive quantities.   💡 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗻𝗲𝘀𝘀: No missing fields, like full transaction records.   💡 𝗧𝗶𝗺𝗲𝗹𝗶𝗻𝗲𝘀𝘀: Current data for real-time applications.   💡 𝗨𝗻𝗶𝗾𝘂𝗲𝗻𝗲𝘀𝘀: No duplicates to distort insights. It's not just a theoretical concept in a vacuum. It's a practical solution you can implement. For example, Databricks Data Quality Framework (link in the comments, kudos to the team Denny Lee Jules Damji Rahul Potharaju), for example, leverages these dimensions, using Delta Live Tables for automated checks (e.g., detecting null values) and Lakehouse Monitoring for real-time metrics. But any robust framework (custom or tool-based) must align with these principles to succeed. 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲, 𝗕𝘂𝘁 𝗛𝘂𝗺𝗮𝗻 𝗢𝘃𝗲𝗿𝘀𝗶𝗴𝗵𝘁 𝗜𝘀 𝗘𝘃𝗲𝗿𝘆𝘁𝗵𝗶𝗻𝗴 Automation accelerates, but human oversight ensures excellence. Tools can flag issues like missing fields or duplicates in real time, saving countless hours. Yet, automation alone isn’t enough—human input and oversight are critical. A framework without human accountability risks blind spots. 𝗛𝗼𝘄 𝘁𝗼 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁 𝗮 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 ✅ Set standards, identify key dimensions for your AI (e.g., completeness for analytics). Define rules, like “no null customer IDs.”   ✅ Automate enforcement, embed checks in pipelines using tools.   ✅ Monitor continuously, track metrics like error rates with dashboards. Databricks’ Lakehouse Monitoring is one option, adapt to your stack.   ✅ Lead with oversight, assign a team to review metrics, refine rules, and ensure human judgment. #DataQuality #AI #DataEngineering #AIEngineering

💡 Mastering Data Quality in Modern Data Pipelines 🎯 “Your analytics are only as good as the data feeding them.” In today’s fast-paced data ecosystems, bad data isn’t just an inconvenience, it’s an invisible cost that compounds over time. Whether you’re building on Spark, Airflow, or dbt, data quality should be treated as a first-class citizen in your architecture. Here’s what separates resilient data platforms from reactive ones 👇 🔹 1. Shift-Left Data Validation  Don’t wait until your dashboards break. Validate early at ingestion.  Use tools like Great Expectations, Soda, or Deequ to catch schema drifts and anomalies before loading data downstream. 🔹 2. Observability as a Core Component  Treat data like infrastructure.  📊 Implement end-to-end monitoring for freshness, volume, and schema consistency.  Platforms like Monte Carlo, Databand, or OpenMetadata can help you see your data flows. 🔹 3. Version Control for Data Models  Use Git + CI/CD for your transformation logic.  ⚙️ dbt tests + automated checks = fewer surprises in production. 🔹 4. Feedback Loops from Consumers  Your downstream users (analysts, ML teams, BI tools) are your best sensors.  💬 Create Slack or Jira-based feedback loops for data issues. 🔹 5. Golden Data Contracts  Define schemas, SLAs, and ownership before data starts flowing.  📄 Data contracts reduce chaos between producers and consumers — aligning expectations around latency, structure, and quality. 💬 Final Thought:  Data quality isn’t a one-time project, it’s a culture. Build trust by designing your pipelines to detect, prevent, and communicate quality issues automatically. 👇 How are you ensuring data reliability in your pipelines today? #DataEngineering #DataQuality #DataObservability #ETL #DataOps #GreatExpectations #dbt #DataContracts #BigData #Airflow #DataTrust #AnalyticsEngineering #DataPipeline

Leadership want a silver bullet to improving data quality, but that doesn’t exist   Below, I list out how you can start to think about data quality with a holistic and logical approach   First, 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝 the lay of the land, including the technology landscape, pain points, and what data is there to fix. This includes: 💻 Data Technology & Tooling Audit/Strategy – Outlining what different tools do within the data journey and align that with data quality needs 🛠️ Root Cause Analysis – A systematic process helps teams understand why data issues occur and enable targeted interventions that address more than just the symptom 🏆 Critical & Master Data Assets – Help focus efforts and resources on the most impactful data   Next, 𝐒𝐭𝐚𝐧𝐝𝐚𝐫𝐝𝐢𝐬𝐞 what data quality means within the organization and have a strategy to tackle these fixes in a proactive (not reactive) way. This includes: 🎯 Data Governance Strategy – Understand how the organisation works with and governs data (including who owns it) 📝 Setting Data Quality Standards – Establish those clear and measurable criteria for data quality to serve as a benchmark for all people across the organisation 📑 Data Contracts – Set clear expectations and responsibilities between the downstream and upstream groups of data users   Finally, 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭 tools, technologies and approaches to combat data quality issues (but don’t skip to this step without doing the others) ⚙️ Data Catalogue & Lineage Tooling – Allow users to search datasets, understand its content, provide access, define ownership, and construct the flow of data assets from source to consumption 🛑 Data Quality Gates – Define checkpoints at various stages of the data platform (usually contained within pipelines) that validate data against predefined criteria before it proceeds further 👀 Data Observability Tooling – Monitor data health metrics to detect, diagnose, and resolve data quality issues in real-time to reduce data downtime and improve visibility into issues   There are other things you can do as well, but the point is to think about these things holistically and in order of implementation.   Check out my article last week (link in comments) about defining data quality issues and stay tuned this week for a lot more on each of these approaches   #dataecosystem #dataquality #newsletter #datastrategy #dylandecodes