How to Improve Data Management Processes

After building 10+ data warehouses over 10 years, I can teach you how to keep yours clean in 5 minutes. Most companies have messy data warehouses that nobody wants to use. Here's how to fix that: 1. Understand the business first Know how your company makes money • Meet with business stakeholders regularly • Map out business entities and interactions  • Document critical company KPIs and metrics This creates your foundation for everything else. 2. Design proper data models Use dimensional modeling with facts and dimensions • Create dim_noun tables for business entities • Build fct_verb tables for business interactions • Store data at lowest possible granularity Good modeling makes queries simple and fast. 3. Validate input data quality Check five data verticals before processing • Monitor data freshness and consistency • Validate data types and constraints • Track size and metric variance Never process garbage data no matter the pressure. 4. Define single source of truth Create one place for metrics and data • Define all metrics in data mart layer • Ensure stakeholders use SOT data only • Track data lineage and usage patterns This eliminates "the numbers don't match" conversations. 5. Keep stakeholders informed Communication drives warehouse adoption and resources • Document clear need and pain points • Demo benefits with before/after comparisons • Set realistic expectations with buffer time • Evangelize wins with leadership regularly No buy-in means no resources for improvement. 6. Watch for organizational red flags Some problems you can't solve with better code • Leadership doesn't value data initiatives • Constant reorganizations disrupt long-term projects • Misaligned teams with competing objectives • No dedicated data team support Sometimes the solution is finding a better company. 7. Focus on progressive transformation Use bronze/silver/gold layer architecture • Validate data before transformation begins • Transform data step by step • Create clean marts for consumption This approach makes debugging and maintenance easier. 8. Make data accessible Build one big tables for stakeholders • Join facts and dimensions appropriately • Aggregate to required business granularity • Calculate metrics in one consistent place Users prefer simple tables over complex joins. Share this with your network if it helps you build better data warehouses. How do you handle data warehouse maintenance? Share your approach in the comments below. ----- Follow me for more actionable content. #DataEngineering #DataWarehouse #DataQuality #DataModeling  #DataGovernance #Analytics

Your data problem didn't start in your warehouse. It started in that free-text 'Region' in your ERP. Spending $1M modernizing your stack, won't fix your data. Everyone wants accurate data. But when you dig you realize their processes were never built to produce good data. They’re trying to analyze chaos. A few months ago, we were talking to finance company. They’d just spent 14 months modernizing their stack. They hired the data engineers. Millions spent. Hundreds of dashboards. And yet: “Revenue” in Salesforce included refunds. “Customer” in Marketing meant prospects too. Operations had 15 different “regions” spelled 8 different ways. The tech wasn’t broken. The process was. Their CRM, ERP, and sales systems were designed for convenience, not for data. Every time a sales rep skips a CRM field.. You create a leak in your data foundation. Until your warehouse is garbage. If your processes weren’t designed with data in mind nothing will save you Here is how to go about stopping bad data 1. Design Every Process as if Data Were the End Goal If you’re setting up a CRM, ERP, or even a Google Form, build it like a data engineer would. Even if it's yet. Develop a process with data in mind. As down the line, you will need, and rather than waiting 3-5 months to get data. Replace free-text fields with controlled dropdowns. Enforce mandatory fields that align with business-critical metrics. Executives say they want clean data but approve workflows that guarantee mess. In my opinion, data should be clean from the source. Becuse if it's not, managing pipelines, modelling becomes a nightmare. And even that can't save it 2. Treat Metrics Like Products Agreeing on definitions is not easy at all. People change, leave. 2 VPs can't agree on it so they create their own spreadsheet. Every metric you report on should have an owner, version history, use case and single definition across the company. If found in a situation can't agree, ask "What finding this info enables you" If can't answer it, archive it. Or if can't agree on metric. Seperate and define clear use case where each. 3. Asssign Owner & Build Feedback Loops Bad data comes from the frontlines, reps skipping CRM fields, creating custom objects in Salesforce. Assign owners of the metrics. Answer: Who owns the data? Who manages the inputs? Who's keeping operational systems clean? (Data stewards) If no one is accountable or owns it, how do you thing it will get fixed. Tie accuracy to incentives. 4. Enforce Standards, Not Opinions Everyone uses their own definition of “good data” Define how data should look: formats, naming, validation rules. If “Region” is free-text in CRM, you’ve built chaos by design. 5. Data quality isn’t a project or a one-time thing Start where it's most important. Track exceptions, expose results, fix patterns. Embed it in the system, so it's proactive rather than reactive.

In my previous post, I explored the hidden costs of data silos. Today, I want to share practical steps that deliver value without requiring immediate organisational restructuring or technology overhauls. The journey from siloed to integrated data follows a maturity curve, beginning with quick wins and progressing toward more substantial transformation. For immediate progress: 1) Identify your "golden datasets": Focus on the 20% of data driving 80% of decisions. Prioritise customer, product, and financial datasets that cross departmental boundaries. 2) Create a simple business glossary: Document how terms differ across departments. When Finance defines "revenue" differently than Sales, capturing both definitions creates transparency without forcing uniformity. 3) Implement read-only integration patterns: Establish one-way flows where analytics platforms access source data without disrupting existing systems. These connections create cross-silo visibility with minimal risk. 4) Build a culture of trust: Reward cross-departmental collaboration. Create incentives that make data sharing a path to recognition rather than a threat to influence or expertise. 5) Establish cross-functional data forums: Host regular meetings where data users share challenges and use cases, building relationships while identifying practical integration opportunities. As these initiatives gain traction, organisations can advance to more substantial approaches: 6) Match your approach to complexity: Smaller organisations often succeed with centralised data management, while larger enterprises typically require domain-centric strategies. 7) Apply bounded contexts: Map where business domains have distinct needs and terminology, creating clear translation points between areas like Sales, Finance, and Operations. 8) Adopt a data product mindset: Designate product owners for critical datasets who treat data as a product with clear consumers and quality standards rather than simply an asset to be stored. 9) Develop a federated metadata approach: Catalogue not just what exists, but how data relates across domains, making relationships between siloed systems explicit. 10) Maintain disciplined data modelling: Well-structured data within domains makes integration between them far more manageable, regardless of your architectural approach. This stepped approach delivers immediate value while building momentum for more sophisticated strategies. The most successful organisations pair technical solutions with cultural transformation, recognising that effective data integration is ultimately about people collaborating across boundaries. In my next post, I'll explore how governance models evolve with data integration maturity. What approaches have you found most effective in addressing data silos? #DataStrategy #DataCulture #DataGovernance #Innovation #Management

Are you treating the symptoms of bad data OR addressing the root causes? Too often, data teams focus on cleaning the data. But if you don’t dig into why the data is bad, you’ll keep spinning your wheels. Enter: The Fishbone Diagram Also known as the Ishikawa or cause-and-effect diagram, it’s one of the most effective tools for identifying the real reasons behind a data quality issue. 🧠 Use it to: • Map out all possible root causes • Group them into logical categories • Spark team collaboration and critical thinking • Present findings in a way that's clear and visual 🔧 Here's how to build one: 1. State the data quality issue (e.g. incorrect customer addresses) — this goes at the “head” of the fish. 2. Determine the main categories — like Tools, Employees, Processes, Standards, Data Sources. 3. Add root causes — what factors contribute to the issue? Connect each to the relevant category. 4. Add sub-causes — ask “why?” to dig deeper and reveal underlying causes. 💡 Pro tip: Apply the “5 Whys” technique for each cause to get to the core issue. If you want a practical example, I've included a free Fishbone Diagram template focused on poor quality address data on my website. What’s one recurring data issue you've seen that deserves a root cause analysis? __ Follow me here for more hands-on insights for the data professional. – George Firican #dataquality #datamanagement

Data quality is fundamental to achieving reliable, impactful program outcomes, especially within the complex landscape of humanitarian and public health interventions. This document, Data Quality and Quality Improvement Training by USAID, provides an in-depth approach to data quality assessment, introducing critical tools like the Data Quality Assessment (DQA) and Routine Data Quality Assessment (RDQA) frameworks. These tools are designed to help organizations evaluate, maintain, and enhance the accuracy, consistency, and timeliness of their data, empowering them to make informed, data-driven decisions. This guide is essential for M&E professionals and program managers who are responsible for data integrity across service sites and reporting systems. It outlines step-by-step processes for verifying data at multiple levels, from on-site service data checks to system-wide evaluations, ensuring that data collection and reporting are aligned with high standards of quality. Practical tools, including Excel-based dashboards and real-time monitoring checklists, support these assessments, allowing for immediate insights into areas that need improvement. Beyond verification, the document emphasizes the value of building data quality into everyday processes, from staff training to cross-referencing data sources, and includes strategies for continuous quality improvement. This resource is indispensable for anyone committed to enhancing program accountability, data reliability, and ultimately, the effectiveness of humanitarian interventions.

This week, I want to talk about something that might not be the most exciting or sexy topic—it might even seem plain boring to some of you. Very impactful, yet even in many large and complex organizations with tons of data challenges this foundational data process simply doesn’t exist: the Data Issue Management Process. Why is this so critical? Because #data issues, such as data quality problems, pipeline breakdowns, or process inefficiencies, can have real business consequences. They cause manual rework, compliance risks, and failed analytical initiatives. Without a structured way to identify, analyze, and resolve these issues, organizations waste time duplicating efforts, firefighting, and dealing with costly disruptions. The image I’ve attached outlines my take on a standard end-to-end data issue management process, broken down below: 📝 Logging the Issue – Make it simple and accessible for anyone in the organization to log an issue. If the process is too complicated, people will bypass it, leaving problems unresolved. ⚖️ Assessing the Impact – Understand the severity and business implications of the issue. This helps prioritize what truly matters and builds a case for fixing the problem. 👤 Assigning Ownership – Ensure clear accountability. Ownership doesn’t mean fixing the issue alone—it means driving it toward resolution with the right support and resources. 🕵️♂️ Analyzing the Root Cause – Trace the problem back to its origin. Most issues aren’t caused by systems, but by process gaps, manual errors, or missing controls. 🛠️ Resolving the Issue – Fix the data AND the root cause. This could mean improving data quality controls, updating business processes, or implementing technical fixes. 👀 Tracking and Monitoring – Keep an eye on open issues to ensure they don’t get stuck in limbo. Transparency is key to driving resolution. 🏁 Closing the Issue and Documenting the Resolution – Ensure the fix is verified, documented, and lessons are captured to prevent recurrence. Data issue management might not be flashy, but it can be very impactful. Giving business teams a place to flag issues and actually be heard, transforms endless complaints (because yes, they do love to complain about “the data”) into real solutions. And when organizations step back to identify and fix thematic patterns instead of just one-off issues, the impact can go from incremental to game-changing. For the full article ➡️ https://lnkd.in/eWBaWjbX #DataGovernance #DataManagement #DataQuality #BusinessEfficiency

Managing data quality is critical in the pharma industry because poor data quality leads to inaccurate insights, missed revenue opportunities, and compliance risks. The industry is estimated to lose between $15 million to $25 million annually per company due to poor data quality, according to various studies. To mitigate these challenges, the industry can adopt AI-driven data cleansing, enforce master data management (MDM) practices, and implement real-time monitoring systems to proactively detect and address data issues. There are several options that I have listed below: Automated Data Reconciliation: Set up an automated and AI enabled reconciliation process that compares expected vs. actual data received from syndicated data providers. By cross-referencing historical data or other data sources (such as direct sales reports or CRM systems), discrepancies, like missing accounts, can be quickly identified. Data Quality Dashboards: Create real-time dashboards that display prescription data from key accounts, highlighting any gaps or missing data as soon as it occurs. These dashboards can be designed with alerts that notify the relevant teams when an expected data point is missing. Proactive Exception Reporting: Implement exception reports that flag missing or incomplete data. By establishing business rules for prescription data based on historical trends and account importance, any deviation from the norm (like missing data from key accounts) can trigger alerts for further investigation. Data Quality Checks at the Source: Develop specific data quality checks within the data ingestion pipeline that assess the completeness of account-level prescription data from syndicated data providers. If key account data is missing, this would trigger a notification to your data management team for immediate follow-up with the data providers. Redundant Data Sources: To cross-check, leverage additional data providers or internal data sources (such as sales team reports or pharmacy-level data). By comparing datasets, missing data from syndicated data providers can be quickly identified and verified. Data Stewardship and Monitoring: Assign data stewards or a dedicated team to monitor data feeds from syndicated data providers. These stewards can track patterns in missing data and work closely with data providers to resolve any systemic issues. Regular Audits and SLA Agreements: Establish a service level agreement (SLA) with data providers that includes specific penalties or remedies for missing or delayed data from key accounts. Regularly auditing the data against these SLAs ensures timely identification and correction of missing prescription data. By addressing data quality challenges with advanced technologies and robust management practices, the industry can reduce financial losses, improve operational efficiency, and ultimately enhance patient outcomes.

GenAI has taken the world by storm and entered the boardrooms, exec suites and labs of most major firms. However the question of how to effectively enable for impact and scale these capabilities is not discussed enough and most have yet to overcome this challenge. My co-authors (Joe Caserta, Holger Harreis, Nikhil Srinidhi and Dr. Asin Tavakoli) and I have identified seven actions that data leaders should consider as they move from experimentation to scale. These include: 1) Let value be your guide. CDOs need to be clear about where the value is and what data is needed to deliver it. 2) Build specific capabilities into the data architecture to support the broadest set of use cases. Build relevant capabilities (such as vector databases and data pre- and post-processing pipelines) into the existing data architecture, particularly in support of unstructured data. 3) Focus on key points of the data life cycle to ensure high quality. Develop multiple interventions—both human and automated—into the data life cycle from source to consumption to ensure the quality of all material data, including unstructured data. 4) Protect your sensitive data, and be ready to move quickly as regulations emerge. Focus on securing the enterprise’s proprietary data and protecting personal information while actively monitoring a fluid regulatory environment. 5) Build up data engineering talent. Focus on finding the handful of people who are critical to implementing your data program, with a shift toward more data engineers and fewer data scientists. 6) Use generative AI to help you manage your own data. Generative AI can accelerate existing tasks and improve how they’re done along the entire data value chain, from data engineering to data governance and data analysis. 7) Track rigorously and intervene quickly. Invest in performance and financial measurement, and closely monitor implementations to continuously improve data performance Happy reading. #data #genai #datascience #ai #analytics #mckinsey

Data cleansing is the most manually intensive activity in data management. Only 14% of organizations have implemented operational tools that automate data quality management processes like profiling, matching, correction, and enhancement. The rest rely on data stewards to close the loop manually. On the other hand, data quality issues detection tools have gotten dramatically better over the past decade: ML-based anomaly detection, automated profiling, and real-time monitoring. But every alert still ends the same way. A steward exports the bad records, chases the source system owner, fixes the data in a spreadsheet, and reimports. Organizations operationalized detection but not remediation. Agentic data cleansing changes that. It uses specialized AI agents that detect failures, analyze the source to find what "correct" looks like, propose a targeted fix, and wait for a human to accept. The steward governs. The agent does the janitorial work. This latest guide covers three generations of data cleansing (manual scripts, rule-based automation, and agentic cleansing), an evaluation framework for deciding when your team needs agentic cleansing, a market landscape comparison, and the architectural insight that makes contract-driven remediation possible. Whether you're a data steward drowning in alert fatigue or a governance leader trying to operationalize quality beyond observability, this is the resource you'll come back to. Read here: https://lnkd.in/d-ceTJzU

An often over-looked, but really important part of any data management framework (DMF) is Reference Data! Any data steward (aka data nerd) will tell you how fun it is to do an exercise to line up reference data lists across different business units or orgs within the same entity. In global trade it becomes important because it ties directly to the compliance posture of a company. Reference Data Management (RDM) is how organizations keep all their “lookup values” - for example country codes, HTS numbers, license types, and incoterms - consistent across every system that uses them. When one system says “US” and another says “USA,” or one system still uses an outdated tariff code, you end up with mismatches that can delay shipments, create reporting errors, or even cause regulatory fines. Reference data management for global trade personnel really is the discipline of defining, governing, and synchronizing all those shared reference lists that underpin every transaction. Here are five simple steps for managing Reference Data as part of your Data Management Framework: 1. Inventory your reference data: identify which lists drive your trade processes - HTS codes, ECCNs, country codes, license types, etc. 2. Define a single “source of truth” - decide where each list should be mastered — for example, tariff codes might live in a central classification database. 3. Establish update cycles and owners: assign ownership (who updates) and timing (how often lists are refreshed, especially after regulatory changes). 4. Synchronize across systems: Work with IT to ensure updates cascade automatically to ERP, trade management, and customs filing systems. 5. Monitor and audit consistency: Track changes and report on mismatches - prevention is far cheaper than rework at customs. Not glamorous at all - but a critical layer of a trade data foundation. When your reference data is clean, current, and connected, the rest of your compliance data management can run a lot mroe smoothly.