Data Governance in Engineering Projects

Two teams. Same data. One made a $3M mistake. The first team had perfect pipelines. Fast ingestion. Clean transformations. Dashboards refreshing every 15 minutes. They still made a critical decision based on a metric nobody agreed how to define. The second team moved slower. Pipelines weren't as polished. But every metric had an owner. Every definition was documented. Every access request had a reason. Better decisions. Worse infrastructure. That's the difference between Data Management and Data Governance. And confusing the two is the most expensive mistake in enterprise data right now. 𝗗𝗮𝘁𝗮 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 is how data moves. Ingestion. Pipelines. Storage. Quality checks. Dashboards. Extracts. Operational. Technical. Mostly automated. This is the work. 𝗗𝗮𝘁𝗮 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 is how data is understood. Definitions. Standards. Ownership. Privacy. Security. Access controls. Strategic. Policy-driven. Cultural. This is the decisions. And in the age of AI agents this gap is getting more dangerous. Your LLM doesn't know that sales and finance define revenue differently. It just picks one and runs with it. That's not a hallucination. That's a governance failure disguised as an AI problem. World-class data management can still produce terrible decisions. Because fast pipelines don't fix a revenue metric that two teams define differently. And bulletproof governance means nothing without data movement. Data Management is the engine. Data Governance is the GPS. Engine without GPS? Fast in the wrong direction. GPS without engine? Standing still. Here's what to do with this right now: 𝟭. Pick your top 5 metrics. Ask 3 people in different teams to define them. If the definitions don't match you have a governance problem. 𝟮. Check your pipeline SLAs. If data is late and nobody notices you have a management problem. 𝟯. Find one metric with no owner. Assign one by Friday. That single move will prevent more bad decisions than any new tool. Not governance after a breach. Not management without standards. Both. From day one. That's the difference between a data team that ships reports and a data team that powers AI systems people actually trust. ♻️ Repost, this can save careers not just projects 🔔 Follow Gabriel for daily AI transformation insights that cut through the noise Graphic: John Wernfeldt

Can Data Governance Shift Left: Embedded in Code, Not Spreadsheets. Is it possible? Most governance frameworks fail, Why? Because they never make it into the code or at minimum integrated with daily processes & procedures. Modern Data Governance is Policy + Procedures + Pipeline. Most governance platforms sit outside your engineering workflows. They require data to land before you can catalog, classify, or monitor the quality. That’s governance after the fact, reactive and bolt-on. In a recent Data on Tap episode with Vamsi Kunaparaju, we talked about how modern data governance needs to shift left and built-in: metadata capture, lineage mapping, quality checks, and compliance tagging happen inside your pipelines, at commit time, before data ever hits production. No extra tools, 100's of connectors/integrations, no double-handling, no lag between creation and governance.Not in PDFs, Excel and SharePoint folders. This framework (see image) shows how we connect: - Policy & Requirements → Regulatory, quality, and access controls - Data Capabilities → Automated cataloging, compliance tagging, anomaly detection - Engineering → Version control, observability, lineage monitoring in the lakehouse - Change Management → Skills, processes, and training for adoption When you treat Data Governance as Code: ✅ Metadata, classification, and tagging is auto-generated during commits ✅ Lineage & quality checks run in CI/CD ✅ Access & compliance rules are enforced instantly ✅ Observability is continuous from ingestion to dashboard Let’s start talking about pipelines and automated procedures not just policies. 📢 Worth a listen below 👇: Shift Left Data Governance

You've heard "garbage in, garbage out" a thousand times. But here's what that actually means: your fancy dashboard is only as good as the data behind it. Quantity is easy to measure—it's just Terabytes. But data quality? Quality is the hard part because it requires discipline, process, and ownership. Data quality and governance are no longer “nice-to-haves.” They define trust across the organization. → Growing demand due to privacy laws like GDPR and CCPA → Core skill required for roles like Data Engineer, Steward, and Architect → Tools like Collibra and Great Expectations now appear in almost every data job description Some numbers speak for themselves: → Data Quality Engineer roles growing 40%+ yearly → Governance Analysts earning around $80K–$120K → Chief Data Officers often crossing $200K+ Clean data isn’t just accuracy—it’s career growth and company credibility. What Good Data Quality Looks Like? Skip the theory. Here's what actually works: → Automated checks that catch issues before they spread → Validation rules that reject bad data at the source → Tracking where data comes from and where it goes → Alerts when something breaks (not after it's been broken for weeks) → Clear ownership so someone actually fixes problems Where in the real world it shows up? 👉This isn't abstract. Here's where data quality makes or breaks things: → Finance: Try explaining bad compliance data to auditors → Healthcare: Patient records need to be right, every time → Retail: Wrong inventory data means lost sales or wasted stock → ML projects: Your model is only as smart as your training data The Real Talk: Data quality feels boring until it's missing. Then suddenly everyone cares. It's not sexy work. Nobody celebrates when pipelines validate correctly. But it's the foundation everything else sits on. Gartner says organizations with formal data governance will see 30% higher ROI by 2026. As data engineers, that’s our call to design solutions that "𝘥𝘰𝘯’𝘵 𝘫𝘶𝘴𝘵 𝘮𝘰𝘷𝘦 𝘥𝘢𝘵𝘢, 𝘣𝘶𝘵 𝘮𝘰𝘷𝘦 𝘵𝘳𝘶𝘴𝘵." Honestly, I feel it's probably more if you count all the fires you don't have to fight. 👉 Folks I admire in this space - George Firican Dylan Anderson Piotr Czarnas 🎯 Mark Freeman II Chad Sanderson Here's a crisp guide on Data Quality & Governance for data engineers! 👇 What's the most annoying, recurring data quality issue you've had to fix lately? I'll go first: dates stored as strings. 🤦♂️

Data governance is usually framed as a compliance problem. In reality, it’s a human one. Good data governance is about building trust. We were brought in to build the data platform for a compensation programme handling highly sensitive medical, legal and financial information. The technical requirements were substantial. → Zero trust architecture → Role based access controls → Infrastructure as Code for rapid deployment Case officers needed to make decisions about compensation claims. Those decisions depended entirely on having reliable, complete information. Vulnerable citizens needed to trust that their sensitive data was protected and their claims would be handled with dignity and accuracy. Before the platform existed, data was fragmented. Spreadsheets scattered across teams. Manual reconciliation consuming hours that should have been spent on casework. No single source of truth. What this meant in practice was → Case officers spent time cross-referencing files instead of supporting claimants → Data inconsistencies created delays and uncertainty → Citizens had no visibility of claim status or timelines Building a unified data platform was about giving case officers the reliable foundation they needed to do their jobs effectively. And it was about treating vulnerable people with the dignity they deserve by ensuring their information was handled with care, accuracy and transparency. When you unify case data and eliminate spreadsheet sprawl, you restore trust in a broken system. Good data governance enables people to do meaningful work. That is what matters. What is the human cost of poor data governance in your organisation? #DataGovernance #PublicSector #Trust

Over the past 10+ years, I’ve had the opportunity to author or contribute to over 100 #datagovernance strategies and frameworks across all kinds of industries and organizations. Every one of them had its own challenges, but I started to notice something: there’s actually a consistent way to approach #data governance that seems to work as a starting point, no matter the region or the sector. I’ve put that into a single framework I now reuse and adapt again and again. Why does it matter? Getting this framework in place early is one of the most important things you can do. It helps people understand what data governance is (and what it isn’t), sets clear expectations, and makes it way easier to drive adoption across teams. A well-structured framework provides a simple, repeatable visual that you can use over and over again to explain data governance and how you plan to implement it across the organization. You’ll find the visual attached. I broke it down into five core components: 🔹 #Strategy – This is the foundation. It defines why data governance matters in your org and what you’re trying to achieve. Without it, governance will be or become reactive and fragmented. 🔹 #Capability areas – These are the core disciplines like policies & standards, data quality, metadata, architecture, and more. They serve as the building blocks of governance, making sure that all the essential topics are covered in a clear and structured way. 🔹 #Implementation – This one is a bit unique because most high-level frameworks leave it out. It’s where things actually come to life. It’s about defining who’s doing what (roles) and where they’re doing it (domains), so governance is actually embedded in the business, not just talked about. This is where your key levers of adoption sit. 🔹 #Technology enablement – The tools and platforms that bring governance to life. From catalogs to stewardship platforms, these help you scale governance across teams, systems, and geographies. 🔹 #Governance of governance – Sounds meta, but it’s essential. This is how you make sure the rest of the framework is actually covered and tracked — with the right coordination, forums, metrics, and accountability to keep things moving and keep each other honest. In next weeks, I’ll go a bit deeper into one or two of these. For the full article ➡️ https://lnkd.in/ek5Yue_H

🚨 #HonestNoBS: Data governance has a branding problem. It’s been labeled as boring, bureaucratic, and the team of “No. But here’s the truth: Governance is finally exciting because it’s the carrot for AI. It’s not about slowing things down, it’s about safe speed. If your data isn’t: ✅ Understood (semantics) ✅ Trusted (business value) ✅ Usable (data products with clear context) ✅ Delivering fast wins (iterative, targeted effort) …then it’s not ready for AI. Here’s how to make governance actually work: 🔥 1. Minimum Valuable Governance Just enough governance to unlock value quickly. No overkill. What this looks like: • Start with the use case, not the policy manual. • Define only what’s necessary (clear terms, roles, semantics). • Engage the right stakeholders early, not everyone all at once. • Allow just enough access and quality to meet the goal. • Use an iterative approach — show quick wins, improve from there. 🛑 No more “boil the ocean” governance programs. ✅ Yes to fit-for-purpose, low-friction, value-first moves. 💡 2. Embedded Governance Built into how people already work, not a separate compliance layer. What this looks like: • Co-design with the business. Front office defines the “what & why,” back office enables the “how.” • Think like an energy company: governance is safety, and everyone owns it. • Governance pioneers = internal personal trainers. Empower, don’t enforce. • Bake governance into tools, workflows, and daily habits — not just into frameworks. Governance isn’t a team: it’s a culture. 📦 3. Data Products & Marketplace Reusable, governed assets people can actually find and use. What this looks like: • Define clear product boundaries and ownership. • Wrap data in contracts: semantics, SLAs, and accountability. • Focus on usability, build for the consumer, not the committee. • Measure impact: usage, satisfaction, business value. And at the center of it all? Metadata. But not stale, siloed metadata. We’re talking: Graphs. Context. Shareability. Here’s what kills governance efforts: ❌ Overengineering & scope creep ❌ Weak communication ❌ No ownership or accountability This is the talk that Tim Gasper and I will be giving at Snowflake today. Our thinking and POV comes from talking to hundreds of day leaders and practitioners, our Catalog & Cocktails Podcast guests (special shoutout to Rebecca O'Kill and Winfried Adalbert Etzel )

Why do so many AI and analytics projects struggle even after the tech is in place? This visual on layers of data governance explains it better than most frameworks I have seen. Most organizations think governance starts with tools. Data catalogs, lineage dashboards, or quality checks. But those sit in the middle, not at the foundation. At the bottom, every company runs on operational reality. Source systems, manual fixes, Excel workarounds, and the one person who “knows how it works.” Above that comes transparency and data management. Pipelines, storage, ingestion. This is where many teams stop and assume they are governed. But real governance starts higher. Clear data quality rules. Shared definitions. Accountability for decisions. A governance operating model that actually runs, not just policies stored in a folder. Only when these layers exist do you get what leaders actually want. Trust at scale. Confident decisions. Self service that does not create chaos. And only then do AI and automation deliver real value instead of fragile demos.

At its core, data quality is an issue of trust. As organizations scale their data operations, maintaining trust between stakeholders becomes critical to effective data governance. Three key stakeholders must align in any effective data governance framework: 1️⃣ Data consumers (analysts preparing dashboards, executives reviewing insights, and marketing teams relying on events to run campaigns) 2️⃣ Data producers (engineers instrumenting events in apps) 3️⃣ Data infrastructure teams (ones managing pipelines to move data from producers to consumers) Tools like RudderStack’s managed pipelines and data catalogs can help, but they can only go so far. Achieving true data quality depends on how these teams collaborate to build trust. Here's what we've learned working with sophisticated data teams: 🥇 Start with engineering best practices: Your data governance should mirror your engineering rigor. Version control (e.g. Git) for tracking plans, peer reviews for changes, and automated testing aren't just engineering concepts—they're foundations of reliable data. 🦾 Leverage automation: Manual processes are error-prone. Tools like RudderTyper help engineering teams maintain consistency by generating analytics library wrappers based on their tracking plans. This automation ensures events align with specifications while reducing the cognitive load of data governance. 🔗 Bridge the technical divide: Data governance can't succeed if technical and business teams operate in silos. Provide user-friendly interfaces for non-technical stakeholders to review and approve changes (e.g., they shouldn’t have to rely on Git pull requests). This isn't just about ease of use—it's about enabling true cross-functional data ownership. 👀 Track requests transparently: Changes requested by consumers (e.g., new events or properties) should be logged in a project management tool and referenced in commits. ‼️ Set circuit breakers and alerts: Infrastructure teams should implement circuit breakers for critical events to catch and resolve issues promptly. Use robust monitoring systems and alerting mechanisms to detect data anomalies in real time. ✅ Assign clear ownership: Clearly define who is responsible for events and pipelines, making it easy to address questions or issues. 📄Maintain documentation: Keep standardized, up-to-date documentation accessible to all stakeholders to ensure alignment. By bridging gaps and refining processes, we can enhance trust in data and unlock better outcomes for everyone involved. Organizations that get this right don't just improve their data quality–they transform data into a strategic asset. What are some best practices in data management that you’ve found most effective in building trust across your organization? #DataGovernance #Leadership #DataQuality #DataEngineering #RudderStack

The Unwritten Laws of Data Governance Data governance used to be a checkbox; today it’s a competitive weapon. The unwritten laws start with treating your datasets like products. That means clear ownership, versioned “releases,” and customer-style SLAs for freshness and quality, no more blaming pipelines when things break. Next, bake governance into daily workflows instead of relegating it to policy manuals. Self-service tooling, real-time lineage tracing, and automated quality gates turn compliance into frictionless guardrails that speed projects up. Finally, measure trust as a leading KPI(Key Performance Indicator). Track data-health scores, monitor usage patterns, and loop stakeholder feedback back into your roadmap. When governance feels like an enabler-not a burden-you unlock faster decisions, deeper insights, and a real strategic edge.

🧩 Output ≠ Outcome. Just because you've produced documentation, implemented tools, or hosted governance meetings (outputs) doesn't mean you've achieved better data quality, trust, or adoption (outcomes). In data stewardship and governance, outcomes rely more on alignment, trust, and collaboration than just ticking off deliverables. - Output: You’ve defined data owners, documented data definitions, or built dashboards. - Outcome: People actually use those definitions to make consistent decisions and trust the data. 👉 Outputs are necessary, but not sufficient for meaningful outcomes. So how do you drive data governance outcomes? 1. SME Alignment. - Subject Matter Experts must agree on definitions, metrics, and ownership. - Alignment across teams creates the foundation for trustworthy data. 2. Cultural Adoption > Tooling. - Even the best governance tools fail if people don’t believe in the process. - Build buy-in through collaboration, not mandates. 3. Define Shared Goals. - Governance should be aligned with business goals (e.g., “reduce revenue leakage due to misreported metrics”). - When governance feels relevant, people care more. 4. Communication Loops. - Regular check-ins between SMEs, data stewards, and consumers help reinforce governance. - It’s an ongoing conversation, not a one-time project. 5. Governance as Enablement, Not Control. - Frame governance as a way to accelerate trusted access and a support system, not a gatekeeping function. - The best way for end users to truly understand data is to use it in real scenarios—to ask questions, spot inconsistencies, and explore patterns. 6. Measure the Right Things. - Don’t just measure “number of governed datasets.” - Measure “reduction in data-related escalations,” “increase in self-service usage,” or “time saved in resolving data issues.” Anything I missed? This Friday, I'm having Jeff Rosen over on Inner Join to talk about this specific topic - accelerating data stewardship & data governance in the age of AI. Please join us for our deep dive conversation 🤠! #datagovernance #datastewardship #dataai #metadata