Delivering Trusted Data Without Disrupting Workflows

I violated data best practices to deliver a $40K ROI. (The client renewed. Here's why.) For 4 years, I've preached data best practices: Build proper data models. Minimize tech debt. Do it right the first time. Then reality hits. A mid-sized healthcare company hires us. They need a manual report automated. Fast. Your offer as a consultant is speed-centric. Their "source of truth" is 400 stored procedures written by a DBA who left 2 years ago. Zero documentation. Spaghetti SQL everywhere. 30+ Power BI reports querying directly off the transactional database. 𝗛𝗲𝗿𝗲'𝘀 𝘄𝗵𝗮𝘁 𝗜 𝘄𝗮𝗻𝘁𝗲𝗱 𝘁𝗼 𝗱𝗼: Build a clean data warehouse from scratch. Proper dimensional modeling. Governed metrics. Best practices. 𝗛𝗲𝗿𝗲'𝘀 𝘄𝗵𝗮𝘁 𝗜 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗱𝗶𝗱: Replicated their messy legacy logic in the cloud. Matched their numbers exactly—even the parts I knew were questionable. Automated the manual report in 6 weeks. Delivered the $40K ROI we guaranteed. 𝗪𝗵𝘆? Because many executives don't care about best practices. They care about results. Now. You don't get 3-6 months to "do it right." You get 6 weeks to prove you're worth keeping. 𝗧𝗵𝗲 𝘁𝗿𝘂𝘀𝘁-𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗽𝗮𝗿𝗮𝗱𝗼𝘅: If you show up and tell them their legacy logic is wrong, they won't trust you. If you replicate it perfectly first, they do. Once trust is built? Then you can challenge the legacy logic. Then you can propose the proper data model. Then you can start fixing the mess. But not before. 𝗛𝗲𝗿𝗲'𝘀 𝗵𝗼𝘄 𝘁𝗼 𝗯𝗮𝗹𝗮𝗻𝗰𝗲 𝘀𝗽𝗲𝗲𝗱 𝗮𝗻𝗱 𝗾𝘂𝗮𝗹𝗶𝘁𝘆: 𝗗𝗲𝗹𝗶𝘃𝗲𝗿 𝗾𝘂𝗶𝗰𝗸 𝘄𝗶𝗻𝘀 𝘁𝗵𝗮𝘁 𝗲𝘀𝘁𝗮𝗯𝗹𝗶𝘀𝗵 𝘁𝗿𝘂𝘀𝘁 Automate one critical report. Match legacy numbers. Show ROI fast. 𝗢𝘃𝗲𝗿𝗰𝗼𝗺𝗺𝘂𝗻𝗶𝗰𝗮𝘁𝗲 𝘁𝗵𝗲 𝘁𝗿𝗮𝗱𝗲-𝗼𝗳𝗳𝘀 "This works, but it creates tech debt. Here's the plan to fix it long-term." 𝗖𝗮𝗿𝘃𝗲 𝗼𝘂𝘁 𝘁𝗶𝗺𝗲 𝗳𝗼𝗿 𝘁𝗵𝗲 𝗿𝗲𝗯𝘂𝗶𝗹𝗱 Once trust is established, allocate hours to build the proper foundation. 𝗞𝗲𝗲𝗽 𝗱𝗲𝗹𝗶𝘃𝗲𝗿𝗶𝗻𝗴 𝘃𝗮𝗹𝘂𝗲 𝘄𝗵𝗶𝗹𝗲 𝘆𝗼𝘂 𝗶𝗺𝗽𝗿𝗼𝘃𝗲 Don't stop showing ROI while you refactor. Balance both. 𝗧𝗟;𝗗𝗥: Best practices are the North Star. But speed to value is survival. Deliver quick wins. Build trust. Then improve the foundation. Perfection kills consulting businesses. Progress builds them. Agree or Disagree? ♻️ Share this if you've ever had to choose between doing it "right" and doing it "fast." Follow me for real talk on what data consulting actually looks like in the wild.

**Building Trust in Data: My Experience with Delta Live Tables and Unity Catalog** I spent time optimizing one of our Azure pipelines, and it reminded me why I love solving real problems in data engineering but also why it can keep you up at night. **The Challenge:** We needed to process multi-channel customer interactions (voice, chat, email) in near real-time. Our Bronze layer was ingesting data smoothly, but our downstream teams needed clean, validated data within minutes. The pressure was on to deliver reliable data without adding operational complexity. **What worked:** I turned to Delta Live Tables (DLT), and honestly, it changed how I think about pipeline development. Instead of writing complex orchestration logic, I could focus on *what* the data should look like, not *how* to move it around. DLT handled dependencies, retries, and quality checks automatically. The EXPECT clauses gave me a simple way to enforce data quality rules right in the pipeline definition. Here’s a simplified example of what the transformation looked like: @dlt.table( name="silver_interactions", comment="Cleaned and validated multi-channel interaction data" ) @dlt.expect_all({"non_null_ids": "interaction_id IS NOT NULL"}) def silver_interactions(): return ( dlt.read("bronze_interactions") .filter("status IS NOT NULL") .withColumn("ingestion_date", current_date()) ) **The governance piece:** Unity Catalog brought peace of mind on the security and compliance side. Instead of managing permissions workspace by workspace, we established consistent access controls across all environments. The automatic lineage tracking meant we could answer “where did this data come from?” in seconds, not hours. **What I learned:** Good engineering isn’t just about making things work — it’s about making them maintainable, secure, and trustworthy. This combination gave us: • Declarative pipelines that are easier to understand and maintain • Built-in data quality monitoring • Centralized governance without extra overhead • Confidence that our data meets business standards The best part? Our team can now focus on building features instead of firefighting pipeline failures. If you’re managing complex ETL workflows or struggling with data governance across teams, I’d encourage you to explore this approach. The learning curve is worth it. #DataEngineering #Databricks #DeltaLiveTables #UnityCatalog #Azure #DataGovernance #RealTimePipeline

The core focus of our internal digital transformation is data strategy 📊 Over the past months, we have moved from fragmented pipelines and isolated analytics to a working Data Fabric. Hundreds of tables and data marts are migrated, ingestion is orchestrated through Airflow with CDC, versioning is enforced via CI/CD, and access is unified across analytical tools. This is no longer an experiment. It is a production-grade platform. This work is led by Айжан Аринова, PMP, ПМКР and a strong cross-functional team combining data engineering, governance, and analytics. The focus is not migration for the sake of numbers, but building something that works reliably every day. A key shift is moving from raw datasets to data products. Each data product has an owner, lifecycle, quality rules, access logic, and a clear business purpose. This is how analytics becomes reusable and scalable across monetary policy, supervision, payments, and financial stability. This foundation is critical for AI 🤖 AI systems do not fail because of models. They fail because of data. Inconsistent reference data, unclear lineage, duplicated datasets, and silent quality issues create risks that no algorithm can fix. Governed data products become the unit of reuse for AI. Master data ensures models learn real economic signals, not artifacts of system fragmentation. Lineage and metadata enable explainability. Quality controls embedded into pipelines reduce model risk and drift. Our key metrics are simple and demanding: time2data and time2analytics. How fast trusted data becomes available. How fast it turns into analysis, insight, or decision support. These metrics matter even more as we scale AI across the organization. In 2026, the focus shifts from migration metrics to outcomes. Fewer parallel systems. Less manual reconciliation. More governed domains and production data products used daily by economists, supervisors, and analysts. Work in progress. Direction is clear 🚀 #DataStrategy #DataFabric #DataProducts #AIReadiness #DataGovernance #Analytics #DigitalTransformation #FinData

I’ve seen a clear pattern across enterprises with 𝗦𝗔𝗣 at the core of their 𝗘𝗥𝗣 landscape The core business data is rock solid.  The questions from the business are sharp.  But the path from SAP data → trusted insights is still more complex than it should be. Too many point tools.  Too much stitching.  Too much time spent just getting data ready. This is where SAP’s 𝗕𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗗𝗮𝘁𝗮 𝗖𝗹𝗼𝘂𝗱 (𝗕𝗗𝗖) (combination of 𝗦𝗔𝗣+𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀) feels like a very intentional shift. At its heart, BDC is SAP saying:  “Let’s stop treating integration, semantics, and access as custom projects every time.” 𝗗𝗮𝘁𝗮 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝘀 are the big idea here, curated, standardized datasets coming out of SAP systems, already shaped for analytics and AI use cases. No repeated extraction logic. No reinventing business definitions across teams. Behind the scenes, SAP’s 𝗙𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻 𝗦𝗲𝗿𝘃𝗶𝗰𝗲𝘀 handle extraction, transformation, and publishing, so consumers don’t have to worry about plumbing. And with 𝗗𝗲𝗹𝘁𝗮 𝗦𝗵𝗮𝗿𝗶𝗻𝗴, those Data Products aren’t locked inside SAP walls anymore. That openness is critical. Because while SAP understands enterprise business context better than anyone, 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗲𝘅𝗰𝗲𝗹𝘀 𝗮𝘁 𝗹𝗮𝗿𝗴𝗲-𝘀𝗰𝗮𝗹𝗲 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴, 𝗮𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗮𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀, 𝗮𝗻𝗱 𝗠𝗟. Together, the pattern becomes powerful: ▪ SAP BDC provides 𝘁𝗿𝘂𝘀𝘁𝗲𝗱, 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀-𝗿𝗲𝗮𝗱𝘆 𝗱𝗮𝘁𝗮  ▪ Databricks provides 𝘀𝗰𝗮𝗹𝗲, 𝗳𝗹𝗲𝘅𝗶𝗯𝗶𝗹𝗶𝘁𝘆, 𝗮𝗻𝗱 𝗮𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴  ▪ Teams spend less time arguing over data definitions and more time delivering outcomes The result? ▪ Fewer silos  ▪ Faster insights  ▪ A much cleaner separation between business semantics and engineering innovation To me, BDC isn’t “yet another data platform.”  It’s SAP tightening the foundation and intentionally leaving the door open for platforms like Databricks to do what they do best. Curious to see how enterprises operationalize this combo over the next year, especially for AI-driven scenarios. #SAP #BusinessDataCloud #Databricks #DataArchitecture #EnterpriseData #Analytics #AI 

FINDING BALANCE IN DATA PROTECTION One day, I had a discussion with an HR team about their recruitment process and the significant amount of personal data they collected from candidates. When I suggested minimizing the data collected to reduce privacy risks, they raised a concern: the less data they gather, the harder it becomes to make confident decisions, potentially increasing risks in the hiring process. This scenario reflects a broader challenge faced by many organizations, not just HR teams. Balancing operational needs with data protection compliance requires careful consideration. Data protection principles emphasize collecting only the necessary data for a specific purpose, but this can often feel at odds with operational goals that rely on detailed information for effective decision-making. The key is redefining what “necessary” means in each context. For example, organizations can implement phased approaches to data collection, gathering only essential details early on and requesting more sensitive information later, if required. This practice not only reduces data risks but also aligns operational processes with privacy standards. Achieving this balance is not about choosing between compliance and efficiency, it’s about integrating them. By designing workflows that respect data protection while supporting business objectives, organizations can build trust, reduce risks, and demonstrate a commitment to ethical practices in an increasingly privacy-conscious world.

Most data pipelines don’t fail because of tools. They fail because of architecture decisions made too early — without understanding data behavior. Let’s talk about something that separates senior Data Engineers from everyone else: Designing pipelines for data characteristics, not just for delivery. In mature platforms, the real challenges are not ingestion. They are: • Handling late-arriving data without corrupting aggregates • Designing idempotent pipelines that survive retries • Balancing BigQuery partitioning vs clustering vs slot usage • Preventing small files problem in object storage • Choosing watermarking strategy in streaming (event-time vs processing-time) • Schema evolution without breaking downstream contracts • Building backfills that don’t bankrupt your cloud bill • Making pipelines observable, not just “green in Airflow” For example: If you’re using BigQuery and still partitioning everything by ingestion_date by default, you’re likely optimizing for convenience, not performance. Event-time partitioning combined with selective clustering can reduce scan cost by 60–80% on large analytical workloads — but only if your upstream pipeline guarantees correctness. Similarly, real-time pipelines aren’t “Pub/Sub → Dataflow → BigQuery”. That’s the diagram. The engineering is in: • Deduplication strategy • Exactly-once semantics • Windowing strategy • State expiration • Replayability • Dead-letter handling The maturity of a data platform is visible in how confidently it handles failure. Because failure is not an edge case in data engineering. It is the default state. Good pipelines move data. Great pipelines protect trust in data. That’s the difference between building systems and engineering platforms. #DataEngineering #BigQuery #StreamingData #CloudArchitecture #GCP #AnalyticsEngineering #TechLeadership

📈 This isn’t just a story about Data analytics. It’s a blueprint for telecoms and other transaction high-volume industries looking to turn complexity into clarity—and insights into impact! 🚀 For telecom companies handling terabytes of billing data each day, analytics can no longer be an afterthought. One U.S.-based broadband provider serving 20 states set out to fix a critical challenge: fragmented, delayed, and manual data processes that slowed down revenue recognition, payment tracking, and financial reporting. Billing records, revenue recognition events, payment processing logs—each minute adds gigabytes of information across critical systems like SAP S/4HANA and SAP Billing and Revenue Innovation Management (BRIM). The business imperative was clear: unlock real-time insights from this ocean of data and do it at scale, efficiently, and securely. The company’s prior approach involved custom queries, end-of-day updates, and constant monitoring—creating a bottleneck in business decision-making. The transformation began with a bold ambition: harmonize data in real time across SAP S/4HANA, BRIM, and Google BigQuery, while eliminating manual overhead and enabling a truly intelligent finance function. The solution? SAP Datasphere, acting as the heart of a modern data fabric. By implementing real-time replication flows with change data capture, and embedding a semantic layer natively in Datasphere, the team enabled curated, trusted data to flow directly into BigQuery and SAP Analytics Cloud—supporting operational and strategic decisions in near real-time. Behind the scenes, this was powered by SAP Business Technology Platform, bridging core transactional systems with cloud analytics, all while maintaining a clean core and future-ready architecture.  The IT and finance teams partnered to rethink data architecture from the ground up. Instead of traditional SLT-based replication, which had previously slowed down operational systems and lacked semantic context, the team leveraged SAP #Datasphere ’s replication flows with change data capture. 🏆 Results that Matter: • Reports that once took hours now run in real-time • 2 FTEs worth of manual effort saved weekly • 40M billing records processed monthly with precision • Foundation laid for AI-driven analytics, forecasting, and insights What made this a standout transformation wasn’t just the technology—it was the mindset. Cloud-first, agile, and committed to making data a strategic asset, not a side project! 🚀 #SAP #SAPDatasphere #Analytics #SAPBTP #S4HANA #DataStrategy #SAPAnalyticsCloud #DataArchitecture Check out the full case study in the comments section! 🚀

The HHS ChatGPT Rollout: Why We Built @Data2us Differently The Department of Health and Human Services just announced the immediate rollout of ChatGPT across the entire agency. As someone who's worked with companies on AI implementation, this perfectly illustrates the problems that drove us to build Data² differently. The Core Problem The HHS email encourages employees to input "procurement sensitive data and routine non-sensitive personally identifiable information" while stating ChatGPT "is currently not approved for disclosure of sensitive personally identifiable information." The guidance doesn't distinguish what constitutes "non-sensitive" versus "sensitive" PII, leaving employees to make judgment calls they're not equipped to make. Even worse, the email tells employees to "be skeptical of everything you read, watch for potential bias, and treat answers as suggestions", essentially acknowledging the system can't be trusted while mandating its use for public health policy. Why We Built Differently When we started working with federal agencies, they consistently told us: "We need AI capabilities, but we can't use systems where we can't trace how decisions are made." This led us to develop our patent around hallucination-resistant, explainable AI. Our reView platform addresses exactly what HHS is struggling with. Instead of telling users to "be skeptical," we provide visual traceability showing exactly what data was analyzed and how conclusions were reached. See our reasoner in action: https://lnkd.in/gVWyX9tq Integration Without Disruption Rather than replacing workflows, we built Data² as an integration layer that enhances existing capabilities while adding explainability. Organizations preserve investments and gain transparency to actually trust AI outputs. The HHS rollout demonstrates why we took a fundamentally different approach. Instead of asking workers to manage unexplainable AI risks, we eliminate them by ensuring every output comes with complete justification. High-reliability organizations deserve both powerful and accountable AI. #AI #Government #Explainability #DataGovernance #ResponsibleAI

Here’s the security dilemma many teams face: we block file sharing in external Teams chats to protect data, and within a week users are emailing attachments to get work done. The data leaves our tenant forever. We've traded a controlled risk for an uncontrolled one. This is a fundamental security challenge: the gap between what policy intends and what people actually do when policy gets in the way. That's why I'm excited about what we just shipped at eSHARE. Trusted Sharing for Microsoft Teams Chat lets users paste an M365 link into any Teams chat with external participants, and eSHARE instantly wraps it in a Trusted Share. The file never leaves our tenant. Access governance is enforced automatically. And the user never leaves their flow of work. No new app. No extra steps. No reason to fall back to email. Below is a quick demo showing how it works. What I love about this from a trust and governance standpoint: it doesn't ask users to choose between doing their job and following policy. It removes that choice entirely. IT gets to say yes to external collaboration in Teams without compromising on control. Guardrails, not barriers. That's the philosophy. This is what it looks like in practice. #TrustedCollaboration #MicrosoftTeams #DataGovernance #ExternalCollaboration #ZeroTrust

Design of Experiments only pays off when your data is trustworthy, connected, and ready to analyze.     Most teams don’t have a data problem. They have a context problem. Experiments cross people, sites, instruments and time, yet the data arrives fragmented. That invites errors, slows tech transfer and forces your scientists to clean data instead of learning from it.     What’s worked across complex pipelines is building a digital backbone that keeps process context attached to every sample and step. In practice, that looks like process-centric workflows, versioning of methods and materials, automatic sample IDs and lineage, QC checks against specs, and instant creation of analysis-ready data frames. When process changes, the data structure updates with it, so your DoE stays intact and computable.     One line from my notes for leaders: aim for FAIR by design. Data should be findable, accessible, interoperable and reusable as it’s collected, not after the fact. When teams can capture experiment context, aggregate instrument and manual inputs, join data across unit operations and run real-time visualization or ML, throughput rises and transfer friction drops. This approach has shown time-to-market reductions, screening throughput increases, and major cuts in data prep effort.     In regulated work, don’t forget the guardrails. Audit trails, electronic signatures for completed experiments, and role-based access keep governance tight while letting collaborators contribute. APIs and SQL access matter too, because DoE is strongest when it connects to your analytics stack and master data.     Try this: pick one high-variance process, map the workflow end-to-end, assign permanent IDs to samples, and enforce QC ranges at data entry. Then push the resulting data frame into your DoE analysis. You’ll see clearer signals and faster iteration.