Gary Rector

I asked o3 (deep research) to create a statistical scenario that it doesn't know how to answer. Perhaps one can use this system prompt to see how models evolve similar to "draw a unicorn using latex". TLDR: it picked causal inference. https://lnkd.in/dPiGSC2g

90

6 Comments

RIP (some) reliable data sets and variables. As data scientists, we rely on consistent, transparent public datasets. The recent removal and alteration of federal data in the USA is a reminder of how vulnerable evidence-based analysis can be when data access changes. A useful reference documenting these losses: https://lnkd.in/ecKRpaJS When the fundamental data changes, our models, comparisons and decisions are disrupted. This isn’t just a technical issue—it affects how we understand our society. #DataScience #OpenData #PublicPolicy

18

1 Comment

I had the pleasure of interviewing Bryan H. Kelly for the Dongmicus Blog — and now, he’s been featured in CDO Magazine as a thought leader in data literacy. Bryan is the author of Cass Freeman: Decision Trees, a compelling novel that fuses STEM education, ethics, and storytelling to inspire data leaders toward inclusive and accountable decision-making. It’s exciting to see his insights gaining global recognition. You can read my original interview with him on the Dongmicus Blog — link in the comments!

24

3 Comments

Over the holiday break, I spent some time revisiting some of the foundational work shaping the field of scientific AI. One paper that stands out—published back in 2021—was co-authored by Damien Guého, now a Senior R&D Engineer at Geminus. The paper tackles a timeless challenge: how do you uncover the true physics of a system when all you have is sparse, noisy, imperfect data? Damien and his co-author's approach is both clever and elegant. Instead of a brute force deep learning strategy, they leveraged clever math to uncover a physics-grounded structure with interpretable dynamics—from messy, nonlinear signals. The result is a robust, reliable and generalizable solution that uses a fraction of the computational effort. They first solved the problem using a deep learning approach that required 7,720 parameters. They then applied their method that used just 3 terms and was 10,000× more accurate. This mindset is core to Geminus' mission to simplify engineering and operational decision making in complex systems where uncertainty abounds and data is sparse and messy. Work like that of Damien and his colleagues continues to inspire where we’re headed next. Proud of the team at Geminus Labs—and excited for what’s coming. Check out his paper here. https://lnkd.in/eHt-5tXX Watch this space - be sure to follow Geminus to be part of the conversation.

32

Moving from "Raw Data" to "Reliable Data": Why the Bronze-to-Silver transition is the hardest part of the Medallion Architecture. My last post about reducing latency to 15 minutes sparked some great conversations about the Medallion architecture. While "Bronze/Silver/Gold" sounds simple on paper, the transition from Bronze (Raw) to Silver (Refined) is where most pipelines break. In our 2TB/day environment, we’re currently rethinking this layer to solve three specific challenges: Schema Evolution vs. Schema Enforcement: In Bronze, we aim for "Schema-on-Read" to prevent data loss. But by Silver, we need strict enforcement. How do you handle a source system changing a FLOAT to a STRING without killing the downstream Silver table? The "Compaction" Problem: Small files are the silent killer of performance. We’re implementing auto-compaction and Z-ordering at the Silver layer to ensure that 15-minute incremental loads don’t result in 15-minute query times for our analysts. Audit Metadata: We’ve started injecting _ingested_at and _source_file metadata into every Silver record. It sounds like overhead until you’re debugging a data drift issue from three weeks ago. For those running Spark or Flink at scale, do you prefer a "Flat" Silver layer (one-to-one with source) or a "Modeled" Silver layer (normalized/cleaned)? I’m leaning towards a hybrid approach, but would love to hear your trade-offs. #DataEngineering #Lakehouse #DeltaLake #BigData #DataScience #AWSDataEngineer #AIEngineer #UAE #Saudi

3

Very, very cool book. I'm going to pick up a copy of this. This looks great. Congrats to the authors (Mark Freeman II, Chad Sanderson, B.E. Schmidt) on their book! https://a.co/d/0331MreX From the description: "Poor data quality can cause major problems for data teams, from breaking revenue-generating data pipelines to losing the trust of data consumers. Despite the importance of data quality, many data teams still struggle to avoid these issues—especially when their data is sourced from upstream workflows outside of their control. The solution: data contracts. Data contracts enable high-quality, well-governed data assets by documenting expectations of the data, establishing ownership of data assets, and then automatically enforcing these constraints within the CI/CD workflow. This practical book introduces data contract architecture with a clear definition of data contracts, explains why the data industry needs them, and shares real-world use cases of data contracts in production. In addition, you'll learn how to implement components of the data contract architecture and understand how they're used in the data lifecycle. Finally, you'll build a case for implementing data contracts in your organization." I love Data Operations. Nice to see this book!

54

2 Comments

One of the most common concerns I hear from data leaders who are looking to go down the graph path is the fear that this will necessitate *yet another* massive lift & shift operation. The answer is no, and I explain the reason here: https://lnkd.in/gSnSWpcu Central to this question is: what kinds of data go into the graph, what is the "minimum viable graph"), and can I grow my graph over time. Foreshadowing the answers: you don't need everything, and yes graph databases--the native ones, i.e. not backed by an RDBMS-- are literally designed for organic expansion and yielding network effects.

149

10 Comments

You don't need a semantic layer. Every vendor in the "modern data stack" keeps saying you do... but they're just wrong. The more you interact with AI agents and data, the more you realize semantic layers don't make sense. This is one of the best write-ups I've seen on the subject. Very few people actually understand this. It's abundantly clear that Ramp, Jay, Ian, and team are way ahead of the curve. https://lnkd.in/gQ_SQfiF

25

7 Comments

🎓 This is a bit of an esoteric post. A fellow #Tableau developer asked me how they could use a line graph of Measure Values and yet have a YOY calc for each value on the Label. That was a bit of a stumper because if you add a value to the Measure Values filter, it adds a line. If you add it to the label, you add the YOY measure to every other line/Measure that you are graphing. One solution I thought of was to use Map layers. That would make this possible with a layer for reach Measure Name/Value and its corresponding YOY calc. After sleeping on it, I came up with another solution that uses what I consider to be an underutilized data modeling solution - doubling (or more) of your data source with a relationship to a tiny multiplier table. Here are the steps: 1. Create a "multiplier" spreadsheet with a row for each desired Measure Name (in this case, Sales, Profit, and Quantity) 2. Add a relationship to it in your data source. Set the condition to a calc of TRUE on each data item (Orders and the Multiplier). This is frequently done by 1=1, which I find less intuitive.... 3. Set up a calcs that use the multiplier to return a different value for each row of that multiplier. 4. The dimension from your Multiplier data source then divides the sheet by color/line. 5. Corresponding calcs for YOY and for numeric format round out the requirements. 🔗 Dashboard Link in comments. With this type of approach, just a little more data allows you to double or triple your data source. Actually, you should never have to literally duplicate your data source. Maybe there is a solution to the stated problem that I'm not thinking of. If so, please tell me in the comments. I'll tag Kirk Munroe and Eric Parker, both of whom may find this interesting. #data #analytics #datafam

10

2 Comments