Max Schmeiser

Ever feel like today’s AI models are getting smarter but somehow less useful? You’re not imagining it. LLM benchmarks have become the obsession. Model makers are locked in a cycle of chasing leaderboard points on MMLU, GSM8K, and other tests no one actually cares about outside of research labs. The result? Models that ace exams but fall apart in real-world use. They misinterpret vague instructions. They forget what you said three messages ago. They can’t think beyond a single step. But sure, the score went up by 0.3 percent. Benchmarks were supposed to be a guide. Instead, they’ve become a crutch. A way to show “progress” without improving anything that matters. The questions we should be asking are simple: Can the model follow messy, human instructions? Can it reason through problems, not just regurgitate answers? Can it actually help people get real work done? If you’ve worked with these models, you’ve felt the gap. You’ve seen the hype, and then you’ve seen it fall short when it counts. Are we building for people, or are we building for points? I’d love to hear what others are seeing in the wild. #AI #LLMs #Benchmarks #MachineLearning #RealWorldAI

27

3 Comments

We're facing more and more questions about the use of synthetic data in market research. I'd recommend this blog post, which really resonated with me. So much of the conversation is "how closely can this replicate human data?" But there are so many other considerations researchers should be thinking about. https://lnkd.in/eSQfSTPj

31

3 Comments

How do you know if your new A/B testing methodology actually works? Researchers continue to propose new ways to analyze A/B tests: new estimators of impacts, new rules for deciding whether to launch. But it's difficult to tell whether these methodologies actually add value because the true impact of any A/B test is unobserved. When they give us different results, are they actually better --- or just different? Traditionally, we'd turn to simulations. But this requires strong assumptions about the data generating process that are unlikely to hold in practice. My new paper (with James McQueen and Miikka Rokkanen) attempts to solve this problem via sample splitting. The idea is to partition each test into two independent splits: one to proxy for the true impact and another to apply the proposed methodology. We show exactly what this approach can and cannot measure, how to conduct statistical inference, and the tradeoffs involved in implementation. We hope this will help practitioners rigorously evaluate whether new methodologies are actually adding value in their A/B testing contexts. Any and all feedback is welcome! https://lnkd.in/g5ZvCbkv

272

6 Comments

There are very few things that excite me more than applying my experience in new ways. The Master of Analytics program in NC State University allows me to do just that, applying lessons I've learned in intelligence research to data science. While there are endless topics in #machine_learning I am eager to write about, one topic that felt especially close to heart was how language is shaping our decision-making. In "Why Lingo Matters in Analytics", I chose to tackle one specific aspect of this: how our framing of the data must be as accurate and formal as our actual data. Our framing must be accurate and formal, because all throughout history people were hearing and reading ideas via *words*, not numbers; and thus we evolved to make decisions based on *words* and not based on numbers. When a data analyst/scientist makes a recommendation to a stakeholder, the actionable part is always in the words, never in the numbers. The numbers are there only to show rigor and make our claims verifiable. To quantify. To answer the "how much", but never the "what" or "why". The true benefit of AI is allowing us to reach conclusions much faster (and make funny animal videos faster too, I guess). But to reach these conclusions safely, and make the right decisions, we need the right framing to judge the numbers by. Have a fun read!

9

AstroBee generates ontologies...but what is an ontology, and if they're important, why don't you already have one? I've written a short piece on Kimball star schemas, ontologies, and how each relate to Bill Inmon's original approach to data warehousing in the early 90s. Dig in to find an overview of a fascinating technological debate and the re-emergence of an approach to data modeling that's been sleeping for 20 years.

18

3 Comments

🚀 Just wrapped up a hands-on project applying Retrieval-Augmented Generation (RAG) to a real-world use case: interpreting LIC’s New Jeevan Shanti policy document using LLMs. The core idea? Combine vector search over regulatory PDFs with GPT-4 to generate grounded, faithful answers to complex policy questions. Key capabilities we built: ✅ LCEL-based RAG architecture using LangChain ✅ Contextual compression retriever for focused document lookup ✅ Modular LLM & embedding selection (OpenAI, HuggingFace, Mistral) ✅ Evaluation using LLM-powered metrics: faithfulness, relevance, correctness 📄 The document used is publicly available here: LIC New Jeevan Shanti Policy PDF 💻 Full code available on GitHub: https://lnkd.in/gS9mEzWA 💡 This architecture brings the best of both worlds — retrieval accuracy and generative flexibility — especially in regulated domains like insurance and financial services. 🔜 Next step: Incorporate Agents that can reason across multiple steps — not just answer a question, but decide which tools to use, when to search, how to validate, and when to stop. That’s where the future of enterprise GenAI lies. Would love to hear how others are pushing RAG, agents, and LLMs in production! #GenAI #LangChain #LLM #RAG #AIForOps #InsuranceTech #LLMEvaluation #LangChainAgents

61

1 Comment

A recent Amazon paper on Insight Agents is an example of the agentic AI patterns and agentic analytics systems I’ve been writing about recently. The system follows an orchestrator–worker pattern: a manager agent handles intent detection, routing, and control; specialized agents perform data retrieval and insight generation; and LLMs are used primarily for reasoning and synthesis, not execution. Analytics is treated as a plan-and-execute workflow, with governance built directly into the architecture through out-of-domain detection, API-based data access, and explicit control flow. Evaluated via human review, the system achieves ~90% accuracy with interactive latency (~14s P90)—showing that agentic analytics can be both reliable and operational at scale when orchestration and constraints are first-class design decisions. Paper: https://lnkd.in/eE_tDkGc #AgenticAI #AgenticAnalytics #AIArchitecture

11

1 Comment

A first-of-its-kind, cross-institutional effort to define a standardized AI evaluation reporting framework that enables rapid benchmark review and easier data reuse.

7

Towards Data Science recently published an article I wrote on covariate balance in experimentation. In it, I unpack the often‑confused distinction between balance and independence. https://lnkd.in/gD_4GZbJ

48

1 Comment