James Serra

New data infrastructure lore just dropped: The Pendantic Layer. Joe Reis has a great article introducing this idea [https://lnkd.in/gaXw_TPr], but this quote sums it up perfectly. "The Pedantic Layer slows us down by focusing attention on taxonomy and semantics over outcomes. It makes practitioners feel like they need to pass some purity test of definitions before they can get value. It feeds a culture where thought leaders posture about whether CSVs are structured “enough,” while the practitioners who ignore the debate are quietly building models that drive millions in revenue." Coming from startups where you are obsessed with execution over perfection, the "pedantic layer" has been one of my largest gripes with the wider data industry. In celebration of yet another made-up layer (YAML), leave a comment about your favorite data concept, and I'll tell you why you are wrong for obscure nuances that don't apply to most use cases.

45

11 Comments

You cloud migration failed becuse you didn't build a 'modern data warehouse.' You built the world's most expensive junkyard. Your migration didn’t fail because of Snowflake or Databricks. It failed because you tried to move too early, too fast, or without proving value. You picked the wrong timing, treated it like IT initaitve and treated as a goal Seen this all the time The CIO announces a “cloud-first data strategy.” $2M+ budget approved. Consultants everywhere. 18-month migration roadmap. Two years later, Finance is still in Excel. Operations still doesn’t trust dashboards. The business still runs on exports and emails. 200-redunded pipelines running into cloud (10% actually used) Over 1TB of usless data that shouldn't be there in first place. The stack changed. But you just moved shit data, to the cloud. Here’s what most teams get wrong (and how to fix it): 1. You moved everything at once. You don’t need to migrate 100% of your data. You need to migrate the 20% that drives 80% of business decisions. You gone see that 80% of parts you don't need, and are usless. Start Fix: Start with one high-value use case, financial close, revenue cycle, or ops reporting. Prove value before moving anything else. 2. You treated the migration like a sprint, not a relay. You can’t “big-bang” your way to trust. Most migrations lose sponsorship halfway because they take too long to show results. Most migrate data but nobody They fund impact. Fix: Deliver one visible business impact every 45-90 days. Show speed-to-value, think in terms of data products focuson on trust, adoption and ROI. 3. You sold the end state, not the next milestone. Your roadmap said: “100% cloud by 2026.” Terrible strategy. No one funds a vision they won’t live to see. Cloud isn’t the destination or outcome, the only thing it matters what you do with data. If your first conversation starts with tools or roadmap instead of business problems, it’s already dead. Fix: Reframe the migration as a sequence of self-contained 90-day wins. Each one should earn credibility and fund the next phase. 4. You forgot that migration ≠ adoption. The new data warehouse is live, but if business users still export to Excel, OR don't trust your data is usless. Fix: Make adoption part of your migration plan. Enable self-service by standardize foundation, but keeping outside agile. Celebrate business wins, not engineering milestones. 5. Avoid Over-engineering. Don't add tools, until you have trust, adoption and semantic layer completed. Organization end up with 10 tools, real time pipelines but can't trust their data. Don’t wait for perfection Moving to cloud doesn't make you data driven or AI readiness Focus on impact use cases, speed, adoption and creating foundation that is trusted. Most just move bad usless redundat data, to cloud Comment below, "Cloud Migration" I’ll send you our playbook that gets you adopted, and ROI in weeks.

101

17 Comments

I was reading about "Metadata-based Pruning Techniques" for large data systems & stumbled upon this new ACM SIGMOD PODS 2025 paper by Snowflake. Overall, a good read and highlights how "metadata" is a central pillar for faster queries. Let me explain 👇 Pruning techniques are key to query optimization! One of the best ways to speed up data queries is to avoid reading unnecessary data in the first place. Traditional pruning skips data based on filters (say min/max values). For example, “don’t scan rows outside a date range.” But queries with LIMIT, ORDER BY … LIMIT k (top-k), or JOIN clauses usually still scan a lot of data because you don’t know up front which rows will qualify after sorting or joining. In this paper, Snowflake explains how they’ve taken data pruning beyond the usual filter-based techniques. Specifically to: - LIMIT queries (which only need a small subset of rows) - top-k queries (like “show me the top 10 customers”) - joins (which traditionally require scanning a lot of data) By extending pruning to these advanced patterns, Snowflake dramatically reduces the amount of data scanned & transferred, cutting both cost and query latency. They actually show that with smart metadata & aggressive pruning strategies, you can skip over 99% of irrelevant data. The paper is called “Working Smarter, Not Harder”: proactively skipping work through intelligent pruning, instead of throwing more compute at the problem. And like I said, a key enabler here is metadata! Snowflake uses detailed min/max stats, zone maps, Iceberg/Parquet metadata to know exactly which micro-partitions to scan and which to ignore. Some takeaways that I noted: ✅ They go beyond traditional filters, skipping unnecessary data even for LIMIT, top-k, or join queries - saves huge in processing. ✅ It can recognize (via metadata) when a partition fully matches the WHERE clause, and contains more rows than the LIMIT, you can scan just that partition and stop, skipping the rest. ✅ Algorithms borrowed from search engines help the engine prioritize the most promising data first - metadata gives you the ability to rank, filter, and prune partitions early. ✅ Joins are optimized by summarizing values (via metadata) on one side, then pruning irrelevant partitions on the other - reducing I/O with redundant scanning. Metadata-driven pruning is also at the heart of modern lakehouse systems. Formats like Apache Hudi not only offers file and column-level information for pruning, but also power advanced indexing with their dedicated metadata table. Indexes like - Bloom Filters, Record-level index, Expression & Secondary indexes are built as partitions inside the metadata table, which significantly improves the efficiency of both writing to and reading from the table. Ultimately, rich metadata is what makes powerful pruning and optimization techniques possible in complex systems. Link in comments. #dataengineering #softwareengineering

88

1 Comment

Claude, ChatGPT, Snowflake Intelligence, Cortex Analyst, Databricks Genie, Zenlytic, TextQL. Think about any AI agent you connect to your data. They all get questions from business users like: “What was the average number of daily active users in the last quarter?” These AI agent connects to Euno’s MCP server to retrieve metadata. That’s when Euno gives these agents the missing context they need to understand your data. ** Just a reminder that Euno is a metadata platform so we only access your metadata ** Euno runs a semantic search and looks for dashboards that use DAU (which is semantically equivalent to Daily Active Users). It filters for certified dashboards using the trust signals you’ve already set in Euno. In this example Euno finds two candidates: 1. DAU used in the User Engagement dashboards 2. DAU_NEW used in the Product Statistics dashboard Both dashboards are certified. But Euno also shows how often each has been used in the past 30 days, along with the owner or creator. With that information, the business user can confidently choose the right one and the AI agent can deliver the correct answer. The same applies to tools like Cursor. Let’s say you’re an engineer and ask: “Can you find me the Tableau dashboard for Customer Success?” Cursor will reach out to Euno’s AI assistant to get that BI context because Cursor is focused on dbt development and doesn’t have access to that layer. You can even go further and ask which dbt models power that dashboard and how those models are calculated. Euno feeds Cursor with that real-time context as you build. It’s not just for business users. It’s for developers too.

25

5 Comments

Is Snowflake Cortex Code (Coco) just another coding assistant? Not even close. Calling it a "coding agent" severaly undersells what it can do. It is like giving every user an elite team of Snowflake engineers right at their fingertips offering 100x Productivity. When you provide an LLM with the full picture of your environment, you eliminate the "guesswork" cycles. Coco generates working code faster, uses fewer tokens, and allows users to tackle complex architecture tasks that used to require a massive team of experts. Offical Info PAge https://lnkd.in/eWWjJ89F WHY NOT USE OTHER BIG BRAND CODING AGENTS? Typical agents are great at generating code, but they often operate in a vacuum. They lack awareness of your specific data, security protocols, or platform features & limitations. This leads to broken code, "hallucinated" table names, and endless manual troubleshooting. Resulting in projects that are almost working but not 100% all while using far too many tokens & $$ constantly reiterating the code to fix unseen issues due to operating blindly. WHY COCO IS LITERALLY A GAME CHANGER IN THE DATA & AI SPACE? Coco is different because it combines leading code LLMs with deep Account-Level Context and Specialized Skills built directly by the Snowflake team. - Full Account Context: Coco has the "keys to the kingdom." It can discover data, view & work with any account level objects such as clusters, network & security settings, RBAC, integration configurations & etc. It knows exactly where your code executes and what it has access to and can fix deployment or code related issues if needed. - Specialized Expertise: It isn't just a generalist. It possesses baked-in skills for #DataEngineering, #MachineLearning, #AI, #App development and others developed by Snowflake engineers to come up with the right solution using the right code much faster and with far less manual help from a user. - Operational Intelligence: Beyond just writing lines of code, Coco acts as an administrator, security expert and performance consultant. It can manage costs, optimize tasks, and build entire end-to-end projects within your account

450

8 Comments

The hype around new data capabilities often ignores the reality of managing costs at scale. I saw this post from Anurag Sharma highlighting a common pain point: the Databricks bill that starts looking like a phone number because engineers are spinning up clusters just to debug basic transformations. The solution is a return to local development fundamentals. By using a local Spark setup in IDEs like IntelliJ or PyCharm combined with a strict Reader-Transformer-Writer template, his team moved 75% of development work off the cloud. What caught my eye in the architecture: The Reader forces early filtering. If you are not predicate pushing or filtering at the source, you are just wasting memory and IO regardless of your cluster size. The Transformer uses the Spark transform function for modularity. This makes unit testing business logic significantly easier than dealing with massive, monolithic notebooks. The Writer then enforces columnar formats like Parquet or Delta to ensure downstream performance. From an architectural standpoint, this reduces the feedback loop. Waiting five minutes for a cluster to spin up just to find a syntax error is a productivity killer. Moving the heavy lifting to local environments and only using the cloud for final integration testing via the Databricks CLI is a win for both cost and developer experience. If you are running large Spark workloads, how are you balancing local development vs. cloud-native features? Do you find local Spark environments drift too much from production cluster configurations? Source here: https://lnkd.in/eAM2J66m #DataEngineering #ApacheSpark #Databricks #CloudArchitecture #CostOptimization

4

A potentially silly question for my SQL family. Have you encountered issues with raising errors in a transaction when the error occurs in executed dynamic SQL? Explanation, I was trying to force an error to test some error logging issues, but I was not getting an error in SSMS as long as it was inside of a transaction. The dynamic SQL in question was calling a bulk insert to a global temp table for staging purposes. I purposely set the file path to fail so I could test error handling. When I ran the same code outside of a transaction, it failed as expected. Example: BEGIN TRY BEGIN TRANSACTION DECLARE @sql VARCHAR(500) SET @sql = 'Imagine some great dynamic sql here' EXEC (@sql) COMMIT TRANSACTION END TRY BEGIN CATCH [Insert Error message and timestamp to a user defined table] ROLLBACK TRANSACTION END CATCH It executed with no results and no error. I printed the SQL query, and it validated the output. It was building the string as expected. When I attempted to execute the pasted string, it failed as expected. So, I took everything out of the transaction and re-ran the query using the dynamic SQL. It failed! I have never seen this type of behavior before with my dynamic SQL. I tested it in SQL Server 2022 and SQL Server 2022 with compatibility level set to 2012. (To mirror one of the working environments) I considered the fact that it's writing to a global temp table and not a native table, but the file path doesn't exist either way. I would have expected an error. I'm open to ideas, comments and criticisms. This is an interesting issue.

1

3 Comments

*** Ontology + DDD: The Semantic Superpower Behind Modern Solutions Architecture *** Most architecture problems today aren’t technical—they’re semantic. Systems break because different teams (or apps) mean different things when they say Customer, Order, or Account. This is where ontology becomes a game-changer. What’s Ontology? It’s a formal, shared model of the key concepts in a domain, how they relate, and the rules that govern them. Example: Customer, Order, Product Customer places Order Order must contain ≥1 Product Think of ontology as the semantic blueprint that humans and machines can understand. --- Why Ontology Matters for Solutions Architecture 🔹 Creates a shared vocabulary across teams 🔹 Defines cleaner domain & service boundaries 🔹 Enables semantic interoperability across legacy + modern systems 🔹 Supports automated governance (“SensitiveData → requires encryption”) 🔹 Powers knowledge graphs for impact analysis 🔹 Provides the semantic backbone for AI-driven architecture tools In short: ontology brings consistency, reasoning, and clarity to complex ecosystems. --- But What About Domain-Driven Design (DDD)? DDD is fantastic—but it solves a different layer. What DDD excels at: ✔ Modeling domains within a bounded context ✔ Establishing clear service boundaries ✔ Creating ubiquitous language for one team or system ✔ Structuring microservices and integrations Where DDD needs help: DDD doesn’t provide: ❌ enterprise-wide semantic consistency ❌ machine-interpretable meaning ❌ compliance rule automation ❌ knowledge graphs across the whole architecture --- The Most Powerful Approach Today: **Use DDD to model the domain. Use Ontology to unify the enterprise.** DDD discovers the language. Ontology formalizes and connects it across systems, teams, and AI tools. Together, they give you architectures that scale—not just technically, but semantically.

3