Tim Fox

Cross Posting #Databricks CEO Just Dropped The Most Honest Advice About The Future Of AI The next wave of AI success is going to be completely dependent on structured data. Maybe that’s a no-brainer to you. Maybe I’m telling you that the next time you take a drink of water, that water will be wet. They don’t say things like all software developers will be replaced by AI (the deadline for which has since passed) or all tech employees should become baristas and bartenders (the dream is over, nerds) or don’t worry, we’ve hammered out all the unintended consequences for letting users create their own AI At the same time, AI is, theoretically anyway, offering executives and leaders actionable insights, options, and answers. AI Is Motion, Data Is Experience AI is fire, while data is fuel. https://lnkd.in/gqcRq6vY

2

Excited to be partnering with Databricks and several of their customers as we build out both contract pods and permanent teams of top-tier Databricks professionals. What we’re seeing across the market is clear: The new wave of AI Ops / MLOps engineers is becoming one of the most in‑demand skill sets - experts who can deploy models into live environments, operationalize AI at scale, and fully leverage the power of the Databricks platform. Across multiple projects, this capability gap is growing fast. Are you seeing the same on your side? If AI model deployment, platform operations, or scaling your Databricks talent is becoming a bottleneck for your team or your customer projects, let’s connect. Always happy to share what we’re learning and help where I can. Barney Green Nathan Miller #Databricks #AIOps #MLOps #AIEngineering #DataEngineering

13

5 Comments

We tested Snowflake's Gen2 warehouses on 2 billion query profiles across 14K warehouses here at Capital One. Here is where we saw improvement: 🍎 Approximately 30% of queries fell into categories that could benefit from Gen2—specifically those involving DELETE, UPDATE or MERGE. 🍎 Long-running DML operations: A significant subset of queries running longer than 30 minutes were performing complex DML tasks. These are prime candidates for Gen2 migration. 🍎 Large scans: Numerous queries scanned over 1 TB of data, where Gen 2’s improved scan performance can provide material runtime reduction. 🍎 Combined heavy patterns: A smaller but critical set of queries both scanned over 1 TB and ran for 30 minutes or more, often involving DML operations—these represent the highest potential gain from Gen2. These findings support a targeted migration strategy: focus on warehouses with a significant concentration of eligible workloads rather than migrating all warehouses indiscriminately. Check out the full blow written by our resident expert and all around awesome guy - Ganesh Bharathan https://lnkd.in/g8XufZaV

292

14 Comments

Interesting post here from Trevor.. ‘The future of DE will be to build pipelines in part for AI retrieval … processes for AI not for humans..’ These sentences in particular stood out. Any thoughts? Great post Trevor! #data #recruitment #hiring #management

1

🚀 𝗞𝗶𝗰𝗸𝗶𝗻𝗴 𝗼𝗳𝗳 𝘁𝗵𝗲 𝘆𝗲𝗮𝗿 𝘄𝗶𝘁𝗵 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗺𝗼𝘀𝘁 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗰𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻𝘀 𝗶𝗻 𝗱𝗮𝘁𝗮 & 𝗔𝗜 The 𝘀𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗹𝗮𝘆𝗲𝗿 is no longer a “nice to have” — it’s the backbone of any 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝘁 𝗱𝗮𝘁𝗮 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺 and absolutely foundational for 𝗔𝗜 𝗮𝘁 𝘀𝗰𝗮𝗹𝗲. That’s why this announcement from Snowflake and AtScale really matters. Snowflake continues to lead the market by doubling down on what actually unlocks value: ✔️ Consistent business metrics ✔️ Governed, reusable data models ✔️ Faster, trusted analytics ✔️ And critically — 𝗔𝗜 𝘁𝗵𝗮𝘁 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝘀 𝘁𝗵𝗲 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗰𝗼𝗻𝘁𝗲𝘅𝘁, 𝗻𝗼𝘁 𝗷𝘂𝘀𝘁 𝘁𝗵𝗲 𝗱𝗮𝘁𝗮 If you want AI that reasons, explains, and drives decisions, it needs a 𝘀𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗹𝗮𝘆𝗲𝗿 sitting between raw data and consumption. Full stop. This is how modern organisations move from 𝘥𝘢𝘵𝘢 𝘱𝘭𝘢𝘵𝘧𝘰𝘳𝘮𝘴 to 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝘁 𝗱𝗮𝘁𝗮 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺𝘀 — and it’s exactly the direction the market is heading in 2026 and beyond. Big year ahead. Let’s get started. 💪🤖📊 #DataStrategy #SemanticLayer #Snowflake #EnterpriseAI #IntelligentDataPlatform #Analytics #AIReadiness #DataAsAProduct #ModernDataStack #2026Kickoff #Versent

3

I've spent years building on Databricks and Spark, and the biggest surprise for most teams isn't the performance—it's the bill that arrives after three months of unmonitored ad-hoc queries. I saw a great breakdown from Anurag Sharma on how to use Databricks system tables to actually see where the money is going. He looks at the system.billing and system.lakeflow schemas. For anyone running production workloads at scale, these tables are mandatory. Most teams rely on high-level cloud provider billing, but that doesn't give you the granularity to map DBUs back to specific job IDs or idle clusters. What caught my eye in his architecture was the method for identifying idle compute. We have all seen developers spin up a 16-node cluster for a quick notebook exploration and then leave it running over the weekend. By joining job run timelines with cluster metadata, you can finally quantify that waste in actual currency. If you are implementing this, remember that cloud infrastructure costs for VMs can often rival the DBU spend. Identifying oversized clusters for lightweight ETL is usually the fastest way to get your cloud budget back under control. Are you using these system tables to automate cluster lifecycle management, or are you still relying on manual tags and basic alerts? Source here: https://lnkd.in/ev_q-zFU #DataEngineering #Databricks #CloudOptimization #ApacheSpark #FinOps

61

1 Comment

I’ve seen teams burn $100k a year on Snowflake compute they didn’t realize was running. Don't believe me? $100k isn't even that out of the ordinary. One of my favorite articles, which I read about three years ago now, was about an engineer who accidentally saved 500k by changing a few Snowflake settings. On a similar note, I’ve cut multiple teams data stack costs by 20, 30 and in some cases, 50%....adding up to millions over the last few years. So I can see this happening. Surprisingly, these costs could have often been avoided. Here are the levels of cost saving activties you can take on from easy to hard. Level 1 - Easy Fixes - Warehouse Configuration - If your warehouse default auto-suspended is 10 minutes, set it to 1 - Switch scaling policy to Economy if you can tolerate minor queueing - Right-size your warehouse! If you’re using a medium where a small will do you could easily be spending 2x what you need to spend Level 2 - Medium Fixes - Query Performance - If you’ve got long running queries that are full refreshes, change them to be incremental where possible - Avoid view-on-view-on-views. I see this all the time for dashboards with both gives terrible performance for said dashboard and runs up your bill - Tune your SQL, filter early, select only needed columns - Use cluster keys on large tables to improve pruning. Level 3 - Culture And Tooling - Set up Resource Monitors to alert or auto-suspend warehouses before they blow through your budget - Tag warehouses and queries by team or project, make usage visible and accountable - There are also some tooling changes you might be able to make whether it be switching out your ingestion solution or running your transforms on different compute How have you saved money on your data stack?

113

16 Comments

Stop letting race conditions silently kill your data integrity. 🛑 In high-concurrency applications, relying on luck isn’t a strategy. If you aren't explicitly managing your critical resources, you are just waiting for a collision. I’ve just published a deep dive into Mastering Concurrency with the Symfony 7.4 Lock Component, where I move beyond the basics and share the architectural patterns we use in production. In this article, I cover: 🔐 The "Golden Rule" of Locking: Why the try-finally pattern is non-negotiable for stability.  🏗️ Named Lockers: Strategies for separating critical business locks (Redis) from local tasks (Files).  ✨ Clean Architecture: A "Senior Dev" pattern using PHP 8 Attributes and Event Listeners to decouple locking logic from your controllers entirely. Whether you are handling inventory updates, payment processing, or complex cron jobs, these practices will help you sleep better at night. The full guide and GitHub repo here: 👉 [https://lnkd.in/dWzQ8Et6] 👉 [https://lnkd.in/dhxM4PB5] #Symfony #PHP #Concurrency #SoftwareArchitecture #WebDevelopment #BestPractices #Symfony7

5