How Databricks Query Federation simplifies data access and analytics

#DailyDataDose ☕💁♀️ – Day 61 Imagine this: You have a table in Databricks and another in SQL Server, and you need to join them. Before Join Pushdown: Databricks would pull all the SQL Server data locally to perform the join. Millions of rows crossing the network… slow queries, long waits, frustrated analysts… and higher compute costs due to unnecessary processing. 😩 After Join Pushdown: With #Join_Pushdown_for_Federated_Queries (Databricks Runtime 17.2+), Databricks asks SQL Server to do the join on its side. Only the final, matched results travel back. Why it matters: 🚀 Faster queries – the remote database does the heavy lifting. 🌐 Less network traffic – no more moving millions of rows unnecessarily. 💡 Efficient resource use – Databricks focuses on analytics instead of data transfer. 💰 Lower cost – reduced compute and storage usage in Databricks thanks to lighter data movement. The result? Federated joins are now smarter, faster, lighter, and more cost-efficient, letting your team focus on insights, not waiting for queries.

Do you ever wonder why SQL refuses to die—even with so many new data tools dropping every year? Because no matter what you use—Spark, Snowflake, Databricks, or any data platform—everything eventually comes back to SQL. It’s the language that lets us shape raw data, tune performance, and build pipelines that actually scale. In one project, I even cut dashboard load times in half just by rewriting a few heavy SQL transformations.1. No fancy tricks. Just better SQL. SQL isn’t “basic.” It’s the glue that holds the whole data ecosystem together. Are you using it to its full potential? #DataEngineering #SQL #BigData #Analytics

In a world obsessed with new tools and languages, SQL remains the undisputed foundation of analytics. It is simple, readable, and incredibly powerful. Whether you are querying millions of records in BigQuery or joining tables in Snowflake, SQL empowers analysts to uncover insights quickly. It bridges the gap between technical teams and business users. Every data professional should master it because trends change, but the fundamentals never do. #SQL #DataAnalytics #Snowflake #DataEngineer #DataManagement

Databricks just improved the alert editing. The multi-tab experience allows you to edit and configure alerts more intuitively. You can: - Test alert conditions before deployment - Easily preview custom notification templates with dynamic variables - Choose what happens if a query returns an empty result set ❗️Don't forget to add the alert schedule before saving your alert, by clicking the calendar icon on the top left of the SQL editor. _________ 👋 Hi, I'm Zoë. I share weekly insights on Databricks. #Databricks #Alerts #SQL #UnityCatalog #DataEngineering #DatabricksMVP

Day 3: Snowflake Data Loading & Unloading – Fueling Your Data Pipeline Like a Champ! 📈❄️ What's up, LinkedIn data warriors! 🔥 Day 3 in my SnowPro Core Certification quest, and we're ramping up the momentum. Loved the Virtual Warehouses vibes from Day 2? [Relive it here] if you need a refresher. Today, it's all about Data Loading & Unloading – the gateway to getting your data into (and out of) Snowflake efficiently. No more CSV nightmares; Snowflake makes it buttery smooth! Why This Matters: Fast ingestion means faster insights. Whether you're bulk-loading terabytes or streaming real-time data, Snowflake's got tools to handle it without the drama. Day 3 Focus: Loading & Unloading Deep Dive Pared-down pearls from my session: Loading Options: COPY INTO: The SQL powerhouse for bulk loads from stages (internal/external like S3/Azure Blob). Supports file formats (CSV, JSON, Parquet, Avro) with transformations on-the-fly. Snowpipe: Auto-ingest for continuous loads – triggers on file arrival, perfect for streaming. Pro move: Use file formats for schema-on-read and error handling (e.g., VALIDATION_MODE = RETURN_ERRORS). Unloading Secrets: COPY INTO FROM: Export tables/views to stages in compressed formats. Partition by date/cluster key for massive efficiency. External stages shine for secure sharing – unload to S3 and share via presigned URLs. Best Practices: Stage data first (CREATE STAGE) to decouple loading from source systems. Monitor with QUERY_HISTORY and VALIDATION_ERRORS views – catch issues early. Heads up: Time Travel & Fail-safe ensures your loads are recoverable, but watch storage costs! Hands-On: Loaded a 1GB CSV into a stage via SnowSQL, transformed columns during COPY, and unloaded a query result to GCS. Load time? Under 2 minutes – scalability unlocked! Your insights? What's your loading war story (or hack)? Smash that 💬 below or tag a friend diving into Snowflake. Day 4: Time Travel & Cloning – the undo button for data. Onward! #Snowflake #SnowProCore #DataEngineering #ETL #LearningInPublic

Iceberg Tables - New Features Iceberg tables are very important. Not just for data warehousing (staging layer) but also for data lakehouse. And most importantly, to avoid vendor locking. And for exit strategy. Doesn't matter if you use Snowflake or Databricks or Microsoft Fabric or BigQuery, Iceberg tables are crucial because of the above. New features for Iceberg tables in Snowflake: 1. Automatic clustering 2. Data compaction 3. Manifest compaction 4. Orphan file deletion 1. Automatic Clustering reorganises data within files or partitions based on frequently queried columns. The file size for Iceberg tables is based on your clustering configuration (unless you set a target file size). 2. Data compaction combines small files into larger, more efficient files to manage storage, maintain an optimal file size, and improve query performance. 3. Manifest compaction optimises the metadata layer by reorganizing and combining smaller manifest files, reducing the metadata overhead and improving query performance. 4. Orphan file deletion systematically identifies and removes data and metadata files that exist in the underlying storage but are no longer referenced by any valid table snapshot. For details see: https://lnkd.in/e7Vr3d3K Using Snowflake for data lakehouse: https://lnkd.in/esy5wEq7 Release notes: https://lnkd.in/e64qrCNb Keep learning! My Linkedin articles: https://lnkd.in/eRTNN6GP. My blog: https://lnkd.in/eDdTNzzW #Snowflake #Iceberg #Databricks #MicrosoftFabric #BigQuery

💡 The Most Valuable Skill I Use Every Day as a Data Engineer It’s not Spark. It’s not Airflow. It’s SQL. No matter how advanced your stack is BigQuery, Snowflake, or Databricks, it all comes down to how well you can query, optimize, and explain data. Over the years, I’ve learned: ✅ Clean SQL > Complex SQL ✅ A single well-tuned query can save hours of compute ✅ Mastering SQL builds confidence in every layer of the pipeline SQL isn’t old-school, it’s the foundation of every great data system. #SQL #DataEngineering #BigQuery #ETL #CloudComputing #Analytics #CareerGrowth

Imagine you spent 10x what you did on your data warehouse ("lakehouse", whatever) by using Databricks, and wanted to move to BigQuery but wanted to do it quickly cause they were bleeding you dry. Gemini can now re-write your Databricks Spark SQl to BigQuery SQL. 🎉

Dataform for BigQuery: a match made in data heaven. Prasanna Venkatesan’s recent blog is a no-nonsense, end-to-end guide to getting started. From setting up your workspace to building transformations and optimising with partitioning and clustering, it’s all there. Check it out 👉 https://lnkd.in/eXgD8Q2V ----- #Dataform #BigQuery #DataEngineering

🐢 Slow queries aren’t always about hardware — sometimes, it’s about understanding 🔥 𝙃𝙤𝙬 𝘿𝙖𝙩𝙖𝙗𝙧𝙞𝙘𝙠𝙨 𝙎𝙌𝙇 𝙒𝙖𝙧𝙚𝙝𝙤𝙪𝙨𝙚𝙨 '𝙨𝙘𝙖𝙡𝙚' 𝙪𝙣𝙙𝙚𝙧 𝙡𝙤𝙖𝙙. Here’s another interesting practice exam question that dives into 𝗦𝗤𝗟 𝗪𝗮𝗿𝗲𝗵𝗼𝘂𝘀𝗲 𝗶𝗻𝘁𝗲𝗿𝗻𝗮𝗹𝘀 — from query concurrency and scaling ranges to how the 𝗶𝗻𝘁𝗲𝗿𝗻𝗮𝗹 𝗹𝗼𝗮𝗱 𝗯𝗮𝗹𝗮𝗻𝗰𝗲𝗿 𝗮𝗻𝗱 𝗮𝘂𝘁𝗼𝘀𝗰𝗮𝗹𝗲𝗿 handle queued workloads. 𝗤𝗨𝗘𝗦𝗧𝗜𝗢𝗡: You have noticed that Databricks SQL queries are running slow. You are asked to look reason why queries are running slow and identify steps to improve the performance. When you looked at the issue you noticed all the queries are running in parallel and using a SQL endpoint [ SQL Warehouse ] with a single cluster. Which of the following steps can be taken to improve the performance/response times of the queries? ⚠️ [ Option 1 ] They can turn on the Serverless feature for the SQL endpoint [ SQL warehouse ]. ✅ [ Option 2 ] They can increase the maximum bound of the SQL endpoint (SQL warehouse)’s scaling range. ⚠️ [ Option 3 ] They can increase the warehouse size from 2X-Small to 4XLarge of the SQL endpoint [ SQL warehouse ]. ❌ [ Option 4 ] They can turn on the Auto Stop feature for the SQL endpoint [ SQL warehouse ]. ❌ [ Option 5 ] They can turn on the Serverless feature for the SQL endpoint [ SQL warehouse ] and change the Spot Instance Policy to “Reliability Optimized.” This question is a perfect way to test: 1️⃣ How 𝗦𝗰𝗮𝗹𝗶𝗻𝗴-𝗨𝗽 𝘃𝘀 𝗦𝗰𝗮𝗹𝗶𝗻𝗴-𝗢𝘂𝘁 actually works inside Databricks SQL Warehouses. 2️⃣ The role of the 𝗶𝗻𝘁𝗲𝗿𝗻𝗮𝗹 𝗾𝘂𝗲𝘂𝗲, 𝗹𝗼𝗮𝗱 𝗯𝗮𝗹𝗮𝗻𝗰𝗲𝗿, 𝗮𝗻𝗱 𝗮𝘂𝘁𝗼𝘀𝗰𝗮𝗹𝗲𝗿 in distributing query load efficiently. I’ve broken this down in an 11-minute explainer video — where I walk through the 𝘦𝘹𝘢𝘤𝘵 𝘧𝘭𝘰𝘸 𝘰𝘧 𝘩𝘰𝘸 𝘲𝘶𝘦𝘳𝘪𝘦𝘴 𝘨𝘦𝘵 𝘥𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘢𝘤𝘳𝘰𝘴𝘴 𝘤𝘭𝘶𝘴𝘵𝘦𝘳𝘴 and 𝘸𝘩𝘺 𝘩𝘰𝘳𝘪𝘻𝘰𝘯𝘵𝘢𝘭 𝘴𝘤𝘢𝘭𝘪𝘯𝘨 (𝘯𝘰𝘵 𝘷𝘦𝘳𝘵𝘪𝘤𝘢𝘭) 𝘧𝘪𝘹𝘦𝘴 𝘵𝘩𝘦 𝘣𝘰𝘵𝘵𝘭𝘦𝘯𝘦𝘤𝘬 𝘩𝘦𝘳𝘦. 💬 If you’re also fascinated by the architecture behind Databricks SQL Warehouses, I’d love to hear your take on this. #Databricks #DataEngineering #SQLWarehouse #Scalability #PerformanceOptimization #Lakehouse #DataArchitecture #SystemDesign