Databricks just improved the alert editing. The multi-tab experience allows you to edit and configure alerts more intuitively. You can: - Test alert conditions before deployment - Easily preview custom notification templates with dynamic variables - Choose what happens if a query returns an empty result set ❗️Don't forget to add the alert schedule before saving your alert, by clicking the calendar icon on the top left of the SQL editor. _________ 👋 Hi, I'm Zoë. I share weekly insights on Databricks. #Databricks #Alerts #SQL #UnityCatalog #DataEngineering #DatabricksMVP

🗺️ Data Engineering Tools in One Map Following up on our Donut Factory Analogy (check the last post if you missed it), let’s dive into the actual tools that power each layer of the modern data stack. Yes, the list can feel overwhelming but don’t panic! 🚀 Start by mastering the fundamentals: Spark ⚡ for large-scale data processing SQL 🗂️ the universal language of data Delta/Iceberg 🧊 for modern data lakehouse table formats Postgres 🐘 a solid foundation for databases & analytics These are the core skills. And since platforms like Databricks and Microsoft Fabric are Spark-based, mastering Spark gives you an immediate advantage. 👉 Use this visual as your roadmap: once you’re comfortable with the basics, pick one skill from each section to progress further. 🔜 What’s next? We’ll kick off an ETL/ELT deep dive starting with Spark-based ingestion and transformation use cases, then moving into other tools. #DataEngineering #BigData #Spark #SQL #Databricks #MicrosoftFabric #DeltaLake #ApacheIceberg #ETL #ELT #DataPipeline #DataTools #DataAnalytics #Postgres #CloudComputing

Snack time 🍫 𝗠𝗮𝗸𝗲 𝗦𝗤𝗟 𝗿𝗲𝗮𝗱 𝗹𝗶𝗸𝗲 𝘁𝗵𝗲 𝘄𝗮𝘆 𝗶𝘁 𝗿𝘂𝗻𝘀 Databricks now supports 𝗽𝗶𝗽𝗲 𝘀𝘆𝗻𝘁𝗮𝘅 (|>) so queries can be written in natural execution order, chaining steps like FROM, WHERE, and AGGREGATE for cleaner, more readable SQL. It’s fully backward compatible, simplifies filtering and aggregations on a single line. Curious how it looks in practice? Check the datasnack for a 𝘀𝗶𝗱𝗲-𝗯𝘆-𝘀𝗶𝗱𝗲 example! 👀 #DataSense #DataSnacks #Databricks #PipeSyntax

Rescued Data Column It automatically captures any unexpected or extra fields that don’t match the defined schema while reading data (especially from JSON or CSV files). Example: When a JSON file has new or unknown keys, instead of failing, Databricks saves those fields inside a column named _rescued_data_ — preventing data loss and supporting schema evolution. #Databricks #DataEngineering #BigData #DeltaLake

📢 Any SQL developers working with dbt on Snowflake here? Now you can define semantic views in dbt! 💡 The new dbt_semantic_view package lets you define and manage the full structure of your Snowflake semantic views (facts, dimensions, relationships, metrics) via dbt and automate their creation, renaming, and dropping. ❓ Out of curiosity: 1) How many of you use (or plan to use) dbt with Snowflake? 2) How many of you already work (or plan to work) with semantic views in Snowflake? ➡️ Read the official announcement from Snowflake: https://lnkd.in/dksscBJm #Snowflake #dbt #SemanticViews #SemanticModeling #SQL

Recursive CTEs are fully supported in Databricks SQL, enabling iterative and hierarchical data queries directly in SQL. Databricks SQL implements recursive CTEs using the ANSI SQL standard, with key syntax and features that make them powerful for data engineering and analytics tasks involving trees or graphs. Full video: https://lnkd.in/eVYV_bRz Holly Smith Kent Marten Fabien Contaminard #Databricks #NextGenlakehouse

One Lake, Two Superpowers: Exploring Databricks + Snowflake on Iceberg vs. Oracle Autonomous Database for Data Lakes. I recently put together some thoughts on a challenge many data teams face: whether to run Databricks and Snowflake together on a shared Apache Iceberg lake, or simplify by using Oracle Autonomous Database as a unified solution. This article walks through how these approaches compare from real-world operational details to total cost of ownership, based on both 100 TB and 1 PB scenarios. I hope it’s a helpful resource if your team is navigating this decision or simply interested in the evolving data lakehouse landscape. #lakehouse #autonomousdatabase Here’s the full article:

Working in Databricks? Here’s how to write SQL that runs faster and stays clear: SQL actually runs in this order: FROM → WHERE → GROUP BY → HAVING → SELECT → ORDER BY → LIMIT Follow these simple tips: 1) Filter data early with WHERE to scan less. 2) Break your SQL into small pieces using WITH (CTEs) so it’s easy to read. 3) Cache heavy steps (CACHE TABLE or Delta Cache) so you don’t recompute them. 4) Keep rules in one spot to stop bugs from spreading. It’s not just about the final SELECT. Shape your data at the right step to speed up Spark jobs and make your notebooks easy to maintain. #DataEngineering #Databricks #SparkSQL #Performance

🚀 Exploring Query Federation with Databricks! 🔍 As data professionals, we often face the challenge of accessing and analyzing data spread across multiple systems. With Databricks Query Federation, that challenge becomes a lot easier to tackle. Recently, I explored how Databricks enables remote queries across external data sources like MySQL, PostgreSQL, Snowflake, and more—without the need to move data around. This capability not only simplifies data access but also enhances performance and governance. 💡 Key benefits: ⏺️ Seamless integration with external databases ⏺️ Unified analytics across diverse data sources ⏺️ Reduced data movement and duplication ⏺️ Improved data governance and security Whether you're building dashboards, running complex analytics, or powering ML models, Query Federation can be a game-changer. 🔗 Learn more: https://lnkd.in/gY2TzMY3 #Databricks #QueryFederation #DataEngineering #BusinessIntelligence #RemoteQueries #DataAnalytics

🚀 Triggered Tasks in Snowflake: Automate Without Polling As data engineers, we often face the dilemma: poll sources too frequently and waste compute, or respond too slowly and compromise freshness. Snowflake’s triggered tasks offer a sleek solution. 🔍 What Are They? Triggered tasks let you run a SQL task only when new data lands in a a stream, no constant polling needed, and no compute consumed until truly needed. ✅ Key Benefits Zero idle compute: tasks only run in response to data events.  Lower latency: near-immediate processing when changes occur. Resource efficiency: minimizes wasted cycles and cost overhead. Support for many object types: tables, views, dynamic tables, Iceberg tables, directory tables (with refresh). #Snowflake #DataEngineering #SnowflakeTasks #DataPipeline #DataPipelineScheduling