Joel Peterson

The Power BI on Databricks Best Practices Cheat Sheet is now live. This concise guide helps BI teams improve report performance, reduce costs, and scale with confidence. Explore the key integration tips and start optimizing: https://lnkd.in/g56FAdQ5

6

Quick win: Excel’s new AI-powered pivot recommendations instantly suggest the right way to summarize your data—perfect for prepping info to push into Power BI or Tableau. In SQL Server 2025, real-time data masking features keep sensitive info safe, while Pentaho and Cloudera continue to make automated pipeline monitoring easier than ever. What’s your favorite way to boost data security or speed up reporting in your analytics workflow? #Excel #PowerBI #Tableau #SQL #Pentaho #Cloudera #Analytics #DataSecurity #QuickTip

1

As a business-side developer, I’ve appreciated watching BI evolve—from the freedom (and chaos) of ad-hoc analysis to the governance, modeling standards, and guardrails we rely on today. Kevin Arnold does an excellent job breaking down how to empower business developers without slowing them down, using governance tools and best practices as enablers rather than constraints. It’s a thoughtful, practical take on what modern BI maturity should actually look like. #PowerBI #SML #LearningTogether

10

The new DAX Enhanced Time Intelligence feature was released in the Power BI Desktop September update. As you can see in the chart below, I have a basic last year calculation working with a non-standard 445 calendar that many organizations connected with retail use for reporting. Essentially it’s pretty straightforward just to get started for these calendar based functions. Yes, the SQLBIs video is 1 hour but it’s probably not necessary to watch the whole thing unless you really want to get into the finer details. The essential part is ensuring that you setup the ‘calendar options’ correctly in the data model. That part is covered in the “defining a Gregorian calendar’ section of the video here: https://lnkd.in/ghQa2Phb There’s also a demo file that you can use to follow along with the video: https://lnkd.in/g873TD5p After you setup the data model, it’s just a simple matter of using basic time intelligence functions with your defined calendar in place of the table[column] for the “old style” time intelligence functions. For example: Old time intelligence function: Impressions LY Dates =   CALCULATE(     [Impressions],     DATEADD( DimDate[Date], -12, MONTH )   ) Will now be: Impressions LY 445 = CALCULATE(   [Impressions],   DATEADD( '445_Calendar', -12, MONTH ) ) Follow for more Power Bi tips & tricks.

127

1 Comment

One of my favorite new-ish features in Snowflake is Notebooks. You can easily pass data from cell to cell, using SQL in one cell, Python in another, and back to SQL again. Super handy! But watch out! They burn a little bit more credits. They have a 15 minute minimum timeout. Just be aware of your warehouse size and timeout settings!

29

22 Comments

I've been on a bit of a data ingestion project lately and stumbled upon something pretty wild while working with Databricks native write operation and RDBMS. I've been running this daily job that uses a DataFrame to overwrite a table in an Azure SQL database, and for a while, I thought I had it all figured out. My write operations code looked something like this: final_df.write.format("jdbc")\       .mode("overwrite")\       .option("url",url)\       .option('encoding','utf-16')\       .option("driver",driver)\       .option("dbtable","dbo.users")\       .option("user",user)\       .option("password",password)\       .save() The job was running smoothly, the table was updating every day, and all my downstream processes were happy. But then, I discovered something. Overwrite mode mechanism It turns out that when you use the overwrite mode with Databricks' JDBC connection, it doesn't just clear out the data and load the new stuff. It literally drops the entire table and then recreates it based on the schema of your DataFrame. This little detail has some pretty big implications. When the table gets dropped, so do all your constraints, like unique keys and primary keys. Even your specific data type lengths are lost. Databricks then maps the DataFrame's schema to the RDBMS data types and builds the table from scratch. At first, it looks like nothing has changed. The table still exists, the columns are there, and everything seems normal. But under the hood, all those crucial constraints and custom data type lengths you set up are gone. This also means if you change a column name in your DataFrame, the target table will be updated to match, which could easily break any applications or processes that rely on the old column name. The Big Question This got me thinking about alternative approaches. One idea is to truncate the table before ingesting the new data. This would keep the original table structure and all its constraints intact. However, from what I've seen, this works best with ODBC drivers and not the native Databricks JDBC connection. So, I'm curious: how do you all handle this? Do you have a workaround for keeping your constraints and table structure while still using Databricks to manage your data updates? I'm sure I'm not the only one who has faced this, and I'd love to hear your solutions and best practices!

9

My Go-To Recommendation for (Nearly) Every SQL Server Deployment Here’s one of the first things I install on any new SQL Server - and it saves hours of pain. Extended Events can be brutal. You’ve got to pick the right event, collect the right data, and then battle an unforgiving GUI... all just to troubleshoot blocking, waits, or slow queries. That’s why I install sp_humanevents. https://lnkd.in/g69sVZ4k It makes troubleshooting blocking, waits, and deadlocks actually... well, humane. It gets you to the answer, fast. Think of it as a modern, lightweight replacement for SQL Server Profiler (bless its dear soul). Massive thanks to Erik Darling for this indispensable tool. My servers (and my sanity) are grateful! What’s your go-to troubleshooting tool? #SQLServer #DBA #PerformanceTuning #ITPros #DatabaseEngineer

1