From the course: Data Quality Testing with Great Expectations

Unlock this course with a free trial

Join today to access over 25,500 courses taught by industry experts.

Understanding data test failures

Understanding data test failures - Great Expectations Tutorial

From the course: Data Quality Testing with Great Expectations

Start my 1-month free trial Buy for my team

Understanding data test failures

“

Now that we've learned how to trigger actions based on checkpoints, let's talk about what we do once an action has been fired. Picture this, it's 9 AM, you just started working and you're checking Slack. You see a notification that your nightly data pipeline run has failed because of a GX test result. What do you do next? Where do you even start? We call the process of figuring out what caused data test failures root cause analysis. I like to call it playing data detective because sometimes it requires quite some detective work. The most common places to look for data issues during root cause analysis are data load and transformation code. Did anything change in the code that loads and transforms the data? API calls and API documentation. Did the code that we use to query an API recently change or did the API itself change what data it returns? Schema changes in the source data. whether any recent changes to the source data models, and finally, log files, whether any issues during the…

Contents