From the course: Data Quality Testing with Great Expectations

Unlock this course with a free trial

Join today to access over 25,500 courses taught by industry experts.

Creating custom Expectations

Creating custom Expectations - Great Expectations Tutorial

From the course: Data Quality Testing with Great Expectations

Start my 1-month free trial Buy for my team

Creating custom Expectations

“

So far, we've only used the built-in expectations that ship with GX, but sometimes we need domain-specific rules that go beyond those defaults. Let's take a look at how we can create custom expectations in GX. In the New York City taxi data, for example, we have both pickup and drop-off timestamps. A simple and common check would be to assert that the pickup happens before the drop-off. That makes intuitive sense, but it's a multi-column rule on those two columns, and there isn't a built-in expectation that handles it cleanly. For this case, Gx has an expectation type called unexpected rows expectation. This lets us define a custom SQL query that returns rows which violate specific assertions. If the query returns any rows, the expectation fails. OK, let's try this out and create a custom expectation that fails when the pickup time is after the drop-off time. First of all, we can write the SQL query just as a string in Python. See that batch in the curly brackets? That's a wild card…

Contents