How to create unit tests for Databricks SDP Pipelines

[u/data-baggins]

https://community.databricks.com/t5/data-engineering/how-to-unit-test-spark-declarative-pipelines/td-p/172364

Hi Folks - PM from Databricks here and just wanted to share a useful technical blog on how to conduct unit tests with SDP Pipelines today! Here's the use case:

Nightmare scenario - Your new data pipeline updates look good before going to production - every table builds, every expectation passes, and the gold tables have plausible values in them. Looks good right?

But if you push to production, the dashboards would still be showing inflated numbers and an unhappy business partner will be knocking on your DM's

For cases like this you need to utilize Unit Tests to ensure your transformation logic is sound. Expectations help ensure your data looks right, but it still misses what your transformations are doing.

With unit tests, you directly validate your transformations on data inputs you control before the real tables are built. Testing edge cases and finding those annoying "silent" errors

Checkout the blog for a deep dive tutorial on how to use unit testing with Databricks SDP Pipelines!