Would you like to check if a table contains stale data? Nothing more embarrassing than having a user of a report point out that your data is wrinkled and saggy!

As responsible Data Engineers / Analytics Engineers / Insert New Buzzword Job Title Here… we would like to be one step ahead of the users that consume our products (dashboards, tables, reports, etc.)

There is an easy solution in Data Build Tool (dbt). You can define “freshness” for a table in sources.yml.

It looks like this:

sources:

  – name: cool_source

    database: some_database

    schema: some_schema

    tables:

    – name: orders

      config:

        freshness: # I want to monitor how fresh my orders data is. Command “dbt source freshness” in terminal

          warn_after: # I want to get a warning when the newest record in my dataset is older than 24 hours

            count: 24

            period: hour

So next time a user of an object that you created points out that your data is past its expiry date, your response can be twofold:

1. First of all, how dare you.

2. Secondly, let me add freshness to my sources.yml and toggle notifications on in my Gitlab pipeline that runs the daily refresh so I get an email notifying me whenever my data is stale. Thát will teach them!