Would you like to check if a table contains stale data? Nothing more embarrassing than having a user of a report point out that your data is wrinkled and saggy!<\/p>\n\n\n\n
As responsible Data Engineers \/ Analytics Engineers \/ Insert New Buzzword Job Title Here… we would like to be one step ahead of the users that consume our products (dashboards, tables, reports, etc.)<\/p>\n\n\n\n
There is an easy solution in Data Build Tool (dbt). You can define \u201cfreshness\u201d for a table in sources.yml.<\/p>\n\n\n\n
It looks like this:<\/p>\n\n\n\n
sources:<\/p>\n\n\n\n
– name: cool_source<\/p>\n\n\n\n
database: some_database<\/p>\n\n\n\n
schema: some_schema<\/p>\n\n\n\n
tables:<\/p>\n\n\n\n
– name: orders<\/p>\n\n\n\n
config:<\/p>\n\n\n\n
freshness: # I want to monitor how fresh my orders data is. Command \u201cdbt source freshness\u201d in terminal<\/p>\n\n\n\n
warn_after: # I want to get a warning when the newest record in my dataset is older than 24 hours<\/p>\n\n\n\n
count: 24<\/p>\n\n\n\n
period: hour<\/p>\n\n\n\n
So next time a user of an object that you created points out that your data is past its expiry date, your response can be twofold:<\/p>\n\n\n\n
1. First of all, how dare you.<\/p>\n\n\n\n
2. Secondly, let me add freshness to my sources.yml and toggle notifications on in my Gitlab pipeline that runs the daily refresh so I get an email notifying me whenever my data is stale. Th\u00e1t will teach them!<\/p>\n","protected":false},"excerpt":{"rendered":"
Would you like to check if a table contains stale data? Nothing more embarrassing than having a user of a report point out that your data is wrinkled and saggy! As responsible Data Engineers \/ Analytics Engineers \/ Insert New Buzzword Job Title Here… we would like to be one step ahead of the users…<\/p>\n