Are you a lazy engineer? Then you are good engineer. Being lazy/good at automation of processes helps save work, time and money. In Data Build Tool (dbt) there are several packages that can help automate tedious tasks. The dbt-codegen package provides you with the macro’s generate_base_model and generate_source, among others. Macro’s in dbt are essentially…
Would you like to check if a table contains stale data? Nothing more embarrassing than having a user of a report point out that your data is wrinkled and saggy! As responsible Data Engineers / Analytics Engineers / Insert New Buzzword Job Title Here… we would like to be one step ahead of the users…
Are you collaborating with multiple Analytics Engineers on the same dbt repository? In the following hypothetical setup, dbt is used to load data into Snowflake, and Power BI is used to create reports based on that data. Each engineer has their own schema when developing and deploying locally, following dbt best practices. In Snowflake, this could look…
Combining Data Build Tool (dbt) with CI/CD is a great way to automate tests during deployments from development environments to production. This approach ensures higher data quality and provides early feedback about potential issues 𝐛𝐞𝐟𝐨𝐫𝐞 end users see the data. There’s nothing more embarrassing than having an end user point out a data mistake that could have been caught earlier. By…
Data pijplijn ontwikkeld met behulp van data build tool (dbt), Snowflake, Dagster en Github. Er is een Medallion structuur geïmplementeerd met behulp van transformaties in dbt. Verder is dbt gebruikt om SCD2 te implementeren door middel van snapshots. dbt is ook gebruikt om generic en custom tests uit te voeren zoals het controleren van de…
Just finished an excellent dbt (Data Build Tool) course on Udemy by Zoltan C. Toth. #dbt is a tool that can be used to develop data models. dbt applies many software engineering best practises to the workflow of transforming your data. What I really liked about dbt: – Version control is easily added to a…