Are you a lazy engineer? Then you are good engineer.<\/p>\n\n\n\n
Being lazy\/good at automation of processes helps save work, time and money.<\/p>\n\n\n\n
In Data Build Tool (dbt) there are several packages that can help automate tedious tasks.<\/p>\n\n\n\n
The dbt-codegen package provides you with the macro\u2019s generate_base_model and generate_source, among others.<\/p>\n\n\n\n
Macro\u2019s in dbt are essentially functions that make your life easier.<\/p>\n\n\n\n
The generate_base_model macro creates a SQL statement {{ including( \u2018relevant jinja\u2019 ) }} (see what I did there) that includes all the column names.<\/p>\n\n\n\n
In staging layers, it is best practise to explicitly write out all the column names that you want to include in the resulting model (meaning, no \u201cselect *…\u201d but \u201cselect column1, column2, etc…\u201d). This is because whenever there are schema changes, like extra columns added, they get picked up automatically by \u201cselect *…\u201d, whether you want it or not. This can result in errors downstream, among other things.<\/p>\n\n\n\n
Writing out all column names can become really annoying, real fast though, when you\u2019re dealing with tables with hundreds of columns.<\/p>\n\n\n\n
Having to write all those columns yourself or jumping through hoops like Snowflake > add columns in editor > format > copy > etc. = BIG sad.<\/p>\n\n\n\n
The generate_base_model macro does this for you. It creates a SQL statement with all the columns.<\/p>\n\n\n\n
Then there is the generate_source macro. This macro generates sources.yml, when you have not created it yourself yet. Also very handy and efficient. Moral of the story is, keep being lazy.<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Are you a lazy engineer? Then you are good engineer. Being lazy\/good at automation of processes helps save work, time and money. In Data Build Tool (dbt) there are several packages that can help automate tedious tasks. The dbt-codegen package provides you with the macro\u2019s generate_base_model and generate_source, among others. Macro\u2019s in dbt are essentially…<\/p>\n