I am a firm believer in (at least on a global level) understanding the data you are working with.

One way to do this is to generate testrecords in the source system.

This offers several advantages:

Not only do you understand the data better which makes for better pipelines, it also forces you to communicate with stakeholders and understand their needs.

Understanding the needs of your stakeholders leads to you being able to develop a product that is better suited to their needs.

This in turn, leads to your data actually being used, instead of it rotting away in some database that was never touched.

As a Data Engineer it is very tempting to just “do the work”: build a pipeline.

However, this does not always result in added value, nor does it result in job satisfaction. After all, there is little that makes a Data Engineer happier than seeing his products actually being used by the business.