I am a firm believer in (at least on a global level) understanding the data you are working with. One way to do this is to generate testrecords in the source system. This offers several advantages: Not only do you understand the data better which makes for better pipelines, it also forces you to communicate…
Ever wondered what COMMIT and ROLLBACK do in Snowflake SQL? I know I have. Let me try to explain it using Snowflake SQL.
Do you have Time for a Date? Understanding Time Zone Offsets in Timestamps As a Data Engineer or a Data Analyst you are working with dates and times constantly. Have you ever wondered what those pesky four numbers at the end of a datetime value mean? Wonder no more! Because I will tell you. Take…
Just finished an excellent dbt (Data Build Tool) course on Udemy by Zoltan C. Toth. #dbt is a tool that can be used to develop data models. dbt applies many software engineering best practises to the workflow of transforming your data. What I really liked about dbt: – Version control is easily added to a…
4 Words I feared saying at the start of my career: “I do not know…” Nowadays, I still don’t use those words that often. Because I also use: I no longer fear saying “I do not know” anymore, because I realized that it is okay to not know everything. In fact, you can’t know everything.…
Did you know you can use the binlog in MySQL database to track changes to the data in said database? This is very useful when you want to perform Change Data Capture operations, like only ingesting data that has been changed. When you want to enable binlog in a MySQL Database in AWS RDS, you…
Recently came across the Rule of Tw…. I mean the Rule of Unlimited Pow… I mean the Rule of Least Power! Within the framework of choosing a programming language to accomplish a certain goal: pick the language that has the least power and will still get the job done. In other words, don’t use a…
Do you have employees who spend hours manually copying values from invoices into Excel or a database? Would you like to automate this? Automatically extracting values from unstructured data like JPG or PDF files is really easy. Snowflake has a feature that will help you with just that. You train a model once to extract…
Dashboard as a Service for multiple customers using solely Power BI. Power BI Dataflows are used to retrieve data from a REST API. Multiple M queries are used to retrieve data from multiple customers. The Dataflow is also used to transform and append the data. Transformations included parsing nested JSON files and applying business logic…