Did you know you can use the binlog in MySQL database to track changes to the data in said database? This is very useful when you want to perform Change Data Capture operations, like only ingesting data that has been changed. When you want to enable binlog in a MySQL Database in AWS RDS, you…
Recently came across the Rule of Tw…. I mean the Rule of Unlimited Pow… I mean the Rule of Least Power! Within the framework of choosing a programming language to accomplish a certain goal: pick the language that has the least power and will still get the job done. In other words, don’t use a…
Do you have employees who spend hours manually copying values from invoices into Excel or a database? Would you like to automate this? Automatically extracting values from unstructured data like JPG or PDF files is really easy. Snowflake has a feature that will help you with just that. You train a model once to extract…
Dashboard as a Service for multiple customers using solely Power BI. Power BI Dataflows are used to retrieve data from a REST API. Multiple M queries are used to retrieve data from multiple customers. The Dataflow is also used to transform and append the data. Transformations included parsing nested JSON files and applying business logic…
Did you know you can view the query plan of Power BI Dataflows? Simply right click on a step in the Dataflow and click on “View query plan”. This shows what plan is used to execute the query. You can also see if queries are being folded (if transformations are being done by the database…
Do you keep your CV up to date? As a freelancer you switch between projects relatively often. Having an up-to-date CV is enormously valuable because most clients are looking for specialists that can start yesterday. Meaning, clients would like to receive your CV right after an introductory call, NOT after you have finally finished updating…
Een cloud reserveringssysteem waarbij potentiele klanten op een website een datum en tijd kunnen selecteren, een emailadres in kunnen voeren en vervolgens een bevestingsmail in hun inbox krijgen. De infrastructuur is ontwikkeld met behulp van een S3 Bucket (static hosting), een Lambda Function, een DynamoDB tabel, een API Gateway en Simple Email Service. CloudFormation is…
Reserveringssysteem met behulp van Github Actions, Docker en Azure Container Registry. Potentiële klanten gebruiken de website door een datum en tijd te selecteren en hun e-mailadres in te voeren. De frontend bestaat uit een HTML-bestand met wat CSS en JavaScript. De backend is een Python-script met een Flask-applicatie en een SQLite-database. JavaScript stuurt verzoeken naar…
Currently reading “Data Pipelines Pocket Reference” by James Densmore. Interesting little book. In the beginning James gives a summarized checklist of what a Data Engineer needs to have to be succesful. In this post I’m trying to see if I have what it takes to be a Data Engineer according to the prerequisites as listed…
𝐂𝐨𝐦𝐩𝐫𝐞𝐬𝐬𝐢𝐧𝐠 𝐝𝐚𝐭𝐚 𝐫𝐞𝐝𝐮𝐜𝐞𝐬 𝐬𝐭𝐨𝐫𝐚𝐠𝐞 (𝐜𝐨𝐬𝐭𝐬). Two compression techniques that are used a lot are called Run Length Encoding (RLE) and Dictionary Encoding (DE). These compression methods are used in the file format parquet but also by the Vertipaq engine in Power BI. Knowing about these compression techniques helps you better understand why the file size…