Extracting data from an AWS RDS MySQL database, storing it locally as a CSV and uploading it to S3. This can be done doing a full load (“extract_mysql_full.py”) or an incremental refresh (“extract_mysql_incremental_binlog.py”) using binlog. In my case, binlog needed to be set up within AWS RDS MySQL by turning on automated backups, creating a…
Do you have Time for a Date? Understanding Time Zone Offsets in Timestamps As a Data Engineer or a Data Analyst you are working with dates and times constantly. Have you ever wondered what those pesky four numbers at the end of a datetime value mean? Wonder no more! Because I will tell you. Take…
𝐂𝐨𝐦𝐩𝐫𝐞𝐬𝐬𝐢𝐧𝐠 𝐝𝐚𝐭𝐚 𝐫𝐞𝐝𝐮𝐜𝐞𝐬 𝐬𝐭𝐨𝐫𝐚𝐠𝐞 (𝐜𝐨𝐬𝐭𝐬). Two compression techniques that are used a lot are called Run Length Encoding (RLE) and Dictionary Encoding (DE). These compression methods are used in the file format parquet but also by the Vertipaq engine in Power BI. Knowing about these compression techniques helps you better understand why the file size…