{"id":728,"date":"2025-06-08T06:39:11","date_gmt":"2025-06-08T06:39:11","guid":{"rendered":"https:\/\/datadandies.nl\/?p=728"},"modified":"2025-06-09T07:45:47","modified_gmt":"2025-06-09T07:45:47","slug":"loading-data-from-an-aws-rds-mysql-database-to-an-s3-bucket-using-python","status":"publish","type":"post","link":"https:\/\/datadandies.nl\/index.php\/2025\/06\/08\/loading-data-from-an-aws-rds-mysql-database-to-an-s3-bucket-using-python\/","title":{"rendered":"Loading data from an AWS RDS MySQL database to an S3 bucket using Python"},"content":{"rendered":"\n<p>Extracting data from an AWS RDS MySQL database, storing it locally as a CSV and uploading it to S3.<\/p>\n\n\n\n<p><br>This can be done doing a full load (&#8220;extract_mysql_full.py&#8221;) or an incremental refresh (&#8220;extract_mysql_incremental_binlog.py&#8221;) using binlog. In my case, binlog needed to be set up within AWS RDS MySQL by turning on automated backups, creating a parameter group with the relevant binlog parameters (binlog_format, binlog_row_image and binlog_row_metadata among others) and attaching the parameter group to the database.<\/p>\n\n\n<div class=\"ead-preview\"><div class=\"ead-document\" style=\"position: relative;padding-top: 90%;\"><div class=\"ead-iframe-wrapper\"><iframe src=\"\/\/docs.google.com\/viewer?url=https%3A%2F%2Fdatadandies.nl%2Fwp-content%2Fuploads%2F2025%2F06%2Fdataflow_diagram_extract_mysql.pdf&amp;embedded=true&amp;hl=en\" title=\"Ingesloten document\" class=\"ead-iframe\" style=\"width: 100%;height: 100%;border: none;position: absolute;left: 0;top: 0;visibility: hidden;\"><\/iframe><\/div>\t\t\t<div class=\"ead-document-loading\" style=\"width:100%;height:100%;position:absolute;left:0;top:0;z-index:10;\">\n\t\t\t\t<div class=\"ead-loading-wrap\">\n\t\t\t\t\t<div class=\"ead-loading-main\">\n\t\t\t\t\t\t<div class=\"ead-loading\">\n\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/datadandies.nl\/wp-content\/plugins\/embed-any-document\/images\/loading.svg\" width=\"55\" height=\"55\" alt=\"Lader\">\n\t\t\t\t\t\t\t<span>Bezig met laden&#8230;<\/span>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t\t<div class=\"ead-loading-foot\">\n\t\t\t\t\t\t<div class=\"ead-loading-foot-title\">\n\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/datadandies.nl\/wp-content\/plugins\/embed-any-document\/images\/EAD-logo.svg\" alt=\"EAD logo\" width=\"36\" height=\"23\"\/>\n\t\t\t\t\t\t\t<span>Duurt het te lang?<\/span>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<p>\n\t\t\t\t\t\t\t<div class=\"ead-document-btn ead-reload-btn\" role=\"button\">\n\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/datadandies.nl\/wp-content\/plugins\/embed-any-document\/images\/reload.svg\" alt=\"Opnieuw laden\" width=\"12\" height=\"12\"\/> Laad het document opnieuw\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<span>|<\/span>\n\t\t\t\t\t\t\t<a href=\"https:\/\/datadandies.nl\/wp-content\/uploads\/2025\/06\/dataflow_diagram_extract_mysql.pdf\" class=\"ead-document-btn\" target=\"_blank\">\n\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/datadandies.nl\/wp-content\/plugins\/embed-any-document\/images\/open.svg\" alt=\"Open\" width=\"12\" height=\"12\"\/> Openen in nieuwe tab\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div><\/div>","protected":false},"excerpt":{"rendered":"<p>Extracting data from an AWS RDS MySQL database, storing it locally as a CSV and uploading it to S3. This can be done doing a full load (&#8220;extract_mysql_full.py&#8221;) or an incremental refresh (&#8220;extract_mysql_incremental_binlog.py&#8221;) using binlog. In my case, binlog needed to be set up within AWS RDS MySQL by turning on automated backups, creating a&hellip;<\/p>\n<p class=\"more-link\"><a href=\"https:\/\/datadandies.nl\/index.php\/2025\/06\/08\/loading-data-from-an-aws-rds-mysql-database-to-an-s3-bucket-using-python\/\" class=\"themebutton\">Read More<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[43,60,63,37],"class_list":["post-728","post","type-post","status-publish","format-standard","hentry","category-blog","tag-aws","tag-dataengineering","tag-project","tag-python"],"_links":{"self":[{"href":"https:\/\/datadandies.nl\/index.php\/wp-json\/wp\/v2\/posts\/728","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/datadandies.nl\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datadandies.nl\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datadandies.nl\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/datadandies.nl\/index.php\/wp-json\/wp\/v2\/comments?post=728"}],"version-history":[{"count":1,"href":"https:\/\/datadandies.nl\/index.php\/wp-json\/wp\/v2\/posts\/728\/revisions"}],"predecessor-version":[{"id":730,"href":"https:\/\/datadandies.nl\/index.php\/wp-json\/wp\/v2\/posts\/728\/revisions\/730"}],"wp:attachment":[{"href":"https:\/\/datadandies.nl\/index.php\/wp-json\/wp\/v2\/media?parent=728"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datadandies.nl\/index.php\/wp-json\/wp\/v2\/categories?post=728"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datadandies.nl\/index.php\/wp-json\/wp\/v2\/tags?post=728"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}