Did you know you can use the binlog in MySQL database to track changes to the data in said database?

This is very useful when you want to perform Change Data Capture operations, like only ingesting data that has been changed.

When you want to enable binlog in a MySQL Database in AWS RDS, you will have to create a parameter group, and adjust the following parameters:

• binlog_format. Setting it to “ROW” will capture all changes to tables in a readable format.
• binlog_row_image. Set it to “FULL”.
• binlog_row_metadata. Set it to “FULL”.

Attach the parameter group to your database: go to your database, click “Modify” and add it there.

You can then check if binlog is enabled by running the following statement in the development environment of your choice:

SHOW VARIABLES LIKE ‘%log_bin%’;                      –log_bin should be “ON”

SHOW VARIABLES LIKE ‘binlog_format’; –binlog_format should be “ROW”

If log_bin is set to “ON” and bin_log format is set to “ROW”, you are good to go. If those changes have not propagated yet, rebooting your database can help.

Now you can use Python BinLogStreamReader class from pymysqlreplication to connect to YourSQL … or rather, YOUR MySQL (… hilarious, I know) Database and get logbin events and use those to create your incremental refresh for example.