A Python script in an Azure Function is used to retrieve data from a REST API. Azure Data Factory is used for orchestration. The Python script is also used to transform and append the data. Transformations included parsing nested JSON files and applying business logic by e.g. adding descriptive columns. The result is several fact-…
Extracting data from an AWS RDS MySQL database, storing it locally as a CSV and uploading it to S3. This can be done doing a full load (“extract_mysql_full.py”) or an incremental refresh (“extract_mysql_incremental_binlog.py”) using binlog. In my case, binlog needed to be set up within AWS RDS MySQL by turning on automated backups, creating a…
Some time ago I developed a reservation system using AWS CDK. Potential customers can visit a website that contains a calendar, click on a date, click on a time and finally, enter their email address. A confirmation email will be sent to their address right after the reservation. I built this infrastructure using an S3…
So you’ve created a wonderful instance method for your class. However, you would like to use the instance attributes created in that first instance method, in a second instance method WITHOUT calling the first instance method. In that case you can use the following inside your second instance method: self.first_instance_method()
Merry Christmas my dear network. Ready for some Python on Christmas day? Here we go. At the risk of kicking in open doors: don’t confuse the semantics of variable declaration in for-loops with functions. Variables that are defined in a function, are local to that function. They exist in the so called “local scope” of…
I remember reading about lazy evaluation for DAX variables the first time several years back in the book “The Definitive Guide to DAX” from Marco Russo & Alberto Ferrari. I also remember being very confused about the concept back then. It took some time and research in order to understand the concept a little better.…
Are you trying to reduce the runtime of your Python code? Faster code can reduce cloud costs, especially with serverless solutions like Azure Functions or AWS Lambda. One way to have your script run faster is by using so called “list comprehension” in Python. List comprehensions are a concise and often faster way to create…
Speed up your SQL development with this one trick! … No but seriously, this little hack makes SQL coding so much more convenient. Most SQL development environments have the possibility to perform so called “Multi Cursor edits” (also called “Box selection”). Meaning, you can edit several lines in your block of code at once. In…
Did you know you can use SQL queries in your Python worksheets in Snowflake? This works as follows: dataframe = session.sql( ”’ YOUR SQL QUERY GOES HERE ”’ ) And voila! Don’t forget to return it in the function in the worksheet if you wish to inspect the resulting dataframe in the results pane. I…
I’ve been using Python for a few years now and I have been using libraries, packages and modules during those years and took them for granted I have never stopped and wondered how these relate to each other. The way I understand it, the relation between libraries, packages and modules is hierarchical: libraries consist of…