How do you know where an AWS Networking Engineer will be at the next networking event? Just check the next record in the Route Table! Now that I have almost killed the mood with that atrocious joke, allow me to deliver the finishing blow with the presentation my latest AWS networking project. This project was…
AWS Project: access an S3 bucket from an EC2 instance in a private subnet using a VPC Gateway Endpoint. The EC2 instance does not use the public internet to access the bucket, but instead uses the AWS backbone network. This means that traffic from the EC2 instance in the private subnet to the S3 bucket…
Another AWS networking project for your amusement. Connecting two EC2 instances that are located in different regions WITHOUT using the public internet. VPC peering is used to connect to EC2-B-private from EC2-A-private. VPC peering uses the AWS backbone network so traffic does not traverse the public internet; both EC2 instances have private IPs. A route…
A Python script in an Azure Function is used to retrieve data from a REST API. Azure Data Factory is used for orchestration. The Python script is also used to transform and append the data. Transformations included parsing nested JSON files and applying business logic by e.g. adding descriptive columns. The result is several fact-…
AWS Project: VPC with a public subnet containing a NAT gateway. The NAT gateway can be used for outbound traffic to the internet from the EC2 instance in the private subnet. A route table connects the destination 0.0.0.0/0 to the internet gateway for the public subnet. In the public subnet, there is an EC2 instance…
Extracting data from an AWS RDS MySQL database, storing it locally as a CSV and uploading it to S3. This can be done doing a full load (“extract_mysql_full.py”) or an incremental refresh (“extract_mysql_incremental_binlog.py”) using binlog. In my case, binlog needed to be set up within AWS RDS MySQL by turning on automated backups, creating a…
AWS Networking Project: VPC with a public and a private subnet and an internet gateway. A route table connects the destination 0.0.0.0/0 to the internet gateway for the public subnet. In the public subnet, there is an EC2 instance that can be accessed via SSH through the correct security group settings.Using the EC2 instance in…
Some time ago I shared that I would try to broaden my knowledge in the networking department. True to my word, I finished my first networking project: deploying an EC2 instance (a sort of virtual machine) in a public subnet within an Amazon Virtual Private Cloud (VPC). A VPC is like a cloud network environment…
Data pijplijn ontwikkeld met behulp van data build tool (dbt), Snowflake, Dagster en Github. Er is een Medallion structuur geïmplementeerd met behulp van transformaties in dbt. Verder is dbt gebruikt om SCD2 te implementeren door middel van snapshots. dbt is ook gebruikt om generic en custom tests uit te voeren zoals het controleren van de…
Dashboard as a Service for multiple customers using solely Power BI. Power BI Dataflows are used to retrieve data from a REST API. Multiple M queries are used to retrieve data from multiple customers. The Dataflow is also used to transform and append the data. Transformations included parsing nested JSON files and applying business logic…