Background As of March 2021, I shifted my career domain from full-stack web development into the data track. Fast-forward to now, the three main data storage services I’ve used most heavily include Redshift, RDS, and DynamoDB. In the beginning, although I was quite sure how each of them was unique in…

Use Case Requirements You have an occasional need to copy and / or unload data from an S3 bucket from one account to a Redshift cluster in another account. This can be achieved through Redshift’s COPY & UNLOAD commands in combination with IAM Roles and Role chaining. Background IAM Role chaining also works…

Background IAM Role chaining is a method of granting temporary access to AWS resources by assuming multiple IAM roles in a sequence. This approach is useful when you have a scenario where one role needs to assume another role to access certain resources or perform specific actions. Typically cross-account. In…

Use Case / Requirements You have been tasked with joining / integrating cross-domain data across more than two separate data stores (i.e. Redshift and S3) stored across multiple AWS accounts for deriving custom analytics for business reporting. This can be achieved by implementing the Data Mesh concept. Background The Data Mesh concept aims to…

Problem statement I was recently dealing with a problem where my API was taking a long time fetching credentials from Vault to connect to Amazon RDS. Vault is a pretty popular tool for managing secrets such as API keys, passwords, and security tokens. It is commonly used with other AWS services…

Background I have spent a great chunk of my career building customer-facing applications. This would frequently involve building UIs in Angular, Ember, and backend APIs in Java+Spring, Express. And then deploying them to various environments/prod using CI/CD pipelines (mostly GitLab CI and Jenkins). More recently (circa 2021), I switched my domain…

The diagram above shows how the Redshift Data Share feature enables you to share data across Redshift clusters in different AWS accounts. Here’s how it works: Redshift data share creates a shared VPC endpoint in the producer account that acts as a gateway for the consumer account to access the producer Redshift cluster. The…

Use Case / Requirements You are tasked with joining / integrating data across two separate data stores (i.e. Redshift and S3) stored in a Single AWS account for deriving custom analytics. High-Level Solution Design

A crash course with infographics on what these mean, and when to use which one. Background One critical aspect of REST API development is the way in which clients make requests and receive responses. In this segment, I will cover the 3 main methods of making requests in REST APIs …