Background In Redshift, there are 2 primary methods for loading data into a Redshift table: INSERT and COPY commands. One might ask, what’s the difference? When is it best to use which one? INSERT Command Per the official AWS docs, INSERT is a SQL command used to, well, insert data row-by-row into a…

Background As of March 2021, I shifted my career domain from full-stack web development into the data track. Fast-forward to now, the three main data storage services I’ve used most heavily include Redshift, RDS, and DynamoDB. In the beginning, although I was quite sure how each of them was unique in…

Use Case Requirements You have an occasional need to copy and / or unload data from an S3 bucket from one account to a Redshift cluster in another account. This can be achieved through Redshift’s COPY & UNLOAD commands in combination with IAM Roles and Role chaining. Background IAM Role chaining also works…

Background IAM Role chaining is a method of granting temporary access to AWS resources by assuming multiple IAM roles in a sequence. This approach is useful when you have a scenario where one role needs to assume another role to access certain resources or perform specific actions. Typically cross-account. In this…

Use Case / Requirements You have been tasked with joining / integrating cross-domain data across more than two separate data stores (i.e. Redshift and S3) stored across multiple AWS accounts for deriving custom analytics for business reporting. This can be achieved by implementing the Data Mesh concept. Background The Data Mesh concept aims to distribute…

Problem statement I was recently dealing with a problem where my API was taking a long time fetching credentials from Vault to connect to Amazon RDS. Vault is a pretty popular tool for managing secrets such as API keys, passwords, and security tokens. …

The diagram above shows how the Redshift Data Share feature enables you to share data across Redshift clusters in different AWS accounts. Here’s how it works: Redshift data share creates a shared VPC endpoint in the producer account that acts as a gateway for the consumer account to access the producer Redshift cluster. The…

Use Case / Requirements You are tasked with joining / integrating data across two separate data stores (i.e. Redshift and S3) stored in a Single AWS account for deriving custom analytics. High-Level Solution Design

Background Terraform is a well-known Infrastructure as Code (IaC) tool that helps in maintaining the state of solution infrastructure and tracks the changes to them over time. However, at times, there can be needs to migrate the Terraform state from one backend to another. …

Background With any cloud service, it is important to secure your credentials to protect your data/service from unauthorized access. In this segment, I will demonstrate how to use Python to generate ephemeral credentials for authenticating to Amazon Redshift. And also explain why ephemeral credentials are safer than static credentials. What are Ephemeral Credentials? 🤔 Ephemeral credentials…

Background Redshift is a powerful cloud-based data warehousing solution provided by AWS. It is designed to handle large amounts of data (petabytes), and support complex analytical queries. While many of its features are well-known, there are several aspects of Redshift that are not widely understood. In this segment, I will explore…

Background One of Redshift’s key features is the ability to create different types of Views. Views are virtual tables that are based on the result of a SQL query. In this post, I’ll do a basic walkthrough of the three common types of views available in Redshift, and when to use…

Background Amazon Redshift is a cloud-based data warehousing solution that allows organizations to store and analyze large volumes of data efficiently. One of the key offerings of Redshift is its ability to handle data sharing between different Redshift clusters, which is critical for organizations that need to share data between multiple…

Background Custom Terraform Modules provide several benefits to infrastructure automation and management. They are reusable code blocks that abstract infrastructure configuration into modular and easily repeatable pieces. Some benefits of using custom Terraform modules Simpler Codebase With a custom Terraform module, you can abstract complex infrastructure logic into a smaller and more manageable codebase. This makes it easier to write…

Redshift offers Data sharing, Data mesh, and Spectrum. But, when to implement which one? Background Cloud data consumption patterns are essential for any organization’s business intelligence operations. The ability to access and analyze data quickly, accurately, and cost-effectively is paramount for success. …

Background Gathering insights from analytic data at a rapid pace for strategic decision making has become a key priority for companies in the last few years. The growth and ubiquity of the AWS Cloud has made it super convenient for companies to leverage various tools and adopt them for rapid application…

Background In my previous post, I shared some security best practices cloud engineers can adhere to when developing applications on AWS. The purpose of this post is to go into depth explaining the main AWS service behind all of those measures that makes security possible: Amazon Identity & Access Management (IAM). IAM at a High-Level …

Background In this blog post, I will walk us through on how to create an ETL job using the following AWS Cloud Services: Amazon S3 (for intermediary data storage) AWS Glue Studio (main engine of the ETL job) Amazon Athena (query tool) ETL stands for Extract, Transform, and Load. In almost…

Background Amazon S3 is a highly-utilized and well-known object storage service offered by Amazon Web Services (AWS). It is virtually used for everything, from storing data related to application logs, to gigabyte-sized media files. High-Level Solution Architecture Manually Upload to S3? Nah, ain’t nobody got time for that… 👎 AWS makes it super easy to utilize S3. …

At my job, we have been working on re-platforming our single-page web app for the past few months, and we’re leveraging Amazon Web Services for rapid development. This is my first AWS project, so it’s a lot to take in. …