Background In Redshift, there are 2 primary methods for loading data into a Redshift table: INSERT and COPY commands. One might ask, what’s the difference? When is it best to use which one? INSERT Command Per the official AWS docs, INSERT is a SQL command used to, well, insert data row-by-row into a…

Background As of March 2021, I shifted my career domain from full-stack web development into the data track. Fast-forward to now, the three main data storage services I’ve used most heavily include Redshift, RDS, and DynamoDB. In the beginning, although I was quite sure how each of them was unique in…

Use Case Requirements You have an occasional need to copy and / or unload data from an S3 bucket from one account to a Redshift cluster in another account. This can be achieved through Redshift’s COPY & UNLOAD commands in combination with IAM Roles and Role chaining. Background IAM Role chaining also works…

Use Case / Requirements You have been tasked with joining / integrating cross-domain data across more than two separate data stores (i.e. Redshift and S3) stored across multiple AWS accounts for deriving custom analytics for business reporting. This can be achieved by implementing the Data Mesh concept. Background The Data Mesh concept aims to distribute…

Problem statement I was recently dealing with a problem where my API was taking a long time fetching credentials from Vault to connect to Amazon RDS. Vault is a pretty popular tool for managing secrets such as API keys, passwords, and security tokens. …

The diagram above shows how the Redshift Data Share feature enables you to share data across Redshift clusters in different AWS accounts. Here’s how it works: Redshift data share creates a shared VPC endpoint in the producer account that acts as a gateway for the consumer account to access the producer Redshift cluster. The…

Use Case / Requirements You are tasked with joining / integrating data across two separate data stores (i.e. Redshift and S3) stored in a Single AWS account for deriving custom analytics. High-Level Solution Design

Background With any cloud service, it is important to secure your credentials to protect your data/service from unauthorized access. In this segment, I will demonstrate how to use Python to generate ephemeral credentials for authenticating to Amazon Redshift. And also explain why ephemeral credentials are safer than static credentials. What are Ephemeral Credentials? 🤔 Ephemeral credentials…

Background Redshift is a powerful cloud-based data warehousing solution provided by AWS. It is designed to handle large amounts of data (petabytes), and support complex analytical queries. While many of its features are well-known, there are several aspects of Redshift that are not widely understood. In this segment, I will explore…

Background One of Redshift’s key features is the ability to create different types of Views. Views are virtual tables that are based on the result of a SQL query. In this post, I’ll do a basic walkthrough of the three common types of views available in Redshift, and when to use…

Background Amazon Redshift is a cloud-based data warehousing solution that allows organizations to store and analyze large volumes of data efficiently. One of the key offerings of Redshift is its ability to handle data sharing between different Redshift clusters, which is critical for organizations that…

Redshift offers Data sharing, Data mesh, and Spectrum. But, when to implement which one? Background Cloud data consumption patterns are essential for any organization’s business intelligence operations. The ability to access and analyze data quickly, accurately, and cost-effectively is paramount for success. As such, businesses have turned to cloud technologies like…

Background Gathering insights from analytic data at a rapid pace for strategic decision making has become a key priority for companies in the last few years. The growth and ubiquity of the AWS Cloud has made it super convenient for companies to leverage various tools and adopt them for rapid application…

Background In this blog post, I will walk us through on how to create an ETL job using the following AWS Cloud Services: Amazon S3 (for intermediary data storage) AWS Glue Studio (main engine of the ETL job) Amazon Athena (query tool) ETL stands for Extract, Transform, and Load. In almost…

Background Amazon S3 is a highly-utilized and well-known object storage service offered by Amazon Web Services (AWS). It is virtually used for everything, from storing data related to application logs, to gigabyte-sized media files. High-Level Solution Architecture Manually Upload to S3? Nah, ain’t nobody got time for that… 👎 AWS makes it super easy to utilize S3. …

Background I heard really cool things about Amazon Athena. So I thought I’d give it a shot while enjoying a shot of delicious ☕ espresso. So, what is Amazon Athena? 🧐 Athena is a serverless interactive query service offered by Amazon Web Services (AWS). It is special by how it makes it easy to analyze data using…

For the past few months, I have been developing with Amazon Redshift. Redshift is a data warehousing service that uses SQL to analyze structured and semi-structured data across data warehouses, operational and analytical data sources, and data lakes. Unlike traditional data warehouse solutions, Redshift offers a wide range of features…