Background As of March 2021, I shifted my career domain from full-stack web development into the data track. Fast-forward to now, the three main data storage services I’ve used most heavily include Redshift, RDS, and DynamoDB. In the beginning, although I was quite sure how each of them was unique in…

Use Case Requirements You have an occasional need to copy and / or unload data from an S3 bucket from one account to a Redshift cluster in another account. This can be achieved through Redshift’s COPY & UNLOAD commands in combination with IAM Roles and Role chaining. Background IAM Role chaining also works…

Use Case / Requirements You have been tasked with joining / integrating cross-domain data across more than two separate data stores (i.e. Redshift and S3) stored across multiple AWS accounts for deriving custom analytics for business reporting. This can be achieved by implementing the Data Mesh concept. Background The Data Mesh concept aims to…

Problem statement I was recently dealing with a problem where my API was taking a long time fetching credentials from Vault to connect to Amazon RDS. Vault is a pretty popular tool for managing secrets such as API keys, passwords, and security tokens. It is commonly used with other AWS services…

The diagram above shows how the Redshift Data Share feature enables you to share data across Redshift clusters in different AWS accounts. Here’s how it works: Redshift data share creates a shared VPC endpoint in the producer account that acts as a gateway for the consumer account to access the producer Redshift cluster. The…

Use Case / Requirements You are tasked with joining / integrating data across two separate data stores (i.e. Redshift and S3) stored in a Single AWS account for deriving custom analytics. High-Level Solution Design

Background With any cloud service, it is important to secure your credentials to protect your data/service from unauthorized access. In this segment, I will demonstrate how to use Python to generate ephemeral credentials for authenticating to Amazon Redshift. And also explain why ephemeral credentials are safer than static credentials. What are Ephemeral Credentials? 🤔 Ephemeral credentials…

Background Redshift is a powerful cloud-based data warehousing solution provided by AWS. It is designed to handle large amounts of data (petabytes), and support complex analytical queries. While many of its features are well-known, there are several aspects of Redshift that are not widely understood. In this segment, I will…

Background One of Redshift’s key features is the ability to create different types of Views. Views are virtual tables that are based on the result of a SQL query. In this post, I’ll do a basic walkthrough of the three common types of views available in Redshift, and when to…

Background Amazon Redshift is a cloud-based data warehousing solution that allows organizations to store and analyze large volumes of data efficiently. One of the key offerings of Redshift is its ability to handle data sharing between different Redshift clusters, which is critical for organizations that need to share data between multiple…

Redshift offers Data sharing, Data mesh, and Spectrum. But, when to implement which one? Background Cloud data consumption patterns are essential for any organization’s business intelligence operations. The ability to access and analyze data quickly, accurately, and cost-effectively is paramount for success. …

Background Gathering insights from analytic data at a rapid pace for strategic decision making has become a key priority for companies in the last few years. The growth and ubiquity of the AWS Cloud has made it super convenient for companies to leverage various tools and adopt them for rapid…

Background In this blog post, I will walk us through on how to create an ETL job using the following AWS Cloud Services: Amazon S3 (for intermediary data storage) AWS Glue Studio (main engine of the ETL job) Amazon Athena (query tool) ETL stands for Extract, Transform, and Load. In almost…

Amazon S3 is a highly-utilized and well-known object storage service offered by Amazon Web Services (AWS). It is virtually used for everything, from storing data related to application logs, to gigabyte-sized media files. Manually Upload to S3? Nah, ain’t nobody got time for that… 👎 AWS makes it super easy to utilize S3. You can just create your bucket, and upload…

I heard really cool things about Amazon Athena. So I thought I’d give it a shot while enjoying a shot of delicious ☕ espresso.

For the past few months, I have been developing with Amazon Redshift. Redshift is a data warehousing service that uses SQL to analyze structured and semi-structured data across data warehouses, operational and analytical data sources, and data lakes.