Use Case / Requirements You have been tasked with joining / integrating cross-domain data across more than two separate data stores (i.e. Redshift and S3) stored across multiple AWS accounts for deriving custom analytics for business reporting. This can be achieved by implementing the Data Mesh concept. Background The Data Mesh concept aims to distribute…

Problem statement I was recently dealing with a problem where my API was taking a long time fetching credentials from Vault to connect to Amazon RDS. Vault is a pretty popular tool for managing secrets such as API keys, passwords, and security tokens. …

Background Terraform is a well-known Infrastructure as Code (IaC) tool that helps in maintaining the state of solution infrastructure and tracks the changes to them over time. However, at times, there can be needs to migrate the Terraform state from one backend to another. This could be due to cost cutting…

Background In many applications, there are times we need to perform tasks that take a lot of time to execute. These tasks might include something like: downloading data from an internet source, processing large sets of data, or performing complex computations. To handle such tasks efficiently, we need a way to…

Background With any cloud service, it is important to secure your credentials to protect your data/service from unauthorized access. In this segment, I will demonstrate how to use Python to generate ephemeral credentials for authenticating to Amazon Redshift. And also explain why ephemeral credentials are safer than static credentials. What are Ephemeral Credentials? 🤔 Ephemeral credentials…

Background Custom Terraform Modules provide several benefits to infrastructure automation and management. They are reusable code blocks that abstract infrastructure configuration into modular and easily repeatable pieces. Some benefits of using custom Terraform modules Simpler Codebase With a custom Terraform module, you can abstract complex infrastructure logic into a smaller and more manageable codebase. This makes it easier to write…

Background It is widely known how JavaScript is a widely used language that runs in web browsers and on the server-side using Node.js. While JavaScript is a powerful and flexible language, it has some drawbacks, such as weak typing and a lack of compile-time error checking. TypeScript is a superset of…

Background Python offers several approaches to concurrency and parallelism. Two of the most popular ways to achieve concurrent execution in Python are multithreading and multiprocessing. While both techniques aim to improve the performance of a Python program, they differ in their approach and use cases. In this segment, I will walk…

Background Software engineering requires constant updates and changes to stay relevant and grow in the market. As software projects grow in complexity and scale, it becomes increasingly important to have a systematic approach to versioning and releasing software updates. This is where Semantic Versioning (SemVer) comes in. In this blog post…

Background Gathering insights from analytic data at a rapid pace for strategic decision making has become a key priority for companies in the last few years. The growth and ubiquity of the AWS Cloud has made it super convenient for companies to leverage various tools…

Step 1- Take this AWS Cloud Essentials Course This is a free course provided by AWS that offers video modules explaining key details all about the AWS Cloud (ex: intro, core and integrated services, security, pricing and billing strategies), and a final course knowledge check. I highly recommend taking this course (it takes about a couple days to…

Um, what!? … Software Engineers, and pretty much all IT Professionals face a wide range of challenges in all stages of their careers. We always want to provide the best experience for our customers, build efficient systems for our business. All while managing things like technical debt, dependencies, and differing personalities with differing…

IT Product Development is far from an easy job. It is like a constantly moving puzzle, and you always have to try to balance stuff. Tech debt Low product suite/team maturity Predefined business requirements Evolving stakeholder expectations Slow to change, internal processes Your product OKRs, goals, backlog Bugs (known and…

Background Software Engineers (and pretty much all IT professionals) are super familiar with this daily phenomenon we engage in called meetings. 🙃 These days looked upon with disdain, meetings have a way of energizing some folks, and utterly draining the life out of others. Over the years, I have accumulated some…

I love Linux. It is my favorite choice for an operating system. As a software engineer, I prefer working with systems that are: Super customizable Not filled with bloatware garbage 🗑️ Fast 🏎️ Easy to use and learn Stable And Linux is one of those, and it has many offerings…

When you have numerous mobile and web applications that are all backed by many hundreds of HTTP endpoints, you need a method to efficiently integrate, deploy, retire, throttle, and secure those microservices/APIs. Each microservice does one specific thing, and serves a particular purpose in the grand context of an application…

Background AWS comes with a ton of freedom and opportunities to build cool stuff. But we all know, with great freedom comes great responsibility. AWS offers a wide array of flexibility in terms of what security configuarations one can set up. …

Background Amazon S3 is a highly-utilized and well-known object storage service offered by Amazon Web Services (AWS). It is virtually used for everything, from storing data related to application logs, to gigabyte-sized media files. High-Level Solution Architecture Manually Upload to S3? Nah, ain’t nobody got time for that… 👎 AWS makes it super easy to utilize S3. …

Background In this blog post, I will walk us through on how to create an ETL job using the following AWS Cloud Services: Amazon S3 (for intermediary data storage) AWS Glue Studio (main engine of the ETL job) Amazon Athena (query tool) ETL stands for Extract, Transform, and Load. In almost…

Some background I love movies, and all things cinema-related 📽 🎞️ 🎬️🍿🎫. I truly think it is immensely cool how so much painstaking and beautiful work goes into making movies. Years and years of hard work, involving meticulous planning, growing expenses, constant deliberation and thinking. All of that hard work compressed into…