How-to: Upload Files to Amazon S3 via an API

Amazon S3 is a highly-utilized and well-known object storage service offered by Amazon Web Services (AWS). It is virtually used for everything, from storing data related to application logs, to gigabyte-sized media files.

Manually Upload to S3? Nah, ain’t nobody got time for that… 👎

AWS makes it super easy to utilize S3. You can just create your bucket, and upload whatever data you want into it.

But, what if… instead of manually navigating to our bucket, we upload objects into it through invoking an API? 😃

API to the rescue! 🦸

Having an API to upload data into S3 will enable:

  • LESS manual work 💪
  • Consumer-friendly interaction
  • Quicker process to upload
  • API-reusability

Let’s go build our API!

We will be building our file upload solution using the following AWS Services:

High-Level Solution Architecture

Step 1 — Create an IAM Role with the required policy permissions

This role will consist of a policy that will allow our API to upload files into our S3 Bucket through an API Gateway endpoint.

Make sure your policy looks similar to what I have shared below:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::file-upload-bucket/*"
}
]
}
IAM Role
Policy attached to the IAM Role with Custom Permissions to PUT in S3
API Gateway service included as a Trusted Entity to our IAM Role

Step 2 — Create the API Gateway

Now, we are going to a create an API Gateway that will be the entry to our File Uplaod API.

As shown below, the folder and file resources here represent our S3 Bucket and S3 Object respectively.

These resources will serve as parameters that we will specify as part of the PUT request endpoint/URL by the client.

2a) Create the {folder} resource
2b) Create the {file} child resource under {folder}
2c) Add the /PUT Request under {folder}/{file} with the following settings
2d) Add your URL Path Parameters to match the S3 Bucket and Objects with the API Resources
2e) Include application/csv as a Binary Media Type.

Step 3 — Deploy the API and Call it!

My Postman call to the API Gateway with the bucket and file name

Check out our S3 Bucket:

test.csv file has landed in S3

Want to learn more about Data Engineering? AWS Cloud?

Check out my series listed below! 🙂

Data Engineering:👇

Data Engineering & Cloud

5 stories

AWS Cloud:👇

AWS & Cloud Computing

11 stories

--

--

--

Multi-disciplinary Software Engineer specialized in building products users love. Today, I manage & secure big data in the cloud. All views shared are my own.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

CASE STUDY ON AZURE KUBERNETES SERVICE

The kiss between Montrose and Sammy would’ve been a

Real Time Streaming Analytics Application using AWS CDK

Prevent Hibernate from doing N+1 selects with constructor expression

Yes, Virginia, You Should Use WordPress

The power of Oracle GoldenGate

My 10 Principles to follow as a Developer

Did You Remember Everything Before Deploying to Production?

In If when we create new digital services, products or make big changes in an already existing there can be a lot to think about, lots of different people to talk to, options and restrictions to navigate.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Abdul Wahab

Abdul Wahab

Multi-disciplinary Software Engineer specialized in building products users love. Today, I manage & secure big data in the cloud. All views shared are my own.

More from Medium

Launching an EC2 Instance for deploying microservices

Build a AWS Custom Authorizer in Python for AppSync

Usage of built-in HTTP endpoints for Lambda functions in Java

AWS Certified Solution Architect — Aurora