In this blog post I will walk through how you connect secrurely to MongoDB Atlas with AWS Lambdas through IAM roles.
Custom File Uploads to S3
There are three kinds of geeks at Gridcognition: energy geeks, data science geeks, and software geeks.
I’m the third kind and I thought I’d share what I learnt last week when I was building a file uploader for Amazon S3.
Making your own file uploader to s3 can be tricky due to a lack of guides and documentation (and useless error messages!).
This guide will provide you with base working code to build your file uploader to s3
What we will be building: a simple react file uploader that gets a signed URL from the backend and uploads to s3 via that URL.
This function will live in your backend and is normally called only after you have authenticated and authorized the request. Depending on your backend setup you will need to give it the right permissions to access s3. Note it will generate a seemingly valid URL even if you do not have to correct permissions. However, once you use the URL it will respond with an authentication error.
If you intend on upload via a web browser you will need to configure cors in the s3 bucket. This is under the permissions tab in your s3 bucket. Here is an example config:
You can adjust these however you like but I would suggest going with the least secure approach first then, once you have everything working, tighten the security because an issue with the config here can lead to errors down the road that are very hard to debug.
The HTML can be the normal file input component.
That’s all the code you need to upload your files. It’s quite straightforward but there are a few tricks here. You don’t use FormData, contrary to most file upload guides – this will corrupt your data if you use it to upload to s3. The event.target.files doesn’t return an array so if you try to iterate over it an error will be thrown. Lastly, you must set the Content-Type header to what you have defined in the backend before otherwise, you will get a cryptic error.
Building your own file component is actually very hard. There are a lot of things you need to do right, that’s why companies have emerged offering provider file uploads as a service.
- Scan files for malware
- Compress files before uploading them
- Chunk parallel file uploads, to improve reliability and performance.
- Tighten your cors configuration
- Provide user feedback on the progress of a file upload
If you’re a software engineer and you are interested in accelerating the energy system transition come and join the team.