This project demonstrates a secure, cost-effective, and automated solution for transferring files between two Amazon S3 buckets within the same region. The system leverages AWS services such as S3, Lambda, Key Management Service (KMS), and Identity and Access Management (IAM) to create a seamless and secure file transfer workflow. This solution addresses various real-world needs, including data backup and replication, triggering downstream processing pipelines, ensuring compliance through secure data copies, and efficiently distributing content across different environments. The automation, driven by S3 event notifications and a Lambda function using Python, eliminates manual intervention, while the serverless architecture ensures scalability and cost-effectiveness through pay-as-you-go pricing and the avoidance of inter-region data transfer costs.
Security is of highest importance, with data encrypted at rest using AWS KMS and access strictly controlled through fine-grained IAM policies. The event-driven nature of the system further reduces the attack surface by only activating the Lambda function when necessary. This combination of automation, cost efficiency, and robust security practices makes this solution a valuable asset for organisations seeking to manage and transfer data securely and efficiently within the AWS ecosystem.
Services Used
Amazon S3
AWS Lambda
AWS Identity and Access Management
AWS Key Management Service
AWS CLI
Terraform
Steps Taken
Source and Destination Buckets: Two S3 buckets are created in the same AWS region. The source bucket is where files are initially uploaded, and the destination bucket is where the files are copied to. Event Notification: An S3 event notification is configured on the source bucket to trigger an AWS Lambda function whenever a new file is uploaded, as displayed in the AWS console. Lambda Function: The Lambda function, written in Python using the Boto3 library, is triggered by the S3 event. It copies the uploaded file from the source bucket to the destination bucket using the 'copy_object' method. Encryption: Both buckets are configured to use AWS KMS for encryption-at-rest, ensuring that all data is securely encrypted. IAM Policies: IAM policies are applied to both buckets and the Lambda function to ensure that only authorised entities have the necessary permissions to perform actions such as uploading, copying, and accessing files.
File Upload: For testing, I uploaded a .txt file to the source bucket using the AWS CLI.
File Uploaded and then Copied: When I used 's3 ls' in the AWS to list the contents of each bucket, I could see that the file was uploaded to the source bucket and automatically uploaded to the destination bucket within a couple of seconds.
File Encryption: The buckets are configured with default server-side encryption with AWS Key Management Service keys (SSE-KMS). This encryption is automatically applied to new objects stored in the bucket.
