Photo by Freddie Collins on Unsplash

Batch Processing- The Serverless Way

Simple yet elegant batch processing with S3, Lambda, DynamoDB and SNS

The Solution Through A Peephole

Figure-1: Schematic solution

Devil Is In The Details

$ aws --version   # Check the installation
$ aws configure # Configure the credentials under default profile
student_id,fname,lname,course
A10001,Jack,Reacher,"Advanced Physical Security"
A10002,Sherlock,Holmes,"Science Of Deduction"
A10002,Sherlock,Holmes,"Modern Forensics"
A10003,Jane,Marple,"Knitting For Pros"
$ aws dynamodb create-table --table-name PRELIM_STUDENT_DATA \
--attribute-definitions \
AttributeName=student_id,AttributeType=S \
AttributeName=course,AttributeType=S \
--key-schema \
AttributeName=student_id,KeyType=HASH \
AttributeName=course,KeyType=RANGE \
--billing-mode PROVISIONED \
--provisioned-throughput \
ReadCapacityUnits=5,WriteCapacityUnits=5
$ aws sns create-topic --name batch_processing_notification$ aws sns subscribe \
--topic-arn arn:aws:sns:ap-south-1:<account-id>:batch_processing_notification \
--protocol email \
--notification-endpoint my-email@foobar.com
$ aws s3 mb s3://inbound-file-drop
>>>>>>>>>> File Name: trustpolicy.json <<<<<<<<<<
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
>>>>>>>>>> File Name: policy.json <<<<<<<<<<
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::inbound-file-drop/*"
]
},
{
"Effect": "Allow",
"Action": [
"sns:Publish"
],
"Resource": [
"arn:aws:sns:ap-south-1:<account-id>:batch_processing_notification"
]
},
{
"Effect": "Allow",
"Action": [
"dynamodb:PutItem"
],
"Resource": [
"arn:aws:dynamodb:ap-south-1:<account-id>:table/PRELIM_STUDENT_DATA"
]
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:CreateLogGroup",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:*:<account-id>:log-group:/aws/lambda/batch_processing_lambda:log-stream:*",
"arn:aws:logs:*:<account-id>:log-group:/aws/lambda/batch_processing_lambda"
]
}
]
}
$ aws iam create-policy --policy-name BatchProcessingLambdaPolicy --policy-document file://./policy.json    # Create policy$ aws iam create-role --role-name BatchProcessingLambdaRole --assume-role-policy-document file://./trustpolicy.json  # Create role$ aws iam attach-role-policy --role-name BatchProcessingLambdaRole --policy-arn arn:aws:iam::<account-id>:policy/BatchProcessingLambdaPolicy # Attach policy to the role
aws lambda create-function \
--function-name batch_processing_lambda \
--runtime nodejs12.x \
--zip-file fileb://batch_processing_lambda.zip \
--handler index.handler \
--role arn:aws:iam::<account-id>:role/BatchProcessingLambdaRole
$ aws lambda add-permission --function-name batch_processing_lambda --principal s3.amazonaws.com \
--statement-id s3invoke --action "lambda:InvokeFunction" \
--source-arn arn:aws:s3:::inbound-file-drop
>>>>>>>>>> File Name: notification.json <<<<<<<<<<
{
"LambdaFunctionConfigurations": [
{
"LambdaFunctionArn": "arn:aws:lambda:ap-south-1:<account-id>:function:batch_processing_lambda",
"Events": [
"s3:ObjectCreated:Put"
],
"Filter": {
"Key": {
"FilterRules": [
{
"Name": "suffix",
"Value": ".csv"
}
]
}
}
}
]
}
$ aws s3api put-bucket-notification-configuration --bucket inbound-file-drop --notification-configuration file://notification.json

Conclusion

Solutions architect by profession, programmer by passion and photographer by choice…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store