Date Published: July 18, 2019TOC
Overview
This blog will show how to Process a request from SNS using AWS lambda and pass it for further processing via SQS. These integration can be typically used for infrastructure automation.
- Lets say you have an event captured in SNS from monitoring system and now want to take action against it.
- Depending on the type of alert you will have lambda function triggered, which will place the request to specific Queue (SQS)
- Jenkins job subscribed to the particular Queue will consume data and take appropriate action as defined in the job
Problem Statement
For this new setup, a JSON blob (input) is to be published to an SNS topic, processed by a Lambda function, and then the output pushed into an SQS queue.
Input: {“text”: “Hello World”, “amount”: 10}
Expected Output: {“Result”: “ Hello World”}
What is Lambda ?
AWS’s Lambda is a compute service that lets you run code without provisioning or managing servers. AWS Lambda executes your code only when needed and scales automatically, from a few requests per day to thousands per second.
REF: https://docs.aws.amazon.com/lambda/latest/dg/welcome.html
What is SQS ?
Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications
REF: https://aws.amazon.com/sqs/
What is SNS ?
Amazon Simple Notification Service (SNS) is a highly available, durable, secure, fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems, and serverless applications. Amazon SNS provides topics for high-throughput, push-based, many-to-many messaging. Using Amazon SNS topics, your publisher systems can fan out messages to a large number of subscriber endpoints for parallel processing.
REF: https://aws.amazon.com/sns/
Setup Details
Prerequisite
- Valid AWS Account with IAM roles for creating LAMBDA, SQS and SNS
- AWS CLI setup in your local system to run the cloudformation templateNoteThis setup may incur cost
Launch the Serverless stack
~: git clone https://github.com/AVM-Consulting/declarative-infrastructure.git
################################# Output #################################
Cloning into 'declarative-infrastructure'...
remote: Enumerating objects: 41, done.
remote: Counting objects: 100% (41/41), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 41 (delta 5), reused 36 (delta 2), pack-reused 0
Unpacking objects: 100% (41/41), done.
################################# Output #################################
cd declarative-infrastructure/lambda
~/declarative-infrastructure/lambda: aws cloudformation create-stack --stack-name serverless-stack --capabilities CAPABILITY_IAM --template-body file://serverless.yaml
################################# Output #################################
~/declarative-infrastructure/lambda: aws cloudformation create-stack --stack-name serverless-stack --capabilities CAPABILITY_IAM --template-body file://serverless.yaml
{
"StackId": "arn:aws:cloudformation:us-west-2:303882392497:stack/serverless-stack/e7d5d520-a971-11e9-82dc-0af0816e310e"
}
################################# Output #################################
What did the above command do ?
- Created a SNS TOPIC called Left-Pad-Topic
- Created a SQS Queue called serverless-stack-SQSQueue
- Created a Lambda Function left-Pad-Test1 whose function is to:
- To accept input from SNS as a json in the form of {“text”:“Hello World”,“space”:“10”}
- Use the second parameter from the input as a space and publish the output to SQS
- Expected output will be {“text”:“Hello World”,“space”:“10”} (See Screen Cap)
Note
NOTE: Above is just a hypothetical case main motive is to explain the integration of SQS, SNS with Lambda
Screen Caps
Lambda Function
AWSTemplateFormatVersion: 2010-09-09
Description: CloudFormation Template for LeftPads Test
Resources:
LambdaExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
- 'arn:aws:iam::aws:policy/AmazonSQSFullAccess'
LambdaFunction1:
Type: 'AWS::Lambda::Function'
Properties:
Code:
ZipFile: |
from __future__ import print_function
import json
import boto3
print('Loading function')
def lambda_handler(event, context):
message = event['Records'][0]['Sns']['Message']
json_msg = json.loads(message)
msg = str(message)
pad = len(msg) + int(json_msg['space'])
print("Result: " + msg.rjust(pad))
cl=boto3.client('sqs')
queue = cl.list_queues(QueueNamePrefix='serverless-stack')
cl.send_message(QueueUrl=str(queue["QueueUrls"][0]), MessageBody=msg.rjust(pad))
return msg.rjust(pad)
Description: LeftPad Test
FunctionName: left-Pad-Test1
Handler: index.lambda_handler
Role: !GetAtt
- LambdaExecutionRole
- Arn
Runtime: python2.7
Timeout: '3'
LambdaPermission1:
Type: 'AWS::Lambda::Permission'
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref LambdaFunction1
Principal: sns.amazonaws.com
SourceArn: !Ref SNSTopic
LambdaLogGroup1:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: !Sub "/aws/lambda/${LambdaFunction1}"
RetentionInDays: '7'
SNSTopic:
Type: 'AWS::SNS::Topic'
Properties:
DisplayName: Left-Pad-Topic
TopicName: Left-Pad-Topic
Subscription:
- Endpoint: !GetAtt
- LambdaFunction1
- Arn
Protocol: lambda
SQSQueue:
Type: 'AWS::SQS::Queue'
AllowLambds2SQSPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues: [!Ref 'SQSQueue']
PolicyDocument:
Version: '2008-10-17'
Id: PublicationPolicy1
Statement:
- Sid: Allow-Lambda-SendMessage
Effect: Allow
Principal: "*"
Action: "sqs:*"
Resource: "*"
Condition:
ArnEquals:
aws:SourceArn: !Ref 'LambdaFunction1'
SNS Topic -> Click Publish Message
SNS-Topic -> Sample Input
SQS-Topic -> View output
Clean the setup
~/declarative-infrastructure/lambda: aws cloudformation delete-stack --stack-name serverless-stack
By the way, check out our best AWS deal: https://www.avmconsulting.net/well-architected-review
Write a Reply or Comment