Scheduled API request processing in AWS

Problem statement

Let us assume a scenario where we have to process events initiated by API calls that will have to be triggered only once at a specific time in the future. Moreover, the ‘future timestamp’ is supplied along with the request. Essentially, this means that the API call will still carry the request. However, the request will have to be processed at a later time. This post attempts to introduce a solution that leverages Amazon API Gateway, Amazon EventBridge, and AWS Lambda function to achieve this.

Solution schematic

With the context in-place, let’s get to the part where we discuss the solution.

Processing delayed invocation of API requests

Following steps are involved in the solution flow:

  1. API Client calls a designated API resource published on Amazon API Gateway and passes a ‘future timestamp’ for the delayed event as part of the request (in JSON format)
  2. The API back-end (a Lambda function named DelayedEventProcessor in the diagram) validates the requests and translates the future timestamp into a cron expression.
  3. Subsequently, DelayedEventProcessor creates a ‘cron’ based rule in Amazon EventBridge and configures a predetermined target for the event (another Lambda function named DelayedEventTarget in the diagram) and passes the incoming request as “Input” for the target
  4. DelayedEventProcessor sends an acknowledgement to the API client
  5. At the designated time in the future, Amazon EventBridge fires the scheduled event, invokes the target (based on the rule), and passes the “Input” request along with name of the cron based rule. Note that, Amazon EventBridge needs appropriate permissions to invoke the target Lambda function. We need to use a Lambda resource based policy that allows specific rule to invoke DelayedEventTarget Lambda function
  6. The target Lambda function processes the request
  7. On successful processing, the target deletes the cron based rule from Amazon EventBridge as part of housekeeping activity. This step avoids creation of too many cron based rule

The target of the event rule could very well be determined at the runtime based on incoming request parameters. The supported rule targets are listed in the following link: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-targets.html.

Further considerations

  • We should not forget to take appropriate measures to secure the API itself
  • The API client must receive acknowledgement (along with a unique correlation identifier). In case, the DelayedEventProcessor faces some error (like invalid request or IAM permission related issues), it must log the same and send the details along with the acknowledgement to the client
  • Amazon EventBridge event rule supports various types of targets including AWS Lambda. However for each type of target, we will have to devise appropriate solution to delete the cron based event rule in Amazon EventBridge, once the request is successfully processed
  • We can configure retry policies and setup dead-letter queues (DLQ) in Amazon EventBridge, in order to handle failures pertaining to triggering of the rule target
  • If the target component is unable to process the request for some reason, we must devise a compensatory action which might also include re-scheduling of the event
  • We may consider sending notifications pertaining to a failure by leveraging Amazon SNS

Sample Code

Sample code for DelayedEventProcessor Lambda function is available in this GitHub repository. The sample also consists for two IAM policies:

  • IAM policy for the execution role of the DelayedEventProcessor
  • IAM resource-based policy for the target Lambda function (DelayedEventTarget)

The code assumes that the Lambda function be configured as an API back-end for Lambda proxy integration. Additionally, the request body field that carries the future timestamp is assumed to be “triggerAt”.

Limitations

The solution presented here has some limitations that needs to be considered.

End note

The solution provides a nicely de-coupled way to handle delayed processing of API requests. For this, it leverages Amazon EventBridge. How did you schedule API requests?

Arik's father | Solution Architect | Amateur Photographer

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Install and Run Monika in MacOS without Package Managers

Why do programming bootcamps fail to live up to their job placement expectations?

Here are 5 fantastic GitHub repositories for learning Python 🐍

5 Stunning Reasons to Learn Python in 2020

Cloud Cost Optimization: A Pivotal part of Cloud Strategy

Update on Proposal 29

API KPIs Part 2: The API Product Management KPIs Worksheet

Agile point-based pricing

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Adrin Mukherjee

Adrin Mukherjee

Arik's father | Solution Architect | Amateur Photographer

More from Medium

AWS NoSQL Workbench & DynamoDB Local

Event Filtering for Lambda Functions using AWS CDK

Create Lambda Layers in AWS Lambda

Hosting a static Single Page Application on AWS using the CDK