Serverless Workflows using AWS Step Functions

By

Overview

AWS Step Functions is a visual workflow service that helps developers use AWS services to build distributed applications, automate processes, orchestrate microservices in a serverless manner.

Step Functions simplifies the process of managing complex workflows, allowing you to focus on business logic rather than infrastructure management.

Key Features and Benefits

1. Visual Workflow Designer
Step Functions provides a visual workflow designer that allows you to define and visualize your workflows graphically. This intuitive interface makes it easy to model complex workflows using a state machine-based approach. By dragging and dropping states onto the canvas, you can design workflows with ease, gaining a comprehensive understanding of the entire process at a glance.

2. Scalability and Resilience
One of the primary advantages of Step Functions is its scalability and resilience. As a fully managed service, AWS automatically handles the scaling and availability of your workflows, ensuring they can handle any workload, from small-scale tasks to enterprise-level processes. Additionally, Step Functions automatically retries failed executions, making your workflows resilient to transient errors and ensuring consistent performance.

3. Integration with AWS Services
Step Functions seamlessly integrates with a wide range of AWS services, allowing you to leverage the full capabilities of the AWS ecosystem within your workflows. Whether you need to invoke Lambda functions, interact with Amazon S3, query data in DynamoDB, or trigger notifications through Amazon SNS, Step Functions provides native integration with these services, enabling you to build robust and versatile workflows tailored to your specific requirements.

4. Error Handling and Monitoring
Monitoring and debugging workflows is made simple with Step Functions. You can track the execution status of your workflows in real-time using Amazon CloudWatch, gaining insights into execution history, duration, and resource utilization. Step Functions also offers built-in error handling capabilities, allowing you to define error handling logic within your workflows and gracefully handle exceptions, ensuring reliable execution even in the face of errors or unexpected conditions.


Demo

Use Case


In this blog, I would like to showcase how the AWS Step functions can be used to simulate a real world approval mechanism.

The demo application will cover the following use-cases:

  • A sample automated task using AWS Lambda to perform some computation logic.
  • Invoking a Lambda function exposed using API Gateway to simulate invocation of third party api.
  • A user task(combination of AWS Lambda and API Gateway) with callback mechanism which suspends the workflow until an end user provides a manual approval.
  • Email Notification using AWS SES to notify the user about the approval tasks.

Architecture



The implementation involves the usage of following tech stack:
  • Serverless framework - Serverless framework helps you to create various AWS Artifacts like AWS Lamdba, API gateway , Step functions etc in a declarative way.
         Serverless Framework
  • Nodejs - Serverless framework  supports a variety of language. For the sake of simplicity, NodeJs is used for the backend.
  • AWS-SDK - Helps you to d evelop and deploy applications with JavaScript.

Pre-Requisites

  • AWS Account with an IAM user(access keys)
  • AWS cli should be installed on your machine and configured using aws configure.
  • Nodejs should be installed on your machine
  • An Email address needs to be setup and verified in the AWS SES service (via AWS Console)
         https://www.freecodecamp.org/news/set-up-aws-simple-email-service/

Execution

Execute the following steps:

1. Clone The Git Repo


2. lambda-approval  ( User approval task)

This sub project will expose 2 endpoints /approve & /reject over API gateway and backed by a AWS Lambda backend. The endpoints are designed to simulate the user approval process and invoke a call back to the Step function. The AWS Step function will be in a suspended state until it receives the callback

Commands:
  • npm install
  • serverless deploy             (this will deploy the project on to aws)
After deployment, please note down the url of the exposed methods (from your command line). This needs to be updated in the env.json in step 4

3. lambda-api-gateway( Third party api simulation)

This sub project will expose an endpoint /double over API Gateway and backed by a AWS Lambda backend. The endpoint performs a simple calculation of doubling the input and then callback the Step function.  The AWS Step function will be in a suspended state until it receives the callback

Commands:
  • npm install
  • serverless deploy             (this will deploy the project on to aws)
After deployment, please note down the url of the exposed method (from your command line) . This needs to be updated in the env.json in step 4

4. chaining-step-functions-callback( The state machine implementation)

This sub project contains the implementation of the core state machine for the AWS Step functions. The State machine is exposed via an API Gateway and accepts 2 integers as an input over the POST method.

After receiving the input, the state machine will transition through various states viz.
 Start-> IsInputValid -> Calculate -> Approval -> Finalize ->END

IsInputValid - This state is backed by a task (AWS Lambda function) which basically validates if inputs are integers and if not the state machine will directly transition to the DefaultResponse End state.
If there are no errors, the state is transitioned to Calculate State.

Calculate - this is backed by a lambda function which initially adds 2 inputs. The output of the addition is then passed on to the third party api exposed by the lambda-api-gateway endpoint to simulate the behavior of third party api.

On successful execution, the state transitions to Approval state, else  DefaultResponse End state.

Approval - This is a wait state in which the state machine remains suspended, until it receives a callback from the api's exposed by the lambda-approval .

The task also generates an AWS SES email to notify the user about the pending task. 
On successful execution, the state transitions to Finalize state, else  DefaultResponse End state.

Finalize - Just prints the output of the approval state 


Before we try to run this project, we must update the following entries in env.json:
API_ENDPOINT :  Add the url exposed by api gateway from step 3 (lambda-api-gateway)

APPROVAL_ENDPOINT :  Add the url exposed by api gateway for the /approve endpoint from step 2(lambda-approval)

REJECTION_ENDPOINT :  Add the url exposed by api gateway for the /approve endpoint from step 2(lambda-approval)

RECEPIENT_EMAIL : the email address which has been registered with AWS SES. (this will also receive the email from SES)

Commands:
  • npm install --save-dev serverless-step-functions
  • npm install
  • serverless deploy             (this will deploy the project on to aws)
After deployment, please note down the url of the exposed API Gateway method. 

Make a post request to the api gateway url with request body like : { "x": 125, "y": 122 }

5. Verification

After deployment, please note down the url of the exposed API Gateway method. 
On sending the post request, the state machine will be triggered. You can verify the execution status of the workflow, on the AWS console.

The calculate and approval tasks will be executed without human intervention and an email will be sent to the recipient configured in step 4 along with the task token required to approve/ reject the tasks.

At this point the workflow is in a suspended state, awaiting user approval.



To manually approve / reject the execution, check your received email and extract the task token. Please note, a timeout of around 10 mins is set for this task. so you need to approve/ reject the task within 10 mins.
The final execution status on the AWS console will be updated as follows:

6. Logs

The logs for the execution can be viewed using cloud watch. Cloud watch will create log groups for each lambda functions.


Pricing

The pricing will be based on the costs for following resources:
  • S3 bucket - the source code will be placed on S3 buckets. (storage costs)
  • Cost of invocation for the underlying lambda functions
  • Cost of state transitions for the step functions. In our example, we have 6 transitions.
The following link should give you a rough estimation of the costs:


Clean-up

Navigate to each sub folder and run the following command to delete the resource

Commands:
  • serverless remove 
Please note, the serverless framework creates cloud formation. So to ensure that the above command removes all created artifacts, kindly do not make any manual changes on the created awes artifacts.