Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add idempotent example aws node express #662

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions idempotent-aws-node-express-dynamodb-api/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
node_modules
.serverless
181 changes: 181 additions & 0 deletions idempotent-aws-node-express-dynamodb-api/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
<!--
title: 'Serverless Framework Node Express API service backed by DynamoDB on AWS with Idempotence Guaranteed'
description: 'This template demonstrates how to develop and deploy a simple Node Express API service backed by DynamoDB running on AWS Lambda using the traditional Serverless Framework.'
layout: Doc
framework: v2
platform: AWS
language: nodeJS
priority: 1
authorLink: 'https://github.com/serverless'
authorName: 'Serverless, inc.'
authorAvatar: 'https://avatars1.githubusercontent.com/u/13742415?s=200&v=4'
-->

# Serverless Framework Node Express API on AWS with Idempotence Guarantee

This template demonstrates how to add idempotence in a simple Node Express API service, backed by DynamoDB database, running on AWS Lambda using the traditional Serverless Framework. It is based on the example of [Serverless Framework Node Express API on AWS](../aws-node-express-dynamodb-api/README.md).

## What is idempotence

Idempotence means multiple invocations of a function have the same side-effect as one invocation.

## Why we need idempotence

AWS Lambda uses retry to perform fault tolerance.
When your function fails because of out of memory or some other reasons, it will be directly retried until it finishes successfully.
For serverless functions with side-effect, retry may cause data inconsistency.
For example, retrying a function purchasing a product may cause multiple deduction of money.
Therefore, AWS Lambda requires programmers to write [idempotent function](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-idempotent/).

## Anatomy of the template

This template configures a single function, `api`, which is responsible for handling all incoming requests thanks to the `httpApi` event. To learn more about `httpApi` event configuration options, please refer to [httpApi event docs](https://www.serverless.com/framework/docs/providers/aws/events/http-api/). As the event is configured in a way to accept all incoming requests, `express` framework is responsible for routing and handling requests internally. Implementation takes advantage of `serverless-http` package, which allows you to wrap existing `express` applications. To learn more about `serverless-http`, please refer to corresponding [GitHub repository](https://github.com/dougmoscrop/serverless-http). Additionally, it also handles provisioning of a DynamoDB database that is used for storing data about users. The `express` application exposes two endpoints, `POST /users` and `GET /user/{userId}`, which allow to create and retrieve users.

## How to guarantee idempotence

The side effect of the function is writing the username.
AWS DynamoDB provides an [idempotent transactional write API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html), which writes name for only once and does nothing on retry.
It needs an argument `clientRequestToken` to check whether the current transactional write is a retry.
The `clientRequestToken` should be a universally unique identifier.
Then we use [`awsRequestId`](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html) to be the `clientRequestToken`.
This is a unique identifier provided by AWS Lambda.
When Lambda retries a function, it will use the same `awsRequestId` as that in the first invocation.

## Usage

### Deployment

Install dependencies with:

```
npm install
```

and then deploy with:

```
serverless deploy
```

After running deploy, you should see output similar to:

```bash
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
........
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service aws-node-express-dynamodb-api.zip file to S3 (718.53 KB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
....................................
Serverless: Stack update finished...
Service Information
service: aws-node-express-dynamodb-api
stage: dev
region: us-east-1
stack: aws-node-express-dynamodb-api-dev
resources: 13
api keys:
None
endpoints:
ANY - https://xxxxxxx.execute-api.us-east-1.amazonaws.com/
functions:
api: aws-node-express-dynamodb-api-dev-api
layers:
None
```

_Note_: In current form, after deployment, your API is public and can be invoked by anyone. For production deployments, you might want to configure an authorizer. For details on how to do that, refer to [`httpApi` event docs](https://www.serverless.com/framework/docs/providers/aws/events/http-api/). Additionally, in current configuration, the DynamoDB table will be removed when running `serverless remove`. To retain the DynamoDB table even after removal of the stack, add `DeletionPolicy: Retain` to its resource definition.

### Invocation

After successful deployment, you can create a new user by calling the corresponding endpoint:

```bash
curl --request POST 'https://xxxxxx.execute-api.us-east-1.amazonaws.com/users' --header 'Content-Type: application/json' --data-raw '{"name": "John", "userId": "someUserId"}'
```

Which should result in the following response:

```bash
{"userId":"someUserId","name":"John"}
```

You can later retrieve the user by `userId` by calling the following endpoint:

```bash
curl https://xxxxxxx.execute-api.us-east-1.amazonaws.com/users/someUserId
```

Which should result in the following response:

```bash
{"userId":"someUserId","name":"John"}
```

If you try to retrieve user that does not exist, you should receive the following response:

```bash
{"error":"Could not find user with provided \"userId\""}
```

### Local development

It is also possible to emulate DynamoDB, API Gateway and Lambda locally using the `serverless-dynamodb-local` and `serverless-offline` plugins. In order to do that, run:

```bash
serverless plugin install -n serverless-dynamodb-local
serverless plugin install -n serverless-offline
```

It will add both plugins to `devDependencies` in `package.json` file as well as will add it to `plugins` in `serverless.yml`. Make sure that `serverless-offline` is listed as last plugin in `plugins` section:

```
plugins:
- serverless-dynamodb-local
- serverless-offline
```

You should also add the following config to `custom` section in `serverless.yml`:

```
custom:
(...)
dynamodb:
start:
migrate: true
stages:
- dev
```

Additionally, we need to reconfigure `AWS.DynamoDB.DocumentClient` to connect to our local instance of DynamoDB. We can take advantage of `IS_OFFLINE` environment variable set by `serverless-offline` plugin and replace:

```javascript
const dynamoDbClient = new AWS.DynamoDB.DocumentClient();
```

with the following:

```javascript
const dynamoDbClientParams = {};
if (process.env.IS_OFFLINE) {
dynamoDbClientParams.region = 'localhost'
dynamoDbClientParams.endpoint = 'http://localhost:8000'
}
const dynamoDbClient = new AWS.DynamoDB.DocumentClient(dynamoDbClientParams);
```

After that, running the following command with start both local API Gateway emulator as well as local instance of emulated DynamoDB:

```bash
serverless offline start
```

To learn more about the capabilities of `serverless-offline` and `serverless-dynamodb-local`, please refer to their corresponding GitHub repositories:
- https://github.com/dherault/serverless-offline
- https://github.com/99x/serverless-dynamodb-local
80 changes: 80 additions & 0 deletions idempotent-aws-node-express-dynamodb-api/handler.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
const AWS = require("aws-sdk");
const express = require("express");
const serverless = require("serverless-http");

const app = express();

const USERS_TABLE = process.env.USERS_TABLE;
const dynamoDbClient = new AWS.DynamoDB.DocumentClient();

app.use(express.json());

app.get("/users/:userId", async function (req, res) {
const params = {
TableName: USERS_TABLE,
Key: {
userId: req.params.userId,
},
};

try {
const { Item } = await dynamoDbClient.get(params).promise();
if (Item) {
const { userId, name } = Item;
res.json({ userId, name });
} else {
res
.status(404)
.json({ error: 'Could not find user with provided "userId"' });
}
} catch (error) {
console.log(error);
res.status(500).json({ error: "Could not retreive user" });
}
});

app.post("/users", async function (req, res) {
const { userId, name } = req.body;
if (typeof userId !== "string") {
res.status(400).json({ error: '"userId" must be a string' });
} else if (typeof name !== "string") {
res.status(400).json({ error: '"name" must be a string' });
}

const params = {
ClientRequestToken: req.context.awsRequestId,
TransactItems: [
{
Update: {
TableName: USERS_TABLE,
Key: { userId: userId },
UpdateExpression: 'set #a = :v',
ExpressionAttributeNames: {'#a' : 'name'},
ExpressionAttributeValues: {
':v': name
}
}
}
]
};
try {
await dynamoDbClient.transactWrite(params).promise();
res.json({ userId, name });
} catch (error) {
console.log(error);
res.status(500).json({ error: "Could not create user" });
}
});

app.use((req, res, next) => {
return res.status(404).json({
error: "Not Found",
});
});


module.exports.handler = serverless(app,{
request: function(req, _event, context) {
req.context = context;
}
});
9 changes: 9 additions & 0 deletions idempotent-aws-node-express-dynamodb-api/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"name": "idempotent-aws-node-express-dynamodb-api",
"version": "1.0.0",
"description": "",
"dependencies": {
"express": "^4.17.1",
"serverless-http": "^2.7.0"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
name: aws-node-express-dynamodb-api
org: serverlessinc
description: Deploys a Node Express API service backed by DynamoDB with Serverless Framework
keywords: aws, serverless, faas, lambda, node, express, dynamodb
repo: https://github.com/serverless/examples/aws-node-express-dynamodb-api
license: MIT
45 changes: 45 additions & 0 deletions idempotent-aws-node-express-dynamodb-api/serverless.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
service: aws-node-express-dynamodb-api
frameworkVersion: '2'

custom:
tableName: 'users-table-${sls:stage}'

provider:
name: aws
runtime: nodejs12.x
lambdaHashingVersion: '20201221'
iam:
role:
statements:
- Effect: Allow
Action:
- dynamodb:Query
- dynamodb:Scan
- dynamodb:GetItem
- dynamodb:PutItem
- dynamodb:UpdateItem
- dynamodb:DeleteItem
Resource:
- Fn::GetAtt: [ UsersTable, Arn ]
environment:
USERS_TABLE: ${self:custom.tableName}

functions:
api:
handler: handler.handler
events:
- httpApi: '*'

resources:
Resources:
UsersTable:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: userId
AttributeType: S
KeySchema:
- AttributeName: userId
KeyType: HASH
BillingMode: PAY_PER_REQUEST
TableName: ${self:custom.tableName}