Don’t use Lambda to move data! API Gateway can help
This article will walk you through on how to use API Gateway to.
API Gateway is the best companion of Lambda, and many people try their first Lambda function with API Gateway. But what many people don’t know is that API Gateway is not just a trigger, it can do more than you may think.
This article will walk you through on how to use API Gateway to:
- Validate user input
- Transform data format
- Integrate with other AWS services
1. Input validation
One of the most crucial security principles is never trusting user input. So you probably have some code validating user input at the beginning of the Lambda function.
Instead of invoking the Lambda function to validate user input and return errors, we can validate the input using API Gateway in the first place and avoid unnecessary Lambda invocation. Here’s how we can do that:
Step 1: Creating the data model
First, we need to define the model of our desired user input so that API Gateway can use it to do the validation.
In AWS console, after you create a REST API, you can see a page called Models.
Model is a JSON schema, which you can define the expected structure of request content. For example, I can define my API endpoint to ingest a JSON containing a string
name ; an integer
id ; and no more extra value, by the following schema:
It’s just a simple example. With JSON Schema, you can define more complex validation rules, like min/max values, enum, pattern matching, nested object.
Step 2: Binding model with the API endpoint
Next, we need to bind the model to the endpoint we want to do the validation. In the Resouces page, click into the endpoint and choose Method Request.
We choose “Validate body, query string parameters, and headers” for Request Validator so that API Gateway validates the request body from the user.
Under the Request Body tab, we add a row with Content Type set to application/json (of course you can use other content types if your API receive different content types). In the Model name column, we choose the model we have created in step 1.
Now, we can test our API. Go back to the Method Execution page and click Test. On this page, we can preview the behaviour of the API without deploying it.
The API returns an HTTP 400 error if the input is not valid
I try using an empty object as input. Because the model states that
id are required, the API immediately returns an HTTP 400 error without invoking downstream processes.
Step 3: Modify the error response
As we can see, the API returns an error message “Invalid request body”. If we input a JSON with
id but in the wrong data type, for example:
"id": "This is a string"
It still returns the same message. It’s not very useful as it doesn’t tell the user which part of the request is invalid.
In Gateway Responses page, we can customise the response for each response type. For our use case, we are going to customise Bad Request Body.
We can customise the response from API Gateway
application/json under Response templates, we can see the default template is:
We need to change it to a more meaningful message. There are many variables we can use to construct the response body. In our case, we are going to use
$context.error.validationErrorString , we change the template into:
After saving the template, we can go back to Method Execution page and do the testing again.
Now, if we input an empty object as input, API Gateway responses an HTTP 400 error with the message
[object has missing required properties (["id","name"])]
If we put a string in id field, it responses with the message
[instance type (string) does not match any allowed primitive type (allowed: ["integer"])]
2. Data Transformation and Service Integration
Many times when people want to provide an AWS service to customers, they don’t directly expose the service API simply because they want to control the request and response format. The first thing in mind must be using Lambda to transform the data format and make the API call to the AWS service.
Example: Getting record from DynamoDB
Querying record from DynamoDB should be one of the most common API use cases. API Gateway can help us translate query string from user input into DynamoDB query. And translate the result into our desire format to the user.
In this example, I am going to create a GET endpoint, taking
id from the query string, then return the related record from DynamoDB.
Step 1: Create an IAM role
We need to create an IAM role for API Gateway to access AWS services. In this example, we are going to create a role granting DynamoDB Query permission.
In the IAM Roles page, we create a new role. For “trusted entity”, we choose API Gateway. Then, we leave all config as the default until the last step. Give the role a name, e.g. DynamoDBQueryForAPIGateway.
Next, we need to detach the default AmazonAPIGatewayPushToCloudWatchLogs policy and add our inline policy.
In the inline policy, we grant DynamoDB Query permission on the table.
Step 2: Require query string parameters from users
Under the Method Request page of the GET method, we choose “Validate query string parameters and headers” for Request Validator.
Under the “URL Query String Parameters” section, we add a row with name
id and mark as required.
Now, the endpoint checks if the user provides parameter
id in the query string, and reject the call if it does not exist.
Make id as a required parameter of the endpoint.
Step 3: Transform user input into DynamoDB API call
Go to Integration Request page, that’s the place we construct the DynamoDB call.
Choose AWS Service for Integration type; DynamoDB for AWS Service; and choose the region of the DynamoDB table for AWS Region.
For execution role, put the ARN of the role we have created in step 1.
For HTTP method and Action, we can check out the API reference. Each AWS service has its API request format. In this case, we are going to use POST Query (API Reference).
We are going to add a new mapping template, with Content-Type set to application/json (We are expecting JSON from user requests).
For Request body passthrough, we choose Never. Any request which is not a JSON will be rejected so that outsiders cannot bypass our mapping and directly access the DynamoDB API.
The most exciting part comes, the template itself. The mapping template in API Gateway uses Apache Velocity Template Language (VTL) format. We can use VTL to construct the API call body, just as we are constructing API call as usual. Combining with some simple logic, like if-condition, for-loop, and the variables provided by API Gateway, we can make DynamoDB API call based on the user input.
In this example, we are going to construct a usual Query call, using
id parameter provided by the user as the primary key condition:
"KeyConditionExpression": "id = :id",
Step 4: Transform query result to the API response
Similar to constructing API call, we can construct the response to our users based on the result from DynamoDB.
In Integration Response page, we can configure the mapping between the response from DynamoDB and the response sending to users. To make it simple, we are going to use the default one.
Under the Mapping Templates section, we add a new template with Content-Type set to application/json.
For the template body, we are going to use a for-loop to list out all the items queried, and show them as a simple JSON object instead of the original format provided by DyanmoDB.
Alternatively, we can use an if condition to check if there is any record returned. We can even override the HTTP response code to 404 if there is no record returned. (AWS documentation of overriding status code)
#set($inputRoot = $input.path('$'))#if($inputRoot.Items.size() == 0)
#set($context.responseOverride.status = 404)
"message":"Record not found"
#foreach($item in $inputRoot.Items)
"id" : $item.id.N,
"name" : "$item.name.S"
Try it out
Now, if we go back to the Test page, we can provide the query string and try out the API we have just created.
If we put
id=1 in the query string (which I had created the record in my DynamoDB table), we can see the API returning the JSON array we have defined, instead of the complex response structure from DynamoDB.
If we put
id=2 (which I don’t have any record with this id), the API returns an HTTP 404 error, along with the error message we have defined in the template.
For many web applications, most of the traffic is read traffic. If we implement them in API Gateway, we can save the cost of those unnecessary Lambda invocations.
Even though the Lambda invocation is inevitable, we can still do the input validation in API Gateway. By guarding the Lambda function from silly mistakes made by the users, we can avoid wasting invocation on doing meaningless things.
You may notice that API Gateway gives you 5000 burst concurrency. However, if every API call ends up in a Lambda function, this concurrency becomes 1000 because Lambda only gives you this amount of concurrency.
By lifting those simple API calls (especially the read traffics) away from Lambda, we can reserve the precious Lambda concurrency to more complex tasks.
AWS DeepRacer League Finalist | AWS Community Builder | Cloud Engineer