In this article, we will learn how to perform pagination in DynamoDB using node js. DynamoDB is a fully managed proprietary NoSQL database service that supports key-value and document data structures. in the DynamoDB table, each item is uniquely identified by a primary key. DynamoDB supports two types of primary keys: a simple primary key made up of just a partition key, and a composite primary key made up of a partition key and a sort key.
Pagination is a process of splitting a database’s records into manageable chunks or pages. DynamoDB supports storing large amounts of data, but it has limits on the number of records we can retrieve – 1MB. DynamoDB returns only the number of records below 1MB on a single request. We can perform pagination in DynamoDB using the LastEvaluatedKey and ExclusiveStartKey, which supports the on-demand loading of pages in your application. LastEvaluatedKey is the primary Key of the record where a query or scan operation is stopped and ExclusiveStartKey is the primary key of the first item that this operation will evaluate.
Prerequisites
How to perform pagination in DynamoDB using node js?
Project Setup
Open the terminal and create a new directory named dynamodb-pagination and switch to that directory using the following command.
mkdir dynamodb-pagination
cd dynamodb-paginationInside the dynamdb-pagination directory, let’s initialize a node project using npm init command.
After initializing a pacakge.json file will be generated like the image provided below.
Now, let’s install the required dependencies required to export the data in an excel file.
npm install express aws-sdk uuidThis will install express ,aws-sdk and uuidmodules that will be used for routing, Dynamodb functionality of AWS will be used with the help of aws-sdk and unique id respectively.
Next, let’s create a index.js file where we will create a different route through which the Dynamodb operations will be performed.
#index.js
const express = require('express');
const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
app.get("/create", async (req, res) => {
// create dynamodb sample data
});
app.get("/sample", async (req, res) => {
// query dynamodb and pagination logic
});
app.listen(3000, function(){
console.log("listening on port 3000");
});The above code will initialize the express framework, set up the server at port 3000, and create two routes i.e. /create and /sample which will create sample data in Dynamodb and query that sample data along with pagination.
Create sample DynamoDB data
In this step, let’s create a file where we will write a logic to create some random sample DynamoDB data on which we will be performing DynamoDB pagination. Let the name of the file be dynamo.js which will be located inside root folder.
#dynamo.js
var AWS = require("aws-sdk");
const { v4: uuidv4 } = require('uuid');
var dynamodb = new AWS.DynamoDB.DocumentClient()
// declare all characters
const characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
function generateString(length) {
let result = '';
const charactersLength = characters.length;
for (let i = 0; i < length; i++) {
result += characters.charAt(Math.floor(Math.random() * charactersLength));
}
return result;
}
async function create() {
var id = uuidv4(); //generate unique id
var year = Math.floor(1000 + Math.random() * 9000); // generate random 4 digit number for year
var title = generateString(5); // generate random string
var params = {
TableName: 'Movie',
Item: {
"pk": "MOVIES",
"id": id,
"year": year,
"title": title,
},
};
await dynamodb.put(params).promise();
}
module.exports = {
create,
};Here, we initialize AWS to access the Dynamodb service. Here, pk is used as the partition key and id as the range key, and year and title as normal columns. We have used the UUID package to generate a random unique id for the primary key. The generateString the function is used to generate a random string for title the column and create the function is used to create dummy records in the DynamoDB table.
Now, let’s update our /create route inside the index.js file, so that when accessing the /create the route, 10 random records will be created.
# index.js
app.get("/create", async (req, res) => {
// Create 10 random data at once
for (let i = 0; i < 10; i++) {
await create();
}
return res.status(200).json({
message: "Executed successfully",
data: ""
});
});DynamoDB Pagination
In this step, let’s add a new function named getData inside our dynamo.js file which will query the DynamoDB records with pagination functionality and update the index.js file to access the getData function while using the /sample route.
#index.js
app.get("/sample", async (req, res) => {
// query dynamodb and pagination logic
const lastEvaluatedKey = req.query.lastEvaluatedKey;
const result = await getData(lastEvaluatedKey);
return res.status(200).json({
message: "Executed successfully",
data: result
});
});#dynamo.js
var AWS = require("aws-sdk");
const { v4: uuidv4 } = require('uuid');
var dynamodb = new AWS.DynamoDB.DocumentClient()
// declare all characters
const characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
function generateString(length) {
let result = '';
const charactersLength = characters.length;
for (let i = 0; i < length; i++) {
result += characters.charAt(Math.floor(Math.random() * charactersLength));
}
return result;
}
async function create() {
var id = uuidv4(); //generate unique id
var year = Math.floor(1000 + Math.random() * 9000); // generate random 4 digit number for year
var title = generateString(5); // generate random string
var params = {
TableName: 'Movie',
Item: {
"pk": "MOVIES",
"id": id,
"year": year,
"title": title,
},
};
await dynamodb.put(params).promise();
}
async function getData(lastEvaluatedKey) {
const params = {
TableName: 'Movie',
Limit: 4,
KeyConditionExpression: "#pk = :pk",
ExpressionAttributeNames: {
"#pk": "pk",
},
ExpressionAttributeValues: {
":pk": "MOVIES",
},
};
if (lastEvaluatedKey) {
params["ExclusiveStartKey"] = {
pk: "MOVIES",
id: lastEvaluatedKey,
};
}
return await dynamodb.query(params).promise();
};
module.exports = {
create,
getData
};Here, the getData the function will query the DynamoDB table and return the 4 records at a time. If the table consists of more than 4 records it will provide the LastEvaluatedKey object in its response like the below:
{
"message": "Executed successfully",
"data": {
"Items": [
],
"Count": 4,
"ScannedCount": 4,
"LastEvaluatedKey": {
"pk": "MOVIES",
"id": "3e843075-ed20-4201-958a-7b48d62b8fad"
}
}
}Then, in the next request, the user should provide the id value from LastEvaluatedKey in the query parameter while triggering the API. If the response does not contain the LastEvaluatedKey object, then it denotes that no more records are available. The next request should be performed like the below for pagination:
Conclusion
In this article, we’ve learned how to implement pagination in DynamoDB using node js. If you want to learn pagination in elastic search using node js, look into this article.