In this article, we will learn how to perform pagination in DynamoDB using node js. DynamoDB is a fully managed proprietary NoSQL database service that supports key-value and document data structures. in the DynamoDB table, each item is uniquely identified by a primary key. DynamoDB supports two types of primary keys: a simple primary key made up of just a partition key, and a composite primary key made up of a partition key and a sort key.
Pagination is a process of splitting a database’s records into manageable chunks or pages. DynamoDB supports storing large amounts of data, but it has limits on the number of records we can retrieve – 1MB. DynamoDB returns only the number of records below 1MB on a single request. We can perform pagination in DynamoDB using the LastEvaluatedKey
and ExclusiveStartKey
, which supports the on-demand loading of pages in your application. LastEvaluatedKey
is the primary Key of the record where a query or scan operation is stopped and ExclusiveStartKey
is the primary key of the first item that this operation will evaluate.
Prerequisites
How to perform pagination in DynamoDB using node js?
Project Setup
Open the terminal and create a new directory named dynamodb-pagination
and switch to that directory using the following command.
mkdir dynamodb-pagination
cd dynamodb-pagination
Inside the dynamdb-pagination
directory, let’s initialize a node project using npm init
command.
After initializing a pacakge.json file will be generated like the image provided below.
Now, let’s install the required dependencies required to export the data in an excel file.
npm install express aws-sdk uuid
This will install express
,aws-sdk
and uuid
modules that will be used for routing, Dynamodb functionality of AWS will be used with the help of aws-sdk
and unique id respectively.
Next, let’s create a index.js
file where we will create a different route through which the Dynamodb operations will be performed.
#index.js
const express = require('express');
const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
app.get("/create", async (req, res) => {
// create dynamodb sample data
});
app.get("/sample", async (req, res) => {
// query dynamodb and pagination logic
});
app.listen(3000, function(){
console.log("listening on port 3000");
});
The above code will initialize the express framework, set up the server at port 3000, and create two routes i.e. /create
and /sample
which will create sample data in Dynamodb and query that sample data along with pagination.
Create sample DynamoDB data
In this step, let’s create a file where we will write a logic to create some random sample DynamoDB data on which we will be performing DynamoDB pagination. Let the name of the file be dynamo.js
which will be located inside root
folder.
#dynamo.js
var AWS = require("aws-sdk");
const { v4: uuidv4 } = require('uuid');
var dynamodb = new AWS.DynamoDB.DocumentClient()
// declare all characters
const characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
function generateString(length) {
let result = '';
const charactersLength = characters.length;
for (let i = 0; i < length; i++) {
result += characters.charAt(Math.floor(Math.random() * charactersLength));
}
return result;
}
async function create() {
var id = uuidv4(); //generate unique id
var year = Math.floor(1000 + Math.random() * 9000); // generate random 4 digit number for year
var title = generateString(5); // generate random string
var params = {
TableName: 'Movie',
Item: {
"pk": "MOVIES",
"id": id,
"year": year,
"title": title,
},
};
await dynamodb.put(params).promise();
}
module.exports = {
create,
};
Here, we initialize AWS to access the Dynamodb service. Here, pk
is used as the partition key and id
as the range key, and year
and title
as normal columns. We have used the UUID package to generate a random unique id for the primary key. The generateString
the function is used to generate a random string for title
the column and create
the function is used to create dummy records in the DynamoDB table.
Now, let’s update our /create
route inside the index.js
file, so that when accessing the /create
the route, 10 random records will be created.
# index.js
app.get("/create", async (req, res) => {
// Create 10 random data at once
for (let i = 0; i < 10; i++) {
await create();
}
return res.status(200).json({
message: "Executed successfully",
data: ""
});
});
DynamoDB Pagination
In this step, let’s add a new function named getData
inside our dynamo.js
file which will query the DynamoDB records with pagination functionality and update the index.js
file to access the getData
function while using the /sample
route.
#index.js
app.get("/sample", async (req, res) => {
// query dynamodb and pagination logic
const lastEvaluatedKey = req.query.lastEvaluatedKey;
const result = await getData(lastEvaluatedKey);
return res.status(200).json({
message: "Executed successfully",
data: result
});
});
#dynamo.js
var AWS = require("aws-sdk");
const { v4: uuidv4 } = require('uuid');
var dynamodb = new AWS.DynamoDB.DocumentClient()
// declare all characters
const characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
function generateString(length) {
let result = '';
const charactersLength = characters.length;
for (let i = 0; i < length; i++) {
result += characters.charAt(Math.floor(Math.random() * charactersLength));
}
return result;
}
async function create() {
var id = uuidv4(); //generate unique id
var year = Math.floor(1000 + Math.random() * 9000); // generate random 4 digit number for year
var title = generateString(5); // generate random string
var params = {
TableName: 'Movie',
Item: {
"pk": "MOVIES",
"id": id,
"year": year,
"title": title,
},
};
await dynamodb.put(params).promise();
}
async function getData(lastEvaluatedKey) {
const params = {
TableName: 'Movie',
Limit: 4,
KeyConditionExpression: "#pk = :pk",
ExpressionAttributeNames: {
"#pk": "pk",
},
ExpressionAttributeValues: {
":pk": "MOVIES",
},
};
if (lastEvaluatedKey) {
params["ExclusiveStartKey"] = {
pk: "MOVIES",
id: lastEvaluatedKey,
};
}
return await dynamodb.query(params).promise();
};
module.exports = {
create,
getData
};
Here, the getData
the function will query the DynamoDB table and return the 4 records at a time. If the table consists of more than 4 records it will provide the LastEvaluatedKey
object in its response like the below:
{
"message": "Executed successfully",
"data": {
"Items": [
],
"Count": 4,
"ScannedCount": 4,
"LastEvaluatedKey": {
"pk": "MOVIES",
"id": "3e843075-ed20-4201-958a-7b48d62b8fad"
}
}
}
Then, in the next request, the user should provide the id
value from LastEvaluatedKey
in the query parameter while triggering the API. If the response does not contain the LastEvaluatedKey
object, then it denotes that no more records are available. The next request should be performed like the below for pagination:
Conclusion
In this article, we’ve learned how to implement pagination in DynamoDB using node js. If you want to learn pagination in elastic search using node js, look into this article.