Export AWS Cloudwatch Logs to S3

Table of Contents

In this article, we will learn how to export AWS Cloudwatch logs to S3. AWS CloudWatch is a monitoring and management service that provides actionable insights into applications and infrastructure on AWS, hybrid clouds, and on-premises. CloudWatch enables users to view and collect monitoring data for AWS infrastructures in one place. The CloudWatch platform provides data collection, monitoring, automated actions, compliance, and security features.

Amazon S3 is a web-based cloud storage service provided by AWS (Amazon Web Services) that stores files in buckets. Amazon S3 provides durability and security for the files stored within the buckets. There are no limitations on the number of files stored in Amazon S3. The Amazon S3 charges only for the resources you used, there are no hidden charges. Amazon S3 can also be used to trigger different resources like AWS Lambda.

Prerequisites

AWS account
Basic AWS Lambda and AWS CloudWatch concepts

How to export AWS Cloudwatch Logs to S3?

Step 1: Create an S3 Bucket

Go to AWS Console and Log In
In the search bar, type S3
Select the S3 options displayed under Services section

After that AWS S3 page will appear. Then click the Create bucket button on the right side of the page so that we can create a new bucket to which we will import our cloud watch logs.

Now, let’s create an S3 bucket.
Here, enter your desired name and region of the bucket, we will be using cloudwatch-logs-store and ap-south-1 respectively.

Step 2: Create a Lambda Function

Go to AWS Console and Log In
In the search bar, type lambda
Select the Lambda options displayed under Services section

After that AWS Lambda page will appear. Then click the Create function button on the right side of the page.
Now, let’s create a function named exportLogs . Then click Create Function as shown below.

It will take some time to create a lambda function and you will be redirected to the following page.

Then, let’s add a new role to our lambda so that it can access the event bridge features.
To do that click the Configuration tab and click the role provided under the Execution Role section.

Then, you will be redirected to the IAM roles page.
Here, click Add permissions the button and then choose Attach policies option.

Now, let’s search for the S3 and cloudwatch log policies to be attached to the lambda.

Now, let’s modify the index.mjs file of our lambda. So, that it will export the cloudwatch logs to the S3 bucket created in the above steps.

const AWS = require('aws-sdk');
const cloudconfig = {
  apiVersion: '2014-03-28',
  region: 'ap-south-1', // replace with your region
}
const cloudwatchlogs = new AWS.CloudWatchLogs(cloudconfig);
exports.handler =  async (event, context) => {
   const params = {
    destination: 'bucket_name', // replace with your bucket name
    from: new Date().getTime() - 8640000,
    logGroupName: 'log-name', // name of the log group to export
    to: new Date().getTime(),
  };
await cloudwatchlogs.createExportTask(params).promise().then((data) => {
    console.log(data)
    return ({
      statusCode: 200,
        body: data,
    });
  }).catch((err) => {
    console.error(err)
    return ({
      statusCode: 501,
        body: err,
    });
  });
}

Here,

destination: The name of the S3 bucket for the exported log data. The bucket must be in the same AWS region.
from: The start time of the range for the request, expressed as the number of milliseconds after Jan 1, 1970, 00:00:00 UTC. Events with a timestamp earlier than this time are not exported.
logGroupName: The name of the log group.
to: The end time of the range for the request, expressed as the number of milliseconds after Jan 1, 1970, 00:00:00 UTC. Events with a timestamp later than this time are not exported.

After modifying the index.mjs file, execute the lambda by clicking the test button as shown in the image below:

Conclusion

In this article, we have learned how to export AWS Cloudwatch logs to S3 using lambda function. This is one of the many approaches to export aws cloudwatch logs to S3 buckets.