Sending nginx access logs to CloudWatch Logs Agent

Recently I wrote about how to setup the CloudWatch Logs Agent to run on Ubuntu 18.04 . In that entry I setup the agent to push the syslog log file, /var/log/syslog to CloudWatch Logs. You will want to go through that first, and then come back here, or if you are not using Ubuntu you will want to make sure you check the AWS docs for installing the CloudWatch Logs Agent on the OS you are using.

In this entry, I'll show you how to push the nginx access log and the nginx error logs to CloudWatch Logs using the AWS CloudWatch Logs Agent.

Let's assume we have two nginx log files we want the agent to consume: /var/log/nginx/access.log and /var/log/nginx/error.log you can add as many nginx log files as you want.

The AWS CloudWatch Logs Agent gets its configuration from the amazon-cloudwatch-agent.json file, which on Ubuntu is located here: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Assuming you already have a file there you just need to add the following under the collect_list array:

{
	"file_path": "/var/log/nginx/access.log",
	"log_group_name": "web-server-log-group",
	"log_stream_name": "{hostname}/access.log",
	"timestamp_format" :"[%d/%b/%Y:%H:%M:%S %z]"
},
{
	"file_path": "/var/log/nginx/error.log",
	"log_group_name": "web-server-log-group",
	"log_stream_name": "{hostname}/error.log",
	"timestamp_format" :"[%d/%b/%Y:%H:%M:%S %z]"
}

The key here is that the timestamp_format matches the format found in the nginx log file, if you are using the default logging settings for nginx on Ubuntu you should be golden.

You will also want to make sure that the log_group_name matches a log group that the CloudWatch Logs Agent identity has IAM permission to create log streams logs:CreateLogStream, describe log streams logs:DescribeLogStreams, and put log events logs:PutLogEvents

After you have updated the amazon-cloudwatch-agent.json file you will need to restart the agent service, eg:

service amazon-cloudwatch-agent restart

You should see your nginx logs in CloudWatch Logs shortly.