Setup CloudWatch Logs Agent on Ubuntu

The AWS CloudWatch Logs Agent can be setup to push logs to the AWS CloudWatch Logs service from any server. In this howto, I'll show you how to set it up on Ubuntu 18.04 LTS, but you should be able to follow similar steps on other versions of Ubuntu or other linux based operating systems. The CloudWatch Logs agent can even be setup to collect EventViewer logs on Windows Servers, but that is beyond the scope of this particular guide

What's also cool is that you can use this technique on any server, you don't have to be running in EC2, you can be using DigitalOcean, Linode, Google Cloud, Azure, etc. The AWS documentation doesn't really make this clear or give you much guidance for running on other providers (for obvious reasons), so here's a simple guide for you to follow.

You should also be able to easily bake this process into a configuration management script or add to a Dockerfile to use in your own Docker images.

Download / Install the Debian Package

AWS publishes the locations of the CloudWatch Logs Agent deb package in their documentation, we are just going to download it with curl and run it:

#download it
curl -o /root/amazon-cloudwatch-agent.deb https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb
#install it
dpkg -i -E /root/amazon-cloudwatch-agent.deb

All commands in this guide are assumed to be run as root or with sudo

Add cwagent User to adm group

Next we are going to modify the linux user account that the installer created cwagent and add it to the adm group, which will give it read permission to many of the default Ubuntu system logs.

usermod -aG adm cwagent

Setup an IAM User Account and Permissions

Now we need to give your Ubuntu server permission to publish its log data to your AWS account's CloudWatch Logs.

If you are running on AWS EC2, you can skip this step, and use the assumed role. You will still need to make sure that the assumed role has IAM permissions

On your Ubuntu server create a file /home/cwagent/.aws/credentials with the following:

[AmazonCloudWatchAgent]
aws_access_key_id = aaaaaaaa
aws_secret_access_key = bbbbbbbb

Next create a file: /home/cwagent/.aws/config with the following:

[AmazonCloudWatchAgent]
output = text
region = us-east-1

Be sure to use the aws_access_key_id and aws_secret_access_key from the new IAM user you created in the AWS console. Also be sure that the region is specified correctly.

Next tell the CloudWatch Logs Agent to look here for credentials by appending to the /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml file:

echo "[credentials]" >> /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
echo '    shared_credential_file = "/home/cwagent/.aws/credentials"' >> /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

Create a Log Group

Take some time to think about how you want to organize your logs. CloudWatch Logs provides a notion of Log Groups and Log Streams, a log group can hold lots of log streams. Some like to create log groups dynamically, others may like all their server logs in the same group. In this tutorial we are going to create a single log group (eg web-server-log-group) that can hold logs for multiple servers by creating log streams like hostname.example.com/syslog

I like this approach because I can then limit the IAM permissions (setup in the next step) to only give access to publish logs and create streams in a single log group.

Grant the IAM User / Role Permission to Publish Logs

Assuming we have a log group named web-server-log-group we will need to attach an IAM policy to the IAM user you created in the previous step. Or if you are running on EC2 you can attach the policy to the assumed role. From the AWS console you can the Add inline policy button, and use something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchLogAgentPutForWebServerLogGroup",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:1234567890:log-group:web-server-log-group",
                "arn:aws:logs:us-east-1:1234567890:log-group:web-server-log-group:*:*"
            ]
        }
    ]
}

Note that the number 1234567890 should be replaced with your AWS account id.

If you want to allow the CloudWatch Logs Agent to publish metrics data (CPU, Memory, Network, Disk) then you will need to attach an additional policy to allow cloudwatch:PutMetricData

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchPutMetricData",
            "Effect": "Allow",
            "Action": "cloudwatch:PutMetricData",
            "Resource": "*"
        }
    ]
}

I need to do some more research to see if the Resource can be further constrained instead of using the wildcard *.

AWS also provides a builtin policy named CloudWatchAgentServerPolicy which you could use, but it is a bit over-permissive in my opinion. It will allow the server that the agent runs on to create new log groups, and publish logs to any log group.

Telling CloudWatch Logs Agent Which Log Files to Collect

The AWS CloudWatch Logs agent has a tool which you can run to create the json configuration file via a wizard:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Or you can just manually create a JSON file and place it here: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json - that file might look like this:

{
	"agent": {
		"metrics_collection_interval": 60,
		"run_as_user": "cwagent",
		"credentials_path":"/home/cwagent/.aws/credentials"
	},
	"logs": {
		"logs_collected": {
			"files": {
				"collect_list": [
					{
						"file_path": "/var/log/syslog",
						"log_group_name": "foundeo-web",
						"log_stream_name": "{hostname}/syslog",
						"timestamp_format" :"%b %d %H:%M:%S"
					}
                                ]
                        }
               }
       }
}

The above will publish the contents of /var/log/syslog from your Ubuntu server to the AWS CloudWatch Logs service. Here is an example of sending nginx logs to CloudWatch Logs.

Enable and Start the CloudWatch Logs Agent service

Our final step is to enable the AWS CloudWatch Logs Ubuntu service (so that it will start after reboot), and then start the service, we can do that with these two commands:

systemctl enable amazon-cloudwatch-agent.service
service amazon-cloudwatch-agent start