cloudwatchlogs

Table of Contents

1. Overview
2. Module Description - What the module does and why it is useful
3. Setup - The basics of getting started with cloudwatchlogs
   • What cloudwatchlogs affects
   • Setup requirements
   • Beginning with cloudwatchlogs
4. Usage - Configuration options and additional functionality
5. Reference - An under-the-hood peek at what the module is doing and how
6. Limitations - OS compatibility, etc.
7. Development - Guide for contributing to the module

Overview

This module is a fork of this repo that has been fixed to work with hiera. Examples have been adjusted to be hiera yaml examples. Otherwise the functionality is very similar. This module installs, configures and manages the service for the AWS Cloudwatch Logs Agent on Amazon Linux, Ubuntu, Red Hat & CentOS EC2 instances.

Module Description

CloudWatch Logs can be used to monitor your logs for specific phrases, values, or patterns. For example, you could set an alarm on the number of errors that occur in your system logs or view graphs of web request latencies from your application logs. You can view the original log data to see the source of the problem if needed. Log data can be stored and accessed for as long as you need using highly durable, low-cost storage so you don’t have to worry about filling up hard drives.

Setup

What cloudwatchlogs affects

• The awslogs package.
• Configuration files under /etc/awslogs.
• The awslogs service.

Setup Requirements

This module does NOT manage the AWS CLI credentials. As such if you are not using an IAM role (recommended) then you will need some other way of managing the credentials.

This module by Justin Downing is recommended for this purpose.

Beginning with cloudwatchlogs

ALL EXAMPLES ARE FOR HIERA IN YAML

The minimum you need to get this module up and running is (assuming your instance is launched with a suitable IAM role):

classes:
- cloudwatchlogs

cloudwatchlogs:

Usage

On NON Amazon Linux instances you also need to provide a default region:

classes:
- cloudwatchlogs
cloudwatchlogs:
  region: 'eu-west-1'

For each log you want send to Cloudwatch Logs you create a cloudwatchlogs::log resource. This shall be set as its own item. It requires a name for each log to create so that a nested object is created.

A simple example that might be used on the RedHat ::osfamily is:

classes:
- cloudwatchlogs
cloudwatchlogs:
  region: 'eu-west-1'
cloudwatchlogs::log:
  'Messages':
    path: '/var/log/messages'
  'Node':
    path: '/path/to/your/node.log'

See the examples/ directory for further examples.

Reference

cloudwatchlogs

state_file:

Defaults:

• Amazon Linux: /var/lib/awslogs/agent-state
• Other: /var/awslogs/state/agent-state

State file for the awslogs agent.

logging_config_file:

Defaults: /etc/awslogs/awslogs_dot_log.conf

Config file for the awslogs agent logging system (http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/AgentReference.html).

region:

Default: undef

log_level:

Default: undef

The region your EC2 instance is running in.

NOTE: This is required for none Amazon distros.

cloudwatchlogs::log

path

Default: undef

Optional. This is the absolute path to the log file being managed. If not set the name of the resource is used instead (and must be an absolute path if that this situation occurs).

streamname

Default: {instance_id}

The name of the stream in Cloudwatch Logs. This should be a string like all the others. See the ams cloudwatch logs docs for options. One other common option is {hostname}

datetime_format

Default: %b %d %H:%M:%S

Specifies how the timestamp is extracted from logs. See the official docs for further info.

log_group_name

Default: Resource Name

Specifies the destination log group. A log group will be created automatically if it doesn't already exist.

multi_line_start_pattern

Default: undef

Optional. This is a regex string that identifies the start of a log line. See the official docs for further info.

Example

classes:
- cloudwatchlogs

cloudwatchlogs:
  region: 'eu-west-1'

cloudwatchlogs::log:
  'node':
    path: '/path/to/your/logfile.log'
    streamname: '{hostname}'
    datetime_format: '%Y-%m-%dT%H:%M:%S%z'
    log_group_name: 'my-node-project'
  'Messages':
    path: '/var/log/messages'
    streamname: '{hostname}'
    log_group_name: 'system-messages'

Http Proxy Usage

If you have a http_proxy or https_proxy then run the following puppet code after calling cloudwatchlogs to modify the launcher script as a workaround bcause awslogs python code currently doesn't have http_proxy support:

$launcher = "#!/bin/sh
# Version: 1.3.5
echo -n $$ > /var/awslogs/state/awslogs.pid
/usr/bin/env -i AWS_CONFIG_FILE=/var/awslogs/etc/awscli.conf HOME=\$HOME HTTPS_PROXY=${http_proxy} HTTP_PROXY=${http_proxy} NO_PROXY=169.254.169.254  /bin/nice -n 4 /var/awslogs/bin/aws logs push --config-file /var/awslogs/etc/awslogs.conf >> /var/log/awslogs.log 2>&1
"

file { '/var/awslogs/bin/awslogs-agent-launcher.sh':
  ensure  => file,
  owner   => root,
  group   => root,
  mode    => '0755',
  content => $launcher,
  require => Class['cloudwatchlogs'],
}

Limitations

This module is currently only compatible with:

• Amazon Linux AMI 2014.09 or later.
• Ubuntu
• Red Hat
• CentOS

More information on support as well as information in general about the set-up of the Cloudwatch Logs agent can be found here.

Development

Contributions are welcome via pull requests. To test and build: Download the Puppet Development Kit from [https://puppet.com/download-puppet-development-kit] To build run pdk build from terminal in project folder To run lint and validator pdk validate To run unit tests pdk test unit

Contributors

Authors:

• Douglas Sappet

Original Repo Authors:

• Danny Roberts
• Russ McKendrick

All other contributions: https://github.com/dsappet/puppet-cloudwatchlogs/graphs/contributors

2016-12-04 - Release 3.2.1

Summary

Small bugfix release.

Bugfixes

• Force creation of symlink at /var/awslogs/etc/conf to prevent issues during upgrades.
• Add appropriate requires based on OS for the main config.

2016-07-12 - Release 2.3.0

Summary

Large feature and bug fix release.

Bugfixes

• Ensure config is concated before installation.
• Various linting & test fixes.

Features

• Support for setting cloudwatchlogs::logs via a Hiera hash.
• Support setting the region in the main awscli config file.
• Support for setting the log_level for logs.
• Add support for cloudwatchlogs::compartment_log.

2016-03-22 - Release 2.2.0

Summary

Support additional CloudWatch Logs functionality & small bigfixes.

Bugfixes

• Ensure wget is installed.

Features

• Now supports the multi_line_start_pattern for each log.
• Logs can now be defined as a hash as a part fo the main class.

2015-09-15 - Release 2.1.0

Summary

Now able to additionally set an additional config line for logs.

Features

• You can now specify the log_group_name explicitly, otherwise it defaults to the resource name as per the previous behaviour.

2015-06-09 - Release 2.0.0

Summary

Major release switching to using puppetlabs/concat to build config files and various other breaking changes.

Features

• Moved to using puppetlabs/concat for each log file entry. This allows use of more customizable entries more easily down the road.
• Removed management of AWS CLI keys, this should be done by mdoules designed for this purpose.
• Documented usage of using a http_proxy on instances that might require it.
• Provided examples of IAM role for Cloudwatch Logs & of various ways to specify log entries.

2015-06-04 - Release 1.1.2

Summary

Small bugfix release.

Bugfixes

• Fixed default stream name.
• Only install wget package if not defined else where in catalogue.

2015-05-26 - Release 1.1.1

Summary

Minor bugfix release.

Bugfixes

• Fix metadata requirement for puppetlabs/stdlib.
• Fix streamname variable format.

2015-04-30 - Release 1.1.0

Summary

Small feature release supporting slightly more fine grained log config.

Features

• Make credentials optional, creating only if info is provided. Otherwise, assume an IAM role is present to use.
• Add streamname variable.

2015-04-18 - Release 1.0.0

Summary

Initial release.