Reference

Table of Contents

Classes

• fluentbit: Installs and configures fluentbit
• fluentbit::config: configures the main fluentbit main config
• fluentbit::install: Installs fluentbit package
• fluentbit::repo: configures the fluentbit repo
• fluentbit::repo::debian: A short summary of the purpose of this class
• fluentbit::service: Manages the td-agent-bit service

Defined types

• fluentbit::filter::modify: Filter to modify records using rules and conditions
• fluentbit::input::forward: Inputs / listen and forward messages
• fluentbit::input::syslog: Ingest logs from (r)syslog
• fluentbit::input::tail: Input plugin to monitor one or several text files.
• fluentbit::output::es: Plugin to output logs to a configured elasticsearch instance
• fluentbit::output::http: Manage the configuration of a HTTP output plugin.

Classes

fluentbit

Installs and configures fluentbit

• See also https://docs.fluentbit.io/manual/

Examples

include fluentbit

Parameters

The following parameters are available in the fluentbit class.

manage_package_repo

Data type: Boolean

Installs the package repositories

service_name

Data type: String[1]

the td-agent-bit service name

input_plugins

Data type: Array[Fluentbit::Plugin]

Hash of the INPUT plugins to be configured

output_plugins

Data type: Array[Fluentbit::Plugin]

Hash of the OUTPUT plugins to be configured

filter_plugins

Data type: Array[Fluentbit::Plugin]

Hash of the filter to be configured

manage_package_repo

Installs the package repositories

repo_key_fingerprint

Data type: String[1]

GPG key identifier of the repository

repo_key_location

Data type: Stdlib::HTTPUrl

GPG key location

package_ensure

Data type: String[1]

Whether to install the Fluentbit package, and what version to install. Values: 'present', 'latest', or a specific version number. Default value: 'present'.

package_name

Data type: String[1]

Specifies the Fluentbit package to manage. Default value: 'td-agent-bit'

manage_service

Data type: Boolean

Whether to manage the service at all. Default value: true

service_enable

Data type: Boolean

Whether to enable the fluentbit service at boot. Default value: true.

service_ensure

Data type: Stdlib::Ensure::Service

Whether the fluentbit service should be running. Default value: 'running'.

service_manage

Whether to manage the fluentbit service. Default value: true.

service_hasstatus

Whether the service has a functional status command. Default value: true.

service_hasrestart

Whether the service has a restart command. Default value: true.

manage_config_dir

Data type: Boolean

Whether to manage the configuration directory. When enabled, will remove all unmanaged files from the directory the configuration resides in. Default value: true

config_file

Data type: Stdlib::Absolutepath

Path of the daemon configuration.

config_file_mode

Data type: Stdlib::Filemode

File mode to apply to the daemon configuration file

storage_path

Data type: Optional[Stdlib::Absolutepath]

Set an optional location in the file system to store streams and chunks of data. If this parameter is not set, Input plugins can only use in-memory buffering.

storage_sync

Data type: Optional[Enum['normal', 'full']]

Configure the synchronization mode used to store the data into the file system. It can take the values normal or full. Default value: 'normal'

storage_checksum

Data type: Boolean

Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. Default value: false

storage_backlog_mem_limit

Data type: Optional[String[1]]

If storage.path is set, Fluent Bit will look for data chunks that were not delivered and are still in the storage layer, these are called backlog data. This option configure a hint of maximum value of memory to use when processing these records. Default value: 5M

manage_plugins_file

Data type: Boolean

Whether to manage the enabled external plugins

plugins_file

Data type: Stdlib::Absolutepath

A plugins configuration file allows to define paths for external plugins.

plugins

Data type: Array[Stdlib::Absolutepath]

List of external plugin objects to enable

manage_streams_file

Data type: Boolean

Whether to manage the stream processing configuration

streams_file

Data type: Stdlib::Absolutepath

Path for the Stream Processor configuration file.

streams

Data type: Array[Fluentbit::Stream]

Stream processing tasks

manage_parsers_file

Data type: Boolean

Whether to manage the parser definitions

parsers_file

Data type: Stdlib::Absolutepath

Path for a parsers configuration file. Multiple Parsers_File entries can be used.

parsers

Data type: Array[Fluentbit::Parser]

List of parser definitions. The default value consists of all the available definitions provided by the upstream project as of version 1.3

flush

Data type: Integer

Set the flush time in seconds. Everytime it timeouts, the engine will flush the records to the output plugin.

daemon

Data type: Boolean

Boolean value to set if Fluent Bit should run as a Daemon (background) or not. Allowed values are: yes, no, on and off.

log_file

Data type: Optional[Stdlib::Absolutepath]

Absolute path for an optional log file.

log_level

Data type: Enum['error', 'warning', 'info', 'debug', 'trace']

Set the logging verbosity level. Values are: error, info, debug and trace. Values are accumulative, e.g: if 'debug' is set, it will include error, info and debug. Note that trace mode is only available if Fluent Bit was built with the WITH_TRACE option enabled.

http_server

Data type: Boolean

Enable built-in HTTP Server

http_listen

Data type: Stdlib::IP::Address::Nosubnet

Set listening interface for HTTP Server when it's enabled

http_port

Data type: Stdlib::Port

Set TCP Port for the HTTP Server

coro_stack_size

Data type: Integer

Set the coroutines stack size in bytes. The value must be greater than the page size of the running system.

variables

Data type: Hash

macro definitions to use in the configuration file the will be registered using the @SET command.

service_has_status

Data type: Boolean

service_has_restart

Data type: Boolean

fluentbit::config

Includes all [input] and [output] configs. (@include) Sets global variables (@set) Configures global [service] section

fluentbit::install

Installs fluentbit package

fluentbit::repo

configures the fluentbit repo

fluentbit::repo::debian

A description of what this class does

Examples

include fluentbit::repo::debian

fluentbit::service

Manages the td-agent-bit service

Defined types

fluentbit::filter::modify

Filter to modify records using rules and conditions

Examples

fluentbit::filter::modify { 'namevar': }

Parameters

The following parameters are available in the fluentbit::filter::modify defined type.

configfile

Data type: String

Path to the filter configfile. Naming should be filter_*.conf to make sure it's getting included by the main config.

Default value: '/etc/td-agent-bit/filter_modify.conf'

set

Data type: Optional

Add a key/value pair with key KEY and value VALUE. If KEY already exists, this field is overwritten

Default value: undef

add

Data type: Optional

Add a key/value pair with key KEY and value VALUE if KEY does not exist

Default value: undef

remove

Data type: Optional

Remove a key/value pair with key KEY if it exists

Default value: undef

remove_wildcard

Data type: Optional

Remove all key/value pairs with key matching wildcard KEY

Default value: undef

remove_regex

Data type: Optional

Remove all key/value pairs with key matching regexp KEY

Default value: undef

rename

Data type: Optional

Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists AND RENAMED_KEY does not exist

Default value: undef

hard_rename

Data type: Optional

Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists. If RENAMED_KEY already exists, this field is overwritten

Default value: undef

copy

Data type: Optional

Copy a key/value pair with key KEY to COPIED_KEY if KEY exists AND COPIED_KEY does not exist

Default value: undef

hard_copy

Data type: Optional

Copy a key/value pair with key KEY to COPIED_KEY if KEY exists. If COPIED_KEY already exists, this field is overwritten

Default value: undef

match

Data type: String

Default value: '*'

fluentbit::input::forward

Forward is the protocol used by Fluent Bit and Fluentd to route messages between peers. This plugin implements the input service to listen for Forward messages.

Parameters

The following parameters are available in the fluentbit::input::forward defined type.

configfile

Data type: String

Path to the input configfile. Naming should be input_*.conf to make sure it's getting included by the main config.

Default value: "/etc/td-agent-bit/inputforward${name}.conf"

listen

Data type: Stdlib::IP::Address::Nosubnet

Listener network interface

Default value: '0.0.0.0'

port

Data type: Stdlib::Port

TCP port to listen for incoming connections.

Default value: 24224

buffer_max_size

Data type: Optional[String]

Specify the maximum buffer memory size used to receive a Forward message. The value must be according to the Unit Size specification.

Default value: undef

buffer_chunk_size

Data type: Optional[String]

By default the buffer to store the incoming Forward messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by Buffer_Chunk_Size. The value must be according to the Unit Size specification.

Default value: undef

fluentbit::input::syslog

Setups config file for fluentbit and configures listen on rsyslog. Reloads services if something changes.

Parameters

The following parameters are available in the fluentbit::input::syslog defined type.

mode

Data type: String

Defines transport protocol mode: unix_udp (UDP over Unix socket), unix_tcp (TCP over Unix socket), tcp or udp

Default value: 'unix_udp'

listen

Data type: Optional[String]

If Mode is set to tcp, specify the network interface to bind.

Default value: $mode

port

Data type: Optional[String]

If Mode is set to tcp, specify the TCP port to listen for incoming connections.

Default value: $mode

path

Data type: Optional[String]

If Mode is set to unix_tcp or unix_udp, set the absolute path to the Unix socket file.

Default value: $mode

parser

Data type: String

Specify an alternative parser for the message. By default, the plugin uses the parser syslog-rfc3164. If your syslog messages have fractional seconds set this Parser value to syslog-rfc5424 instead.

Default value: 'syslog-rfc3164'

rsyslog_config

Data type: String

Specify the path to the rsyslog config file for fluentbit to enable listening.

Default value: '/etc/rsyslog.d/60-fluent-bit.conf'

configfile

Data type: String

Path to the input configfile. Naming should be input_*.conf to make sure it's getting included by the main config.

Default value: '/etc/td-agent-bit/input_syslog.conf'

fluentbit::input::tail

The tail input plugin allows to monitor one or several text files. It has a similar behavior like tail -f shell command.

Parameters

The following parameters are available in the fluentbit::input::tail defined type.

configfile

Data type: Stdlib::Absolutepath

Path to the input configfile. Naming should be input_*.conf to make sure it's getting included by the main config.

Default value: "/etc/td-agent-bit/inputtail${name}.conf"

path

Data type: Stdlib::Absolutepath

Pattern specifying a specific log files or multiple ones through the use of common wildcards.

routing_tag

Data type: Optional[String[1]]

Set a tag (with regex-extract fields) that will be placed on lines read.

Default value: undef

storage_type

Data type: Optional[Enum['memory', 'filesystem']]

Specify the buffering mechanism to use. Values: memory, filesystem

Default value: undef

buffer_chunk_size

Data type: Optional[Fluentbit::Sizeunit]

Set the initial buffer size to read files data.

Default value: undef

buffer_max_size

Data type: Optional[Fluentbit::Sizeunit]

Set the limit of the buffer size per monitored file.

Default value: undef

path_key

Data type: Optional[String[1]]

Appends the name of the monitored file as part of the record. The value assigned becomes the key in the map.

Default value: undef

exclude_path

Data type: Array[String]

Shell patterns to exclude files matching a certain criteria.

Default value: []

refresh_interval

Data type: Optional[Integer[1]]

The interval of refreshing the list of watched files in seconds.

Default value: undef

rotate_wait

Data type: Optional[Integer[1]]

Specify the number of extra time in seconds to monitor a file once is rotated in case some pending data is flushed.

Default value: undef

ignore_older

Data type: Optional[Fluentbit::Timeunit]

Ignores files that have been last modified before this time.

Default value: undef

skip_long_lines

Data type: Boolean

When a monitored file reach it buffer capacity due to a very long line (Buffer_Max_Size), the default behavior is to stop monitoring that file. Skip_Long_Lines alters that behavior and instructs Fluent Bit to skip long lines and continue processing other lines that fits into the buffer size.

Default value: false

db

Data type: Optional[Stdlib::Absolutepath]

Specify the database file to keep track of monitored files and offsets.

Default value: undef

db_sync

Data type: Optional[Enum['Extra', 'Full', 'Normal', 'Off']]

Set a default synchronization (I/O) method. Values: Extra, Full, Normal, Off.

Default value: undef

mem_buf_limit

Data type: Optional[Fluentbit::Sizeunit]

Set a limit of memory that Tail plugin can use when appending data to the Engine.

Default value: undef

key

Data type: Optional[String[1]]

Name of the key to append any unstructured data to the output as.

Default value: undef

tag_regex

Data type: Optional[String[1]]

Set a regex to exctract fields from the file.

Default value: undef

multiline

Data type: Boolean

Try to discover multiline messages and use the proper parsers to compose the outgoing messages.

Default value: false

multiline_flush

Data type: Optional[Integer[1]]

Wait period time in seconds to process queued multiline messages.

Default value: undef

parsers

Data type: Array[String[1]]

Name of parsers to use for disecting the message. When multiline is enabled, the first array entry becomes the configuration value for Parser_Firstline, with the remaining entry being assigned to the Parser_N attributes. When multiline is disabled, only the first item is used for the Parser configuration value.

Default value: []

docker_mode

Data type: Boolean

Recombine split Docker log lines before passing them to any parser.

Default value: false

docker_mode_flush

Data type: Optional[Integer[1]]

Wait period time in seconds to flush queued unfinished split lines.

Default value: undef

fluentbit::output::es

Plugin to output logs to a configured elasticsearch instance

Examples

include fluentbit::output::es

Parameters

The following parameters are available in the fluentbit::output::es defined type.

configfile

Data type: String

Path to the output configfile. Naming should be output_*.conf to make sure it's getting included by the main config.

Default value: "/etc/td-agent-bit/outputes${name}.conf"

match

Data type: String

Tag to route the output.

Default value: '*'

tls

Data type: Enum['on', 'off']

Turn TLS encrypted communication with Elasticsearch on / off.

Default value: 'off'

host

Data type: Stdlib::Host

IP address or hostname of the target Elasticsearch instance

Default value: '127.0.0.1'

port

Data type: Stdlib::Port

TCP port of the target Elasticsearch instance

Default value: 9200

path

Elasticsearch accepts new data on HTTP query path "/_bulk". But it is also possible to serve Elasticsearch behind a reverse proxy on a subpath. This option defines such path on the fluent-bit side. It simply adds a path prefix in the indexing HTTP POST URI.

buffer_size

Specify the buffer size used to read the response from the Elasticsearch HTTP service. This option is useful for debugging purposes where is required to read full responses, note that response size grows depending of the number of records inserted. To set an unlimited amount of memory set this value to False, otherwise the value must be according to the Unit Size specification.

pipeline

Newer versions of Elasticsearch allows to setup filters called pipelines. This option allows to define which pipeline the database should use. For performance reasons is strongly suggested to do parsing and filtering on Fluent Bit side, avoid pipelines.

http_user

Data type: Optional[String]

Optional username credential for Elastic X-Pack access

Default value: undef

http_passwd

Data type: Optional[String]

Password for user defined in HTTP_User

Default value: undef

index

Data type: String

Index name

Default value: 'fluentbit'

type

Data type: String

Type name

Default value: 'flb_type'

logstash_format

Enable Logstash format compatibility.

logstash_prefix

When Logstash_Format is enabled, the Index name is composed using a prefix and the date, e.g: If Logstash_Prefix is equals to 'mydata' your index will become 'mydata-YYYY.MM.DD'. The last string appended belongs to the date when the data is being generated.

logstash_dateformat

Time format (based on strftime) to generate the second part of the Index name.

time_key

When Logstash_Format is enabled, each record will get a new timestamp field. The Time_Key property defines the name of that field.

time_key_format

When Logstash_Format is enabled, this property defines the format of the timestamp.

include_tag_key

When enabled, it append the Tag name to the record.

tag_key

When Include_Tag_Key is enabled, this property defines the key name for the tag.

generate_id

When enabled, generate _id for outgoing records. This prevents duplicate records when retrying ES.

replace_dots

When enabled, replace field name dots with underscore, required by Elasticsearch 2.0-2.3.

trace_output

When enabled print the elasticsearch API calls to stdout (for diag only)

current_time_index

Use current time for index generation instead of message record

logstash_prefix_key

Prefix keys with this string

fluentbit::output::http

The http output plugin allows to flush your records into a HTTP endpoint. This type manages the configuration for it.

Examples

fluentbit::output::http { 'logstash': }

Parameters

The following parameters are available in the fluentbit::output::http defined type.

configfile

Data type: Stdlib::Absolutepath

Path to the output configfile. Naming should be output_*.conf to make sure it's getting included by the main config.

Default value: "/etc/td-agent-bit/outputhttp${name}.conf"

match

Data type: Optional[String[1]]

Tag to route the output.

Default value: undef

tls

Data type: Fluentbit::TLS

TLS configuration. By default TLS is disabled.

Default value: {}

headers

Data type: Hash[String[1], String[1]]

Map of headers to add to requests.

Default value: {}

retry_limit

Data type: Variant[Undef, Boolean, Integer[1]]

Number of retries if upstream refuses the records. false will disable retries, true will cause it to retry once. All other values are passed on verbatim.

Default value: undef

host

Data type: Optional[Stdlib::Host]

IP address or hostname of the target HTTP Server.

Default value: undef

port

Data type: Optional[Stdlib::Port]

TCP port of the target HTTP Server.

Default value: undef

http_user

Data type: Optional[String[1]]

Basic Auth Username.

Default value: undef

http_passwd

Data type: Optional[String[1]]

Basic Auth Password. Requires HTTP_User to be set.

Default value: undef

proxy

Data type: Optional[Stdlib::HTTPUrl]

Specify an HTTP Proxy.

Default value: undef

uri

Data type: Optional[Stdlib::Absolutepath]

Specify an optional HTTP URI for the target web server.

Default value: undef

format

Data type: Optional[Enum['msgpack', 'json', 'json_stream', 'json_lines', 'gelf']]

Specify the data format to be used in the HTTP request body.

Default value: undef

header_tag

Data type: Optional[String[1]]

Specify an optional HTTP header field for the original message tag.

Default value: undef

json_date_key

Data type: Optional[String[1]]

Specify the name of the date field in output.

Default value: undef

json_date_format

Data type: Optional[Enum['double', 'iso8601']]

Specify the format of the date.

Default value: undef

gelf_timestamp_key

Data type: Optional[String[1]]

Specify the key to use for timestamp in gelf format.

Default value: undef

gelf_host_key

Data type: Optional[String[1]]

Specify the key to use for host in gelf format.

Default value: undef

gelf_short_messge_key

Data type: Optional[String[1]]

Specify the key to use for short in gelf format.

Default value: undef

gelf_full_message_key

Data type: Optional[String[1]]

Specify the key to use for full in gelf format.

Default value: undef

gelf_level_key

Data type: Optional[String[1]]

Specify the key to use for level in gelf format.

Default value: undef

Changelog

All notable changes to this project will be documented in this file.

Release 0.2.0

Features

• add support for managing multiple plugins of the same type

Release 0.1.0

Features

Bugfixes

Known Issues