Version information
This version is compatible with:
- Puppet Enterprise 2023.8.x, 2023.7.x, 2023.6.x, 2023.5.x, 2023.4.x, 2023.3.x, 2023.2.x, 2023.1.x, 2023.0.x, 2021.7.x, 2021.6.x, 2021.5.x, 2021.4.x, 2021.3.x, 2021.2.x, 2021.1.x, 2021.0.x
- Puppet >= 7.0.0 < 9.0.0
- , , , , ,
Start using this module
Add this module to your Puppetfile:
mod 'puppet-kafka', '9.0.0'Learn more about managing modules with a PuppetfileDocumentation
Kafka module for Puppet
Table of Contents
- Overview
- Module Description - What the module does and why it is useful
- Setup - The basics of getting started with Kafka
- Usage - Configuration options and additional functionality
- Reference - An under-the-hood peek at what the module is doing and how
- Limitations - OS compatibility, etc.
- Development - Guide for contributing to the module
Overview
The Kafka module for managing the installation and configuration of Apache Kafka.
Module Description
The Kafka module for managing the installation and configuration of Apache Kafka: it's brokers, producers and consumers.
Setup
What Kafka affects
Installs the Kafka package and creates a new service.
Setup requirements
This module has the following dependencies:
Beginning with Kafka
To successfully install Kafka using this module you need to have Apache ZooKeeper already running at localhost:2181. You can specify another ZooKeeper host:port configuration using the config hash of the kafka::broker class.
The default configuration installs Kafka 0.11.0.3 binaries with Scala 2.11:
class { 'kafka': }
If you want a Kafka broker server that connects to ZooKeeper listening on port 2181:
class { 'kafka::broker':
config => {
'broker.id' => '0',
'zookeeper.connect' => 'localhost:2181'
}
}
Usage
You can specify different Kafka binaries packages versions to install. Please take a look at the different Scala and Kafka versions combinations at the Apache Kafka Website
Installing Kafka version 1.1.0 with scala 2.12
We first install the binary package with:
class { 'kafka':
version => '1.1.0',
scala_version => '2.12'
}
Then we set a minimal Kafka broker configuration with:
class { 'kafka::broker':
config => {
'broker.id' => '0',
'zookeeper.connect' => 'localhost:2181'
}
}
Reference
The reference documentation of this module is generated using puppetlabs/puppetlabs-strings.
Limitations
This module only supports Kafka >= 0.9.0.0.
This module is tested on the following platforms:
- Debian 8
- Debian 9
- Debian 10
- Ubuntu 16.04
- Ubuntu 18.04
- CentOS 7
It is tested with the OSS version of Puppet (>= 5.5) only.
Development
This module has grown over time based on a range of contributions from people using it. If you follow these contributing guidelines your patch will likely make it into a release a little more quickly.
Author
This module is maintained by Vox Pupuli. It was originally written and maintained by Liam Bennett.
Reference
Table of Contents
Classes
Public Classes
kafka: This class handles the Kafka requirements.kafka::broker: This class handles the Kafka (broker).kafka::consumer: This class handles the Kafka (consumer).kafka::mirror: This class handles the Kafka (mirror).kafka::params: This class provides default parameters.kafka::producer: This class handles the Kafka (producer).
Private Classes
kafka::broker::config: This class handles the Kafka (broker) config.kafka::broker::install: This class handles the Kafka (broker) package.kafka::broker::service: This class handles the Kafka (broker) service.kafka::consumer::config: This class handles the Kafka (consumer) config.kafka::consumer::install: This class handles the Kafka (consumer) package.kafka::consumer::service: This class handles the Kafka (consumer) service.kafka::mirror::config: This class handles the Kafka (mirror) config.kafka::mirror::install: This class handles the Kafka (mirror) package.kafka::mirror::service: This class handles the Kafka (mirror) service.kafka::producer::config: This class handles the Kafka (producer) config.kafka::producer::install: This class handles the Kafka (producer) package.kafka::producer::service: This class handles the Kafka (producer) service.
Defined types
kafka::topic: This defined type handles the creation of Kafka topics.
Classes
kafka
This class handles the Kafka requirements.
Examples
Basic usage
class { 'kafka': }
Parameters
The following parameters are available in the kafka class:
kafka_versionscala_versioninstall_dirmirror_urlmanage_javapackage_dirpackage_namemirror_subpathproxy_serverproxy_portproxy_hostproxy_typepackage_ensureuser_nameuser_shellgroup_namesystem_usersystem_groupuser_idgroup_idmanage_usermanage_groupconfig_dirlog_dirinstall_mode
kafka_version
Data type: String[1]
The version of Kafka that should be installed.
Default value: $kafka::params::kafka_version
scala_version
Data type: String[1]
The scala version what Kafka was built with.
Default value: $kafka::params::scala_version
install_dir
Data type: Stdlib::Absolutepath
The directory to install Kafka to.
Default value: $kafka::params::install_dir
mirror_url
Data type: Stdlib::HTTPUrl
The url where the Kafka is downloaded from.
Default value: $kafka::params::mirror_url
manage_java
Data type: Boolean
Install java if it's not already installed.
Default value: $kafka::params::manage_java
package_dir
Data type: Stdlib::Absolutepath
The directory to install Kafka.
Default value: $kafka::params::package_dir
package_name
Data type: Optional[String[1]]
Package name, when installing Kafka from a package.
Default value: $kafka::params::package_name
mirror_subpath
Data type: Optional[String[1]]
The sub directory where the source is downloaded from.
Default value: $kafka::params::mirror_subpath
proxy_server
Data type: Optional[String[1]]
Set proxy server, when installing Kafka from source.
Default value: $kafka::params::proxy_server
proxy_port
Data type: Optional[String[1]]
Set proxy port, when installing Kafka from source.
Default value: $kafka::params::proxy_port
proxy_host
Data type: Optional[String[1]]
Set proxy host, when installing Kafka from source.
Default value: $kafka::params::proxy_host
proxy_type
Data type: Optional[String[1]]
Set proxy type, when installing Kafka from source.
Default value: $kafka::params::proxy_type
package_ensure
Data type: String[1]
Package version or ensure state, when installing Kafka from a package.
Default value: $kafka::params::package_ensure
user_name
Data type: String[1]
User to run Kafka as.
Default value: $kafka::params::user_name
user_shell
Data type: Stdlib::Absolutepath
Login shell of the Kafka user.
Default value: $kafka::params::user_shell
group_name
Data type: String[1]
Group to run Kafka as.
Default value: $kafka::params::group_name
system_user
Data type: Boolean
Whether the Kafka user is a system user or not.
Default value: $kafka::params::system_user
system_group
Data type: Boolean
Whether the Kafka group is a system group or not.
Default value: $kafka::params::system_group
user_id
Data type: Optional[Integer]
Create the Kafka user with this ID.
Default value: $kafka::params::user_id
group_id
Data type: Optional[Integer]
Create the Kafka group with this ID.
Default value: $kafka::params::group_id
manage_user
Data type: Boolean
Create the Kafka user if it's not already present.
Default value: $kafka::params::manage_user
manage_group
Data type: Boolean
Create the Kafka group if it's not already present.
Default value: $kafka::params::manage_group
config_dir
Data type: Stdlib::Absolutepath
The directory to create the Kafka config files to.
Default value: $kafka::params::config_dir
log_dir
Data type: Stdlib::Absolutepath
The directory for Kafka log files.
Default value: $kafka::params::log_dir
install_mode
Data type: Stdlib::Filemode
The permissions for the install directory.
Default value: $kafka::params::install_mode
kafka::broker
This class handles the Kafka (broker).
Examples
Basic usage
class { 'kafka::broker':
config => {
'broker.id' => '0',
'zookeeper.connect' => 'localhost:2181'
}
}
Parameters
The following parameters are available in the kafka::broker class:
kafka_versionscala_versioninstall_dirmirror_urlmanage_javapackage_dirpackage_namepackage_ensureuser_nameuser_shellgroup_nameuser_idgroup_idmanage_usermanage_groupconfig_modeconfig_dirlog_dirbin_dirservice_namemanage_serviceservice_ensureservice_restartservice_requireslimit_nofilelimit_coretimeout_stopexec_stopdaemon_startenvconfigheap_optsjmx_optslog4j_optsmanage_log4jlog4j_contentlog_file_sizelog_file_countopts
kafka_version
Data type: String[1]
The version of Kafka that should be installed.
Default value: $kafka::params::kafka_version
scala_version
Data type: String[1]
The scala version what Kafka was built with.
Default value: $kafka::params::scala_version
install_dir
Data type: Stdlib::Absolutepath
The directory to install Kafka to.
Default value: $kafka::params::install_dir
mirror_url
Data type: Stdlib::HTTPUrl
The url where the Kafka is downloaded from.
Default value: $kafka::params::mirror_url
manage_java
Data type: Boolean
Install java if it's not already installed.
Default value: $kafka::params::manage_java
package_dir
Data type: Stdlib::Absolutepath
The directory to install Kafka.
Default value: $kafka::params::package_dir
package_name
Data type: Optional[String[1]]
Package name, when installing Kafka from a package.
Default value: $kafka::params::package_name
package_ensure
Data type: String[1]
Package version or ensure state, when installing Kafka from a package.
Default value: $kafka::params::package_ensure
user_name
Data type: String[1]
User to run Kafka as.
Default value: $kafka::params::user_name
user_shell
Data type: Stdlib::Absolutepath
Login shell of the Kafka user.
Default value: $kafka::params::user_shell
group_name
Data type: String[1]
Group to run Kafka as.
Default value: $kafka::params::group_name
user_id
Data type: Optional[Integer]
Create the Kafka user with this ID.
Default value: $kafka::params::user_id
group_id
Data type: Optional[Integer]
Create the Kafka group with this ID.
Default value: $kafka::params::group_id
manage_user
Data type: Boolean
Create the Kafka user if it's not already present.
Default value: $kafka::params::manage_user
manage_group
Data type: Boolean
Create the Kafka group if it's not already present.
Default value: $kafka::params::manage_group
config_mode
Data type: Stdlib::Filemode
The permissions for the config files.
Default value: $kafka::params::config_mode
config_dir
Data type: Stdlib::Absolutepath
The directory to create the Kafka config files to.
Default value: $kafka::params::config_dir
log_dir
Data type: Stdlib::Absolutepath
The directory for Kafka log files.
Default value: $kafka::params::log_dir
bin_dir
Data type: Stdlib::Absolutepath
The directory where the Kafka scripts are.
Default value: $kafka::params::bin_dir
service_name
Data type: String[1]
Set the name of the service.
Default value: 'kafka'
manage_service
Data type: Boolean
Install the systemd service.
Default value: $kafka::params::manage_service
service_ensure
Data type: Enum['running', 'stopped']
Set the ensure state of the service.
Default value: $kafka::params::service_ensure
service_restart
Data type: Boolean
Whether the configuration files should trigger a service restart.
Default value: $kafka::params::service_restart
service_requires
Data type: Array[String[1]]
Set the list of services required to be running before Kafka.
Default value: $kafka::params::service_requires
limit_nofile
Data type: Optional[String[1]]
Set the 'LimitNOFILE' option of the systemd service.
Default value: $kafka::params::limit_nofile
limit_core
Data type: Optional[String[1]]
Set the 'LimitCORE' option of the systemd service.
Default value: $kafka::params::limit_core
timeout_stop
Data type: Optional[String[1]]
Set the 'TimeoutStopSec' option of the systemd service.
Default value: $kafka::params::timeout_stop
exec_stop
Data type: Boolean
Set the 'ExecStop' option of the systemd service to 'kafka-server-stop.sh'.
Default value: $kafka::params::exec_stop
daemon_start
Data type: Boolean
Use the '-daemon' option when starting Kafka with 'kafka-server-start.sh'.
Default value: $kafka::params::daemon_start
env
Data type: Hash
A hash of the environment variables to set.
Default value: {}
config
Data type: Hash[String[1], Any]
A hash of the broker configuration options.
Default value: {}
heap_opts
Data type: String[1]
Set the Java heap size.
Default value: $kafka::params::broker_heap_opts
jmx_opts
Data type: String[1]
Set the JMX options.
Default value: $kafka::params::broker_jmx_opts
log4j_opts
Data type: String[1]
Set the Log4j options.
Default value: $kafka::params::broker_log4j_opts
manage_log4j
Data type: Boolean
Enable or disable the management of the log4j configuration file.
Default value: $kafka::params::manage_log4j
log4j_content
Data type: Optional[String[1]]
Provide custom content for the log4j configuration file if manage_log4j is enabled, instead of using the content provided in this module.
Default value: undef
log_file_size
Data type: Pattern[/[1-9][0-9]*[KMG]B/]
Set the max size of the Kafka log files before they are rolled.
Default value: $kafka::params::log_file_size
log_file_count
Data type: Integer[1, 50]
Set the number of Kafka log files to keep.
Default value: $kafka::params::log_file_count
opts
Data type: String[0]
Set the Kafka options.
Default value: $kafka::params::broker_opts
kafka::consumer
This class handles the Kafka (consumer).
Examples
Basic usage
class { 'kafka::consumer':
config => {
'client.id' => '0',
'zookeeper.connect' => 'localhost:2181'
}
}
Parameters
The following parameters are available in the kafka::consumer class:
kafka_versionscala_versioninstall_dirmirror_urlmanage_javapackage_dirpackage_namepackage_ensureuser_nameuser_shellgroup_nameuser_idgroup_idmanage_usermanage_groupconfig_modeconfig_dirlog_dirbin_dirservice_namemanage_serviceservice_ensureservice_restartservice_requireslimit_nofilelimit_coreenvconfigservice_configjmx_optslog4j_optsmanage_log4jlog_file_sizelog_file_count
kafka_version
Data type: String[1]
The version of Kafka that should be installed.
Default value: $kafka::params::kafka_version
scala_version
Data type: String[1]
The scala version what Kafka was built with.
Default value: $kafka::params::scala_version
install_dir
Data type: Stdlib::Absolutepath
The directory to install Kafka to.
Default value: $kafka::params::install_dir
mirror_url
Data type: Stdlib::HTTPUrl
The url where the Kafka is downloaded from.
Default value: $kafka::params::mirror_url
manage_java
Data type: Boolean
Install java if it's not already installed.
Default value: $kafka::params::manage_java
package_dir
Data type: Stdlib::Absolutepath
The directory to install Kafka.
Default value: $kafka::params::package_dir
package_name
Data type: Optional[String[1]]
Package name, when installing Kafka from a package.
Default value: $kafka::params::package_name
package_ensure
Data type: String[1]
Package version or ensure state, when installing Kafka from a package.
Default value: $kafka::params::package_ensure
user_name
Data type: String[1]
User to run Kafka as.
Default value: $kafka::params::user_name
user_shell
Data type: Stdlib::Absolutepath
Login shell of the Kafka user.
Default value: $kafka::params::user_shell
group_name
Data type: String[1]
Group to run Kafka as.
Default value: $kafka::params::group_name
user_id
Data type: Optional[Integer]
Create the Kafka user with this ID.
Default value: $kafka::params::user_id
group_id
Data type: Optional[Integer]
Create the Kafka group with this ID.
Default value: $kafka::params::group_id
manage_user
Data type: Boolean
Create the Kafka user if it's not already present.
Default value: $kafka::params::manage_user
manage_group
Data type: Boolean
Create the Kafka group if it's not already present.
Default value: $kafka::params::manage_group
config_mode
Data type: Stdlib::Filemode
The permissions for the config files.
Default value: $kafka::params::config_mode
config_dir
Data type: Stdlib::Absolutepath
The directory to create the Kafka config files to.
Default value: $kafka::params::config_dir
log_dir
Data type: Stdlib::Absolutepath
The directory for Kafka log files.
Default value: $kafka::params::log_dir
bin_dir
Data type: Stdlib::Absolutepath
The directory where the Kafka scripts are.
Default value: $kafka::params::bin_dir
service_name
Data type: String[1]
Set the name of the service.
Default value: 'kafka-consumer'
manage_service
Data type: Boolean
Install the systemd service.
Default value: $kafka::params::manage_service
service_ensure
Data type: Enum['running', 'stopped']
Set the ensure state of the service.
Default value: $kafka::params::service_ensure
service_restart
Data type: Boolean
Whether the configuration files should trigger a service restart.
Default value: $kafka::params::service_restart
service_requires
Data type: Array[String[1]]
Set the list of services required to be running before Kafka.
Default value: $kafka::params::service_requires
limit_nofile
Data type: Optional[String[1]]
Set the 'LimitNOFILE' option of the systemd service.
Default value: $kafka::params::limit_nofile
limit_core
Data type: Optional[String[1]]
Set the 'LimitCORE' option of the systemd service.
Default value: $kafka::params::limit_core
env
Data type: Hash
A hash of the environment variables to set.
Default value: {}
config
Data type: Hash[String[1], Any]
A hash of the consumer configuration options.
Default value: {}
service_config
Data type: Hash[String[1],String[1]]
A hash of the kafka-console-consumer.sh script options.
Default value: {}
jmx_opts
Data type: String[1]
Set the JMX options.
Default value: $kafka::params::consumer_jmx_opts
log4j_opts
Data type: String[1]
Set the Log4j options.
Default value: $kafka::params::consumer_log4j_opts
manage_log4j
Data type: Boolean
Enable or disable the management of the log4j configuration file.
Default value: $kafka::params::manage_log4j
log_file_size
Data type: Pattern[/[1-9][0-9]*[KMG]B/]
Set the max size of the Kafka log files before they are rolled.
Default value: $kafka::params::log_file_size
log_file_count
Data type: Integer[1, 50]
Set the number of Kafka log files to keep.
Default value: $kafka::params::log_file_count
kafka::mirror
This class handles the Kafka (mirror).
Examples
Basic usage
class { 'kafka::mirror':
consumer_config => {
'group.id' => 'kafka-mirror',
'zookeeper.connect' => 'localhost:2181'
},
producer_config => {
'zookeeper.connect' => 'localhost:2181',
},
service_config => {
'whitelist' => '.*',
}
}
Parameters
The following parameters are available in the kafka::mirror class:
kafka_versionscala_versioninstall_dirmirror_urlmanage_javapackage_dirpackage_namepackage_ensureuser_nameuser_shellgroup_nameuser_idgroup_idmanage_usermanage_groupconfig_modeconfig_dirlog_dirbin_dirservice_namemanage_serviceservice_ensureservice_restartservice_requireslimit_nofilelimit_coreenvconsumer_configproducer_configservice_configheap_optsjmx_optslog4j_optsmanage_log4jlog_file_sizelog_file_count
kafka_version
Data type: String[1]
The version of Kafka that should be installed.
Default value: $kafka::params::kafka_version
scala_version
Data type: String[1]
The scala version what Kafka was built with.
Default value: $kafka::params::scala_version
install_dir
Data type: Stdlib::Absolutepath
The directory to install Kafka to.
Default value: $kafka::params::install_dir
mirror_url
Data type: Stdlib::HTTPUrl
The url where the Kafka is downloaded from.
Default value: $kafka::params::mirror_url
manage_java
Data type: Boolean
Install java if it's not already installed.
Default value: $kafka::params::manage_java
package_dir
Data type: Stdlib::Absolutepath
The directory to install Kafka.
Default value: $kafka::params::package_dir
package_name
Data type: Optional[String[1]]
Package name, when installing Kafka from a package.
Default value: $kafka::params::package_name
package_ensure
Data type: String[1]
Package version or ensure state, when installing Kafka from a package.
Default value: $kafka::params::package_ensure
user_name
Data type: String[1]
User to run Kafka as.
Default value: $kafka::params::user_name
user_shell
Data type: Stdlib::Absolutepath
Login shell of the Kafka user.
Default value: $kafka::params::user_shell
group_name
Data type: String[1]
Group to run Kafka as.
Default value: $kafka::params::group_name
user_id
Data type: Optional[Integer]
Create the Kafka user with this ID.
Default value: $kafka::params::user_id
group_id
Data type: Optional[Integer]
Create the Kafka group with this ID.
Default value: $kafka::params::group_id
manage_user
Data type: Boolean
Create the Kafka user if it's not already present.
Default value: $kafka::params::manage_user
manage_group
Data type: Boolean
Create the Kafka group if it's not already present.
Default value: $kafka::params::manage_group
config_mode
Data type: Stdlib::Filemode
The permissions for the config files.
Default value: $kafka::params::config_mode
config_dir
Data type: Stdlib::Absolutepath
The directory to create the Kafka config files to.
Default value: $kafka::params::config_dir
log_dir
Data type: Stdlib::Absolutepath
The directory for Kafka log files.
Default value: $kafka::params::log_dir
bin_dir
Data type: Stdlib::Absolutepath
The directory where the Kafka scripts are.
Default value: $kafka::params::bin_dir
service_name
Data type: String[1]
Set the name of the service.
Default value: 'kafka-mirror'
manage_service
Data type: Boolean
Install the systemd service.
Default value: $kafka::params::manage_service
service_ensure
Data type: Enum['running', 'stopped']
Set the ensure state of the service.
Default value: $kafka::params::service_ensure
service_restart
Data type: Boolean
Whether the configuration files should trigger a service restart.
Default value: $kafka::params::service_restart
service_requires
Data type: Array[String[1]]
Set the list of services required to be running before Kafka.
Default value: $kafka::params::service_requires
limit_nofile
Data type: Optional[String[1]]
Set the 'LimitNOFILE' option of the systemd service.
Default value: $kafka::params::limit_nofile
limit_core
Data type: Optional[String[1]]
Set the 'LimitCORE' option of the systemd service.
Default value: $kafka::params::limit_core
env
Data type: Hash
A hash of the environment variables to set.
Default value: {}
consumer_config
Data type: Hash[String[1],String[1]]
A hash of the consumer configuration options.
Default value: {}
producer_config
Data type: Hash[String[1],String[1]]
A hash of the producer configuration options.
Default value: {}
service_config
Data type: Hash[String[1],String[1]]
A hash of the mirror script options.
Default value: {}
heap_opts
Data type: String[1]
Set the Java heap size.
Default value: $kafka::params::mirror_heap_opts
jmx_opts
Data type: String[1]
Set the JMX options.
Default value: $kafka::params::mirror_jmx_opts
log4j_opts
Data type: String[1]
Set the Log4j options.
Default value: $kafka::params::mirror_log4j_opts
manage_log4j
Data type: Boolean
Enable or disable the management of the log4j configuration file.
Default value: $kafka::params::manage_log4j
log_file_size
Data type: Pattern[/[1-9][0-9]*[KMG]B/]
Set the max size of the Kafka log files before they are rolled.
Default value: $kafka::params::log_file_size
log_file_count
Data type: Integer[1, 50]
Set the number of Kafka log files to keep.
Default value: $kafka::params::log_file_count
kafka::params
This class provides default parameters.
kafka::producer
This class handles the Kafka (producer).
Examples
Basic usage
class { 'kafka::producer':
config => {
'client.id' => '0',
'zookeeper.connect' => 'localhost:2181'
}
}
Parameters
The following parameters are available in the kafka::producer class:
inputkafka_versionscala_versioninstall_dirmirror_urlmanage_javapackage_dirpackage_namepackage_ensureuser_nameuser_shellgroup_nameuser_idgroup_idmanage_usermanage_groupconfig_modeconfig_dirlog_dirbin_dirservice_namemanage_serviceservice_ensureservice_restartservice_requireslimit_nofilelimit_coreenvconfigservice_configjmx_optslog4j_opts
input
Data type: Optional[String[1]]
Set named pipe as input.
kafka_version
Data type: String[1]
The version of Kafka that should be installed.
Default value: $kafka::params::kafka_version
scala_version
Data type: String[1]
The scala version what Kafka was built with.
Default value: $kafka::params::scala_version
install_dir
Data type: Stdlib::Absolutepath
The directory to install Kafka to.
Default value: $kafka::params::install_dir
mirror_url
Data type: Stdlib::HTTPUrl
The url where the Kafka is downloaded from.
Default value: $kafka::params::mirror_url
manage_java
Data type: Boolean
Install java if it's not already installed.
Default value: $kafka::params::manage_java
package_dir
Data type: Stdlib::Absolutepath
The directory to install Kafka.
Default value: $kafka::params::package_dir
package_name
Data type: Optional[String[1]]
Package name, when installing Kafka from a package.
Default value: $kafka::params::package_name
package_ensure
Data type: String[1]
Package version or ensure state, when installing Kafka from a package.
Default value: $kafka::params::package_ensure
user_name
Data type: String[1]
User to run Kafka as.
Default value: $kafka::params::user_name
user_shell
Data type: Stdlib::Absolutepath
Login shell of the Kafka user.
Default value: $kafka::params::user_shell
group_name
Data type: String[1]
Group to run Kafka as.
Default value: $kafka::params::group_name
user_id
Data type: Optional[Integer]
Create the Kafka user with this ID.
Default value: $kafka::params::user_id
group_id
Data type: Optional[Integer]
Create the Kafka group with this ID.
Default value: $kafka::params::group_id
manage_user
Data type: Boolean
Create the Kafka user if it's not already present.
Default value: $kafka::params::manage_user
manage_group
Data type: Boolean
Create the Kafka group if it's not already present.
Default value: $kafka::params::manage_group
config_mode
Data type: Stdlib::Filemode
The permissions for the config files.
Default value: $kafka::params::config_mode
config_dir
Data type: Stdlib::Absolutepath
The directory to create the Kafka config files to.
Default value: $kafka::params::config_dir
log_dir
Data type: Stdlib::Absolutepath
The directory for Kafka log files.
Default value: $kafka::params::log_dir
bin_dir
Data type: Stdlib::Absolutepath
The directory where the Kafka scripts are.
Default value: $kafka::params::bin_dir
service_name
Data type: String[1]
Set the name of the service.
Default value: 'kafka-producer'
manage_service
Data type: Boolean
Install systemd service.
Default value: $kafka::params::manage_service
service_ensure
Data type: Enum['running', 'stopped']
Set the ensure state of the service.
Default value: $kafka::params::service_ensure
service_restart
Data type: Boolean
Whether the configuration files should trigger a service restart.
Default value: $kafka::params::service_restart
service_requires
Data type: Array[String[1]]
Set the list of services required to be running before Kafka.
Default value: $kafka::params::service_requires
limit_nofile
Data type: Optional[String[1]]
Set the 'LimitNOFILE' option of the systemd service.
Default value: $kafka::params::limit_nofile
limit_core
Data type: Optional[String[1]]
Set the 'LimitCORE' option of the systemd service.
Default value: $kafka::params::limit_core
env
Data type: Hash
A hash of the environment variables to set.
Default value: {}
config
Data type: Hash[String[1], Any]
A hash of the producer configuration options.
Default value: {}
service_config
Data type: Hash[String[1],String[1]]
A hash of the kafka-console-producer.sh script options.
Default value: {}
jmx_opts
Data type: String[1]
Set the JMX options.
Default value: $kafka::params::producer_jmx_opts
log4j_opts
Data type: String[1]
Set the Log4j options.
Default value: $kafka::params::producer_log4j_opts
Defined types
kafka::topic
This defined type handles the creation of Kafka topics.
Examples
Basic usage
kafka::topic { 'test':
ensure => present,
zookeeper => 'localhost:2181',
replication_factor => 1,
partitions => 1,
}
Parameters
The following parameters are available in the kafka::topic defined type:
ensure
Data type: Optional[String[1]]
Should the topic be created.
Default value: undef
zookeeper
Data type: Optional[String[1]]
The connection string for the ZooKeeper connection in the form host:port. Multiple hosts can be given to allow fail-over. Kafka < 3.0.0 only!
Default value: undef
bootstrap_server
Data type: Optional[String[1]]
The Kafka server to connect to in the form host:port. Kafka >= 2.2.0 only!
Default value: undef
replication_factor
Data type: Integer
The replication factor for each partition in the topic being created. If not supplied, defaults to the cluster default.
Default value: 1
partitions
Data type: Integer
The number of partitions for the topic being created or altered. If not supplied for create, defaults to the cluster default.
Default value: 1
bin_dir
Data type: String[1]
The directory where the file kafka-topics.sh is located.
Default value: '/opt/kafka/bin'
config
Data type: Optional[Hash[String[1],String[1]]]
A topic configuration override for the topic being created or altered. See the Kafka documentation for full details on the topic configs.
Default value: undef
Changelog
All notable changes to this project will be documented in this file. Each new release typically also includes the latest modulesync defaults. These should not affect the functionality of the module.
v9.0.0 (2024-12-13)
Breaking changes:
- Drop Ubuntu 18.04 support #371 (zilchms)
- Drop Debian 10 support #370 (zilchms)
- Drop RedHat, CentOs 7 and 8 and Rocky 8 support #368 (zilchms)
- Drop support for non systemd OS #357 (zilchms)
- Drop Puppet 6 support #346 (bastelfreak)
- drop support for debian 9 (EOL) #345 (jhoblitt)
- drop support for SLES 11 (EOL) #344 (jhoblitt)
- drop support for ubuntu 16.04 (EOL) #343 (jhoblitt)
- drop support for debian 8 (EOL) #342 (jhoblitt)
Implemented enhancements:
- fix: honor service_restart in kafka::broker::service #378 (schustersv)
- update puppet-systemd upper bound to 8.0.0 #366 (TheMeier)
- feat: add option to allow for custom log4j configuration #363 (TheMeier)
- puppet/systemd: Allow 6.x #361 (zilchms)
- puppetlabs/java: Allow 10.x #360 (zilchms)
- puppet/archive: Allow 7.x #359 (zilchms)
- Add Debian 11 and 12 support #356 (zilchms)
- Add Rocky, EL 8 and 9 support #354 (zilchms)
- Add ubuntu 20.04 and 22.04 support #353 (zilchms)
- Add Puppet 8 support #348 (bastelfreak)
- puppetlabs/stdlib: Allow 9.x #347 (bastelfreak)
- bump puppet/systemd to \< 5.0.0 #341 (jhoblitt)
- Support kafka 3.0 topics #337 (der-eismann)
Merged pull requests:
v8.0.0 (2021-08-05)
Breaking changes:
- Drop EoL Puppet 5; Add Puppet 7 support #333 (bastelfreak)
- Remove centos 6 #322 (bjoernhaeuser)
- No longer accept deprecated String params #321 (der-eismann)
Implemented enhancements:
- Enable customized log4j.properties files #200
- Allow to manage Log4j configuration #328 (evgenkisel)
- allow latest dependencies #326 (bastelfreak)
Fixed bugs:
- kafka::producer class does not work with systemd #324
Closed issues:
- New release #330
- Old versions cannot be installed from apache download area #323
- systemv init created log files are not being rotated #85
Merged pull requests:
- switch from camptocamp/systemd to voxpupuli/systemd #329 (bastelfreak)
- Removing SystemD reloading step #327 (evgenkisel)
v7.0.0 (2020-06-03)
Breaking changes:
- Add parameter user_shell #315 (dhoppe)
- Rename parameters #303 (dhoppe)
- Make owner of files/directories configurable #299 (dhoppe)
Implemented enhancements:
- Add support for Hiera #316 (dhoppe)
- Add support for SLES 11/12/15 #314 (dhoppe)
- Add support for Puppet Strings #304 (dhoppe)
- Add function assert_private() #301 (dhoppe)
- Add support for Debian 10, Ubuntu 18.04 #297 (dhoppe)
Fixed bugs:
- Fix init script parameter bug #307 (pmbuko)
- Match params of init script with the ones of systemd unit #281 (tooooots)
Closed issues:
- Incorrect tag style in init.erb template causes minor bug in generated file #306
- config_dir permissions not set, owned by root, kafka user can't read - kafka 2.5.0 #305
- Add support for SLES / OpenSuSE #300
- Support for Kafka 2.0.0? #280
Merged pull requests:
- Refactor unit tests #313 (dhoppe)
- Specify data types as deeply as possible #312 (dhoppe)
- Use latest available version of ZooKeeper in acceptance tests #309 (dhoppe)
- Allow deric/zookeeper 1.x #308 (dhoppe)
- Use voxpupuli-acceptance #296 (ekohl)
v6.0.0 (2020-04-07)
Breaking changes:
- Drop Puppet 4 support #292 (alexjfisher)
- Update default version to 2.4.1 with scala 2.12 #291 (alexjfisher)
Implemented enhancements:
- Puppet-archive 3.0.0 support #256
Closed issues:
- Is this module abandoned? #290
- New version/tag #289
- Proxy Server Support #285
- Root owns config directory (but not files in it), ignoring $user #273
- check debian support #254
- Documentation: OS Compability #244
Merged pull requests:
- Fix puppet-lint
absolute_classname-check#293 (alexjfisher) - bump java version dependency #287 (TheMeier)
- Clean up acceptance spec helper #286 (ekohl)
- Allow
puppet/archive4.x,puppetlabs/java4.x andpuppetlabs/stdlib6.x #284 (alexjfisher) - update spec_helper_acceptance to work with latest beaker #279 (bastelfreak)
v5.3.0 (2018-07-07)
Implemented enhancements:
- add debian 8/9 & ubuntu 16.04 support, drop 14.04 #267 (bastelfreak)
- Use closest mirrors #261 (bjoernhaeuser)
- add support for topic configurations at creation time #260 (TheMeier)
Fixed bugs:
- Use correct mirror url #263 (bjoernhaeuser)
Merged pull requests:
- Remove coveralls badge #268 (bjoernhaeuser)
- Enable Acceptance tests #266 (bjoernhaeuser)
- [WIP] enable acceptance tests #265 (bastelfreak)
- cleanup spec_helper_acceptance.rb #264 (bastelfreak)
- Add test for different kafka version #262 (bjoernhaeuser)
v5.2.0 (2018-06-24)
Implemented enhancements:
- Some updates to be able to use an internal http mirror with proxy to download the .tar.gz archive #257 (idrissneumann)
Fixed bugs:
- Ensure the process is killed before finishing stop #135 (elmendalerenda)
Merged pull requests:
- Remove docker nodesets #255 (bastelfreak)
- drop EOL OSs; fix puppet version range #253 (bastelfreak)
v5.1.0 (2018-04-05)
Implemented enhancements:
- Config files should not be world readable #249
- Add options for more sensible config permissions; puppet stdlib to 4.22.0 #250 (langesven)
v5.0.1 (2018-03-28)
Merged pull requests:
- bump puppet to latest supported version 4.10.0 #247 (bastelfreak)
- allow camptocamp/systemd 2.X #245 (bastelfreak)
v5.0.0 (2017-12-12)
Breaking changes:
- Kafka's default version should be the latest stable version #216
- replaced service_requires_zookeeper by service_requires #236 (LionelCons)
- changed limit_nofile from Integer to String #229 (LionelCons)
- made variables consistent #223 (LionelCons)
- change the mirror class to get script options via a hash (just like consumer and producer) #221 (LionelCons)
- change the default Kafka version to 0.11.0.1 #217 (LionelCons)
- remove support for Kafka \< 0.9.0.0 #215 (LionelCons)
Implemented enhancements:
- It should be possible to specify arbitrary service dependencies #234
- It should be possible to start a broker with the -daemon option #230
- Kafka default script values should not be used #218
- Kafka default values should not be used #208
- use Wants instead of Requires #238 (LionelCons)
- added a daemon_start option to be able to start the broker with the -daemon option #231 (LionelCons)
- made LimitNOFILE and LimitCORE fully configurable #227 (LionelCons)
Fixed bugs:
- LimitNOFILE cannot be set to 'infinity' #228
Closed issues:
- The systemd script should use Wants instead of Requires #237
- LimitNOFILE and LimitCORE should be unset by default #232
- Systemd limits should be configurable #226
- The properties templates should be merged #224
- Variable names should be simple and consistent #222
- Mirror script options should be given as a hash #220
- Support for Kafka \< 0.9 should be removed #214
- params.pp contains unused parameters #211
Merged pull requests:
- removed the (undocumented) hard-coded defaults for limit_nofile and limit_core #233 (LionelCons)
- merged all the properties templates #225 (LionelCons)
- remove the unneeded service default parameters #219 (LionelCons)
- remove the unneeded broker, consumer and producer default parameters #213 (LionelCons)
- remove unused parameters in params.pp #212 (LionelCons)
v4.0.0 (2017-10-17)
Implemented enhancements:
- Adjust puppet version boundaries for puppet6 #209 (bastelfreak)
- Add support for TimeoutStopSec and ExecStop in the systemd unit file #207 (LionelCons)
- Higher allowed version for module java and lower for zookeeper #205 (hp197)
Closed issues:
- Kafka stopping should be configurable in the systemd unit file #206
v3.2.0 (2017-09-29)
Breaking changes:
- renamed kafka::broker::topic to kafka::topic and added a bin_dir param #189 (LionelCons)
- The ZooKeeper service is not required by default anymore #184 (LionelCons)
- Java is not installed by default anymore #181 (LionelCons)
Implemented enhancements:
- Added manage_user and manage_group parameters #185 (LionelCons)
Fixed bugs:
- Typo in init.erb #198
- Fixed typo in init.erb #199 (LionelCons)
- kafka::consumer now calls kafka::consumer::config #187 (LionelCons)
- kafka::mirror::service does not inherit from kafka::params anymore #177 (LionelCons)
Closed issues:
- There are parameter discrepancies amongst kafka::*::service #192
- There are parameter discrepancies between kafka::init and kafka::*::install #190
- The kafka::broker::topic defined type should rather be kafka::topic #188
- The consumer class does not call its config subclass #186
- The ZooKeeper service should not be required by default #183
- Java should not be installed by default #180
- include statements do not need quotes #178
- mirror/service.pp should not inherit from kafka::params #176
- Module user/group management should be optional #173
Merged pull requests:
- fix mirror maker with new consumer #203 (jacobmw)
- Update README.md #201 (confiq)
- enable possibility of system users for kafka #197 (Wayneoween)
- MirrorMaker service needed path to .properties files #196 (jacobmw)
- cleanup related to the service classes #193 (LionelCons)
- cleanup related to the install classes #191 (LionelCons)
- include statements do not need quotes #179 (LionelCons)
v3.1.0 (2017-07-19)
Implemented enhancements:
- handle the user, group, user_id and group_id parameters consistently #172 (LionelCons)
- the *.properties configuration files are now owned by root #166 (LionelCons)
- added a "bin_dir" parameter to configure where the Kafka scripts are #159 (LionelCons)
- Allow changing the LimitNOFILE parameter for systemd unit file #157 (jacobmw)
- add support to kafka::broker logs dir #148 (jolivares)
- Add log.message.format.version to broker params #142 (winks)
Fixed bugs:
- create $package_dir and $install_directory ony when needed #167 (LionelCons)
- fix wrong defaults for mirror/service.pp #164 (LionelCons)
- declared service dependencies: Kafka requires networking and syslog #162 (LionelCons)
- change ownership of the configuration directory #160 (LionelCons)
Closed issues:
- The user and group are hard-coded in the service templates #171
- The init script always requires the zookeeper service #168
- The configuration files should be owned by root #165
- mirror/service.pp gets some defaults from params.pp instead of mirror.pp #163
- The Systemd unit file should declare dependencies #161
- Centos 7 install #151
- systemd config should not contain dependencies on zookeeper #150
- The configuration directory should be owned by root #146
- The "bin" directory is not configurable #145
- Arbitrary shell variables cannot be set #144
- Package-based installation creates useless directories #143
Merged pull requests:
- Release 3.1.0 #175 (bastelfreak)
- added an $env parameter to control the environment passed to Kafka #170 (LionelCons)
- the init script now uses the service_requires_zookeeper parameter #169 (LionelCons)
- replace validate_* with datatypes #153 (bastelfreak)
- Add missing documentation #138 (seanmalloy)
- Add params to allow customizing user, group, and log_dir #137 (seanmalloy)
v3.0.0 (2017-02-13)
Implemented enhancements:
v2.3.0 (2017-02-11)
Closed issues:
- broker.id should also be set in $logs_dir/meta.properties #117
Merged pull requests:
- allow using any mirror_url if it ends with tgz #126 (petetodo)
- Bump dependencies #125 (juniorsysadmin)
- Mirror url custom port #121 (ellamdav)
v2.2.0 (2016-12-25)
Closed issues:
Merged pull requests:
- Fix bounding of puppet version in metadata #116 (ghoneycutt)
- Fix puppet 4 versioncmp need for string #115 (marclambrichs)
- Update readme with the current Kafka & Scala versions #112 (atrepca)
- Add missing badges #111 (dhoppe)
- adding the port to the mirror URL #108 (petetodo)
- Make the systemd manifest configurable #105 (bjoernhaeuser)
v2.1.0 (2016-08-31)
Closed issues:
- Fix regexp for URL validation (or remove URL validation at all) #92
- systemv init: "$PID_FILE does not exist, process is not running" can be wrong #90
- kafka service should wait for zookeeper #81
- 'archive' module conflict #61
Merged pull requests:
- Document missing parameters #100 (igalic)
- Fix incompatibility beetween $install_dir and $version params #99 (igalic)
- Add support for MirrorMaker abort.on.send.failure #97 (SegFaultAX)
- Init fix #94 (igalic)
- Remove URL validation #93 (volkorny)
- [skip-ci]Small typo fix #88 (skade)
- remove trailing whitespace #86 (bastelfreak)
- init clean-up #82 (igalic)
- Sync metadata.json license to be same as LICENSE (MIT) #78 (juniorsysadmin)
- parameterized group-id and user-id for kafka #77 (MaltePaulsen)
- Removed dependency which is not valid with package installation #76 (mike-petersen)
- Allowing installation via package #74 (mike-petersen)
- Add support for optional default file with environment vars #35 (knumor)
v2.0.0 (2016-05-26)
Implemented enhancements:
- Upgrade to Kafka 0.9.0.0 #30
Closed issues:
- Fix parameter install_dir/install_directory #52
- Fix parameter package_dir #51
- kafka::mirror is critically broken #39
- stop trying to make fetch happen #31
- producer init template is missing #13
- Readme states that "config" is available within the "kafka" class #8
Merged pull requests:
- Fix wrong syntax (default value for ssl.enabled.protocols) #70 (jolivares)
- Allow to set KAFKA_OPTS #69 (jolivares)
- add the jmx_opts and log4j_opts to the init scripts #67 (eliranbz)
- Removing broker gc opts #66 (bjoernhaeuser)
- add the ability to optimize all the different roles, not just the broker #65 (eliranbz)
- fix the mirror.unit template #62 (eliranbz)
- Adding support to KAFKA_HEAP_OPTS #59 (ortz)
- Support CentOS 7 #57 (bjoernhaeuser)
- Upgrade to Kafka 0.9.0.1. This should fix #30 #56 (dhoppe)
- Extend Beaker tests #55 (dhoppe)
- Fix parameter package_dir. This closes #51 #54 (dhoppe)
- Fix parameter install_dir/install_directory. This closes #52 #53 (dhoppe)
- Use module voxpupuli/archive instead of wget. This should fix #31 #50 (dhoppe)
- Submit service_name to cosumer::config and producer::config #49 (dhoppe)
- (#8) README: sync class parameter lists #48 (ffrank)
- Fix classes broker::consumer and broker::producer #46 (dhoppe)
- Pin rake to avoid rubocop/rake 11 incompatibility #45 (roidelapluie)
- Add missing dependency #44 (dhoppe)
- Fix indentation of class parameters #43 (dhoppe)
- Fix several issues regarding mirror, producer and RSpec tests #41 (dhoppe)
- Fix empty PID file creation on failed daemon start #40 (dhoppe)
- Mirror num_streams num_producers max_heap params #37 (travees)
- Widen output to find the kafka string when getting PID #34 (knumor)
- Support CentOS 7 #23 (bjoernhaeuser)
- attempt to fix #3 (allow to configure service manually) #12 (feniix)
v1.0.3 (2016-01-22)
Closed issues:
- Puppet forge module outdated: ArgumentError: Could not find declared class java #27
- Puppetforge module outdated: The init script leaves pidfile if there's an error when starting #26
- Ubuntu 12.04/14.04 LTS Compatibility Broken #24
- kafka starting as a root user #17
- Both "version" and "scala_version" are ignored within "kafka::broker" #9
- init script is incompatible with chkconfig #5
Merged pull requests:
v1.0.2 (2015-12-11)
Closed issues:
- Missing dependency on puppetlabs-java #19
Merged pull requests:
- Update module to version 1.0.2 #28 (dhoppe)
- README changes with an example of use. #20 (GabrielNicolasAvellaneda)
- Added missing dependency on puppetlabs-java #18 (GabrielNicolasAvellaneda)
- Update init.erb #16 (VuokkoVuorinnen)
- Make wget installation optional #15 (danieldreier)
- Make beaker run on centos #7 (bjoernhaeuser)
v1.0.1 (2015-03-24)
Closed issues:
- ability to set the service to disabled #3
Merged pull requests:
- Moar file descriptors! (quoting https://github.com/stack72\) #1 (pablete)
v1.0.0 (2014-10-10)
v0.2.1 (2014-06-02)
v0.2.0 (2014-06-02)
v0.1.0 (2014-05-28)
* This Changelog was automatically generated by github_changelog_generator
Dependencies
- puppet/archive (>= 1.0.0 < 8.0.0)
- puppetlabs/java (>= 1.4.2 < 11.0.0)
- puppetlabs/stdlib (>= 4.22.0 < 10.0.0)
- deric/zookeeper (>= 0.5.1 < 2.0.0)
- puppet/systemd (>= 0.4.0 < 8.0.0)
Copyright (c) 2014 OpenTable, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.