fluentbit

Manage fluentbit installation

Project URL RSS Feed

Module Stats

1,810 downloads

453 latest version

3.1 quality score

Version information

released Aug 27th 2024

This version is compatible with:

Puppet Enterprise 2023.8.x, 2023.7.x, 2023.6.x, 2023.5.x, 2023.4.x, 2023.3.x, 2023.2.x, 2023.1.x, 2021.7.x
Puppet >= 7.24 < 9.0.0
, , , , ,

Start using this module

Installation method

Add this module to your Puppetfile:

mod 'deric-fluentbit', '1.2.0'

Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add deric-fluentbit

Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install deric-fluentbit --version 1.2.0

Documentation

deric/fluentbit — version 1.2.0 Aug 27th 2024

puppet-fluentbit

A Puppet module to manage Fluent Bit installation.

Description

Fluent Bit is a fast, lightweight agent for trasporting logs, metrics, traces, etc.

In order to install the package and setup fluent-bit service, simply include the main class:

include fluentbit

Usage

Fluent Bit supports wide range of inputs

fluentbit pipeline

Define some inputs:

fluentbit::inputs:
  'tail-syslog':
    plugin: tail
    properties:
      Path: /var/syslog

outputs:

fluentbit::outputs:
  'prometheus':
    plugin: prometheus_exporter
    properties:
      match: nginx.metrics.*
      host: 0.0.0.0
      port: 2021

Configuration

Limit maximum memory usage per systemd unit:

fluentbit::service_override_unit_file: true
fluentbit::memory_max: 2G

All Puppet variables are documented in REFERENCE.md.

Acceptance tests

BEAKER_destroy=no BEAKER_setfile=debian11-64 bundle exec rake beaker

Reference

Classes

fluentbit
fluentbit::config: configures the main fluentbit main config
fluentbit::install: Installs fluentbit package
fluentbit::repo: Setup the fluentbit repo
fluentbit::repo::debian: Manage fluentbit apt repo
fluentbit::repo::redhat: Manage yum repo
fluentbit::service: Manage fluent-bit service

Defined types

fluentbit::pipeline: Configures fluentbit pipeline (input, output or filter)
fluentbit::upstream: Configures fluentbit upstream

Classes

`fluentbit`

The fluentbit class.

Examples

include fluentbit

Parameters

The following parameters are available in the fluentbit class:

manage_storage_dir
config_dir
plugins_dir
scripts_dir
data_dir
binary_file
config_file
config_file_mode
config_folder_mode
storage_path
storage_sync
storage_checksum
storage_backlog_mem_limit
storage_max_chunks_up
storage_metrics
storage_delete_irrecoverable_chunks
health_check
hc_errors_count
hc_retry_failure_count
hc_period
manage_plugins_file
plugins_file
plugins
manage_streams_file
streams_file
streams
upstreams
manage_parsers_file
parsers_file
parsers
multiline_parsers
flush
grace
daemon
dns_mode
log_level
http_server
http_listen
http_port
coro_stack_size
scheduler_cap
scheduler_base
json_convert_nan_to_null
variables
manage_package_repo
package_ensure
package_name
manage_service
service_enable
service_has_status
service_restart_command
service_has_restart
service_override_unit_file
service_ensure
service_name
manage_config_dir
manage_data_dir
inputs
outputs
filters
memory_max

`manage_storage_dir`

Data type: Boolean

Whether to manage the storage directory. storage_path must be defined. Default value: true

`config_dir`

Data type: Stdlib::Absolutepath

Absolute path to directory where configuration files are stored.

`plugins_dir`

Data type: String

Directory name for plugins, relative to @config_dir

`scripts_dir`

Data type: String

Directory name for scripts, relative to @config_dir

`data_dir`

Data type: Stdlib::Absolutepath

Path to data directory that will be used by plugins using DB feature.

`binary_file`

Data type: Stdlib::Absolutepath

Path of the daemon binary.

`config_file`

Data type: Stdlib::Absolutepath

Path of the daemon configuration.

`config_file_mode`

Data type: Stdlib::Filemode

File mode to apply to the daemon configuration file

`config_folder_mode`

Data type: Stdlib::Filemode

File mode to apply folders managed by the module

`storage_path`

Data type: Optional[Stdlib::Absolutepath]

Set an optional location in the file system to store streams and chunks of data. If this parameter is not set, Input plugins can only use in-memory buffering.

`storage_sync`

Data type: Optional[Enum['normal', 'full']]

Configure the synchronization mode used to store the data into the file system. It can take the values normal or full. Default value: 'normal'

`storage_checksum`

Data type: Boolean

Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. Default value: false

`storage_backlog_mem_limit`

Data type: Optional[String[1]]

If the input plugin has enabled filesystem storage type, this property sets the maximum number of Chunks that can be up in memory. This is the setting to use to control memory usage when you enable storage.type filesystem. Default value: 128

`storage_max_chunks_up`

Data type: Optional[Integer]

If storage.path is set, Fluent Bit will look for data chunks that were not delivered and are still in the storage layer, these are called backlog data. This option configure a hint of maximum value of memory to use when processing these records. Default value: 5M

`storage_metrics`

Data type: Boolean

If http_server option has been enabled in the main [SERVICE] section, this option registers a new endpoint where internal metrics of the storage layer can be consumed. Default value: false

`storage_delete_irrecoverable_chunks`

Data type: Boolean

When enabled, irrecoverable chunks will be deleted during runtime, and any other irrecoverable chunk located in the configured storage path directory will be deleted when Fluent-Bit starts. Default value: false

`health_check`

Data type: Boolean

Enable or disable health_check Default Off

`hc_errors_count`

Data type: Integer

Only in use if health_check is enabled. The error count after which the healcheck returns an error. Default 5

`hc_retry_failure_count`

Data type: Integer

Only in use if health_check is enabled. Retry count till a check returns an error Default 5

`hc_period`

Data type: Integer

Only in use if health_check is enabled. Time period by second to count the error and retry failure data point Default 60

`manage_plugins_file`

Data type: Boolean

Whether to manage the enabled external plugins

`plugins_file`

Data type: Stdlib::Absolutepath

A plugins configuration file allows to define paths for external plugins.

`plugins`

Data type: Array[Stdlib::Absolutepath]

List of external plugin objects to enable

Default value: []

`manage_streams_file`

Data type: Boolean

Whether to manage the stream processing configuration

`streams_file`

Data type: Stdlib::Absolutepath

Path for the Stream Processor configuration file.

`streams`

Data type: Fluentbit::Stream

Stream processing tasks

Default value: {}

`upstreams`

Data type: Hash[String, Hash]

Upstreams used by forward plugins

Default value: {}

`manage_parsers_file`

Data type: Boolean

Whether to manage the parser definitions

`parsers_file`

Data type: Stdlib::Absolutepath

Path for a parsers configuration file. Multiple Parsers_File entries can be used.

`parsers`

Data type: Fluentbit::Parser

List of parser definitions. The default value consists of all the available definitions provided by the upstream project as of version 1.3

`multiline_parsers`

Data type: Fluentbit::MultilineParser

List of parser definitions. The default value consists of all the available definitions provided by the upstream project as of version 2.1

Default value: {}

`flush`

Data type: Integer

Set the flush time in seconds. Everytime it timeouts, the engine will flush the records to the output plugin.

`grace`

Data type: Integer

Set the grace time in seconds. The engine loop uses a Grace timeout to define the wait time on exit.

`daemon`

Data type: Boolean

Boolean value to set if Fluent Bit should run as a Daemon (background) or not. Allowed values are: yes, no, on and off.

`dns_mode`

Data type: Enum['UDP', 'TCP']

Sets the primary transport layer protocol used by the asynchronous DNS resolver.

`log_level`

Data type: Fluentbit::Loglevel

Set the logging verbosity level. Values are: error, info, debug and trace. Values are accumulative, e.g: if 'debug' is set, it will include error, info and debug. Note that trace mode is only available if Fluent Bit was built with the WITH_TRACE option enabled.

`http_server`

Data type: Boolean

Enable built-in HTTP Server

`http_listen`

Data type: Stdlib::IP::Address::Nosubnet

Set listening interface for HTTP Server when it's enabled

`http_port`

Data type: Stdlib::Port

Set TCP Port for the HTTP Server

`coro_stack_size`

Data type: Integer

Set the coroutines stack size in bytes. The value must be greater than the page size of the running system.

`scheduler_cap`

Data type: Integer

Set a maximum retry time in seconds. The property is supported from v1.8.7.

`scheduler_base`

Data type: Integer

Sets the base of exponential backoff. The property is supported from v1.8.7.

`json_convert_nan_to_null`

Data type: Boolean

If enabled, NaN is converted to null when fluent-bit converts msgpack to json.

`variables`

Data type: Hash

macro definitions to use in the configuration file the will be registered using the @SET command or using Env section in YAML syntax.

Default value: {}

`manage_package_repo`

Data type: Boolean

`package_ensure`

Data type: String[1]

`package_name`

Data type: String[1]

`manage_service`

Data type: Boolean

`service_enable`

Data type: Boolean

`service_has_status`

Data type: Boolean

`service_restart_command`

Data type: Optional[String[1]]

`service_has_restart`

Data type: Boolean

`service_override_unit_file`

Data type: Boolean

`service_ensure`

Data type: Stdlib::Ensure::Service

`service_name`

Data type: String[1]

`manage_config_dir`

Data type: Boolean

`manage_data_dir`

Data type: Boolean

`inputs`

Data type: Fluentbit::PipelinePlugin

Default value: {}

`outputs`

Data type: Fluentbit::PipelinePlugin

Default value: {}

`filters`

Data type: Fluentbit::PipelinePlugin

Default value: {}

`memory_max`

Data type: Optional[String[1]]

Default value: undef

`fluentbit::config`

configures the main fluentbit main config

Parameters

The following parameters are available in the fluentbit::config class:

config_dir
plugins_path
scripts_path

`config_dir`

Data type: Stdlib::Absolutepath

Absolute path to main configuration directory

Default value: $fluentbit::config_dir

`plugins_path`

Data type: Stdlib::Absolutepath

Default value: $fluentbit::plugins_path

`scripts_path`

Data type: Stdlib::Absolutepath

Default value: $fluentbit::scripts_path

`fluentbit::install`

Installs fluentbit package

`fluentbit::repo`

Setup the fluentbit repo

`fluentbit::repo::debian`

Manage fluentbit apt repo

Parameters

The following parameters are available in the fluentbit::repo::debian class:

key_fingerprint
key_location
flavour
release

`key_fingerprint`

Data type: String[1]

GPG key identifier of the repository

`key_location`

Data type: Stdlib::HTTPUrl

GPG key URI

`flavour`

Data type: String[1]

e.g. Debian/Ubuntu

Default value: $facts['os']['distro']['id']

`release`

Data type: String[1]

distribution code name

Default value: $facts['os']['distro']['codename']

`fluentbit::repo::redhat`

Manage yum repo

`fluentbit::service`

Manage fluent-bit service

Defined types

`fluentbit::pipeline`

Configures fluentbit pipeline (input, output or filter)

Note This resource add extra configuration elements for some combinations of type-plugin_names, like db configuration for input plugins, or upstream configuration for output-forward plugin
See also
- https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/configuration-file#config_pipeline

Examples

fluentbit::pipeline { 'input-dummy':
  pipeline => 'input',
  plugin   => 'dummy',
}

fluentbit {
  input_plugins => {
    'input-dummy' => { 'plugin' => 'dummy' },
  },
}

Parameters

The following parameters are available in the fluentbit::pipeline defined type:

pipeline
plugin
order
properties

`pipeline`

Data type: Fluentbit::PipelineType

Defines the pipeline type to be configured

`plugin`

Data type: String[1]

fluent-bit plugin name to be used

`order`

Data type: Integer

Order to be applied to concat::fragment

Default value: 10

`properties`

Data type: Hash[String, Any]

Hash of rest of properties needed to configure the pipeline-plugin

Default value: {}

`fluentbit::upstream`

Configures fluentbit upstream

See also
- https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/upstream-servers

Examples

fluentbit::upstream { 'upstreams':
  nodes  => {
    'n1' => { 'host' => '127.0.0.1', port => 1234 },
    'n2' => { 'host' => '127.0.0.1', port => 1235 },
  },
}

Parameters

The following parameters are available in the fluentbit::upstream defined type:

upstream_name
nodes
config_dir

`upstream_name`

Data type: String[1]

Name to be assigned to upstream, defaults to resource namevar

Default value: $name

`nodes`

Data type: Hash[String, Fluentbit::UpstreamNode]

Hash of nodes assigned to this upstream

`config_dir`

Data type: Stdlib::Absolutepath

Absolute path to config directory

Default value: $fluentbit::config_dir

Data types

`Fluentbit::Field_decoder`

The Fluentbit::Field_decoder data type.

Alias of

Struct[{
  decoder => Enum['json'],
  field   => String[1],
}]

`Fluentbit::Field_type`

The Fluentbit::Field_type data type.

Alias of

Struct[{
  type  => Enum['string', 'integer', 'bool', 'float', 'hex'],
  field => String[1],
}]

`Fluentbit::Loglevel`

The Fluentbit::Loglevel data type.

Alias of Enum['error', 'warning', 'info', 'debug', 'trace']

`Fluentbit::MultilineParser`

The Fluentbit::MultilineParser data type.

Alias of

Hash[String, Struct[{
  type                    => Enum['regex'],
  rules                   => Array[Struct[{
                            state      => String[1],
                            regex      => String[1],
                            next_state => String[1],
                          }]],
  Optional[parser]        => String[1],
  Optional[key_content]   => String[1],
  Optional[flush_timeout] => Integer,
}]]

`Fluentbit::Parser`

The Fluentbit::Parser data type.

Alias of

Hash[String, Struct[{
  format                 => Enum['json', 'regex', 'ltsv', 'logfmt'],
  Optional[regex]        => String[1],
  Optional[time_key]     => String[1],
  Optional[time_format]  => String[1],
  Optional[time_offset]  => String[1],
  Optional[time_keep]    => Boolean,
  Optional[types]        => Array[Fluentbit::Field_type],
  Optional[decode_field] => Array[Fluentbit::Field_decoder],
}]]

`Fluentbit::PipelinePlugin`

The Fluentbit::PipelinePlugin data type.

Alias of

Hash[String, Struct[{
  plugin               => String[1],
  Optional[pipeline]   => Fluentbit::PipelineType,
  Optional[order]      => Integer,
  Optional[properties] => Hash,
}]]

`Fluentbit::PipelineType`

The Fluentbit::PipelineType data type.

Alias of Enum['input', 'filter', 'output']

`Fluentbit::Stream`

The Fluentbit::Stream data type.

Alias of

Hash[String, Struct[{
    exec => String[1],
  }]]

`Fluentbit::UpstreamNode`

The Fluentbit::UpstreamNode data type.

Alias of

Struct[{
  host => String[1],
  port => Integer,
  Optional[tls]  => Enum['On', 'Off'],
  Optional[tls_verify] => Enum['On', 'Off']
}]

Changelog

All notable changes to this project will be documented in this file.

[2024-08-27] Release 1.2.0

Features

make tls configurable

Changes

Support puppet/systemd 7.x

Full changes

[2024-06-27] Release 1.1.0

Changes

Fixed puppetlabs/apt requirement
Fixed Puppet 8 warnings
Replaced merge() function calls by native + operand

Full changes

[2024-06-21] Release 1.0.0

Changes

BC use stdlib:: prefix for puppet 4.x functions
Puppet 8 compatible

Full changes

[2024-03-13] Release 0.4.0

Features

Limit memory for systemd unit

Full changes

[2024-03-11] Release 0.3.0

Bugfixes

Fix yaml issues

Full changes

[2024-02-11] Release 0.2.0

Features

Support health checks #2
Support newer systemd dependency

[2023-07-28] Release 0.1.0

Features

Initial release

Bugfixes

Known Issues

Modules

Discover

Contribute

Puppet

Premium features

Downloads

Resources

About Forge

Getting Started

fluentbit

Version information

This version is compatible with:

Start using this module

Documentation

puppet-fluentbit

Description

Usage

Configuration

Acceptance tests

Reference

Table of Contents

Classes

Defined types

Data types

Classes

fluentbit

Examples

Parameters

manage_storage_dir

config_dir

plugins_dir

scripts_dir

data_dir

binary_file

config_file

config_file_mode

config_folder_mode

storage_path

storage_sync

storage_checksum

storage_backlog_mem_limit

storage_max_chunks_up

storage_metrics

storage_delete_irrecoverable_chunks

health_check

hc_errors_count

hc_retry_failure_count

hc_period

manage_plugins_file

plugins_file

plugins

manage_streams_file

streams_file

streams

upstreams

manage_parsers_file

parsers_file

parsers

multiline_parsers

flush

grace

daemon

dns_mode

log_level

http_server

http_listen

http_port

coro_stack_size

scheduler_cap

scheduler_base

json_convert_nan_to_null

variables

manage_package_repo

package_ensure

package_name

manage_service

service_enable

service_has_status

service_restart_command

service_has_restart

`fluentbit`

`manage_storage_dir`

`config_dir`

`plugins_dir`

`scripts_dir`

`data_dir`

`binary_file`

`config_file`

`config_file_mode`

`config_folder_mode`

`storage_path`

`storage_sync`

`storage_checksum`

`storage_backlog_mem_limit`

`storage_max_chunks_up`

`storage_metrics`

`storage_delete_irrecoverable_chunks`

`health_check`

`hc_errors_count`

`hc_retry_failure_count`

`hc_period`

`manage_plugins_file`

`plugins_file`

`plugins`

`manage_streams_file`

`streams_file`

`streams`

`upstreams`

`manage_parsers_file`

`parsers_file`

`parsers`

`multiline_parsers`

`flush`

`grace`

`daemon`

`dns_mode`

`log_level`

`http_server`

`http_listen`

`http_port`

`coro_stack_size`

`scheduler_cap`

`scheduler_base`

`json_convert_nan_to_null`

`variables`

`manage_package_repo`

`package_ensure`

`package_name`

`manage_service`

`service_enable`

`service_has_status`

`service_restart_command`

`service_has_restart`

`service_override_unit_file`

`service_ensure`

`service_name`

`manage_config_dir`

`manage_data_dir`

`inputs`

`outputs`

`filters`

`memory_max`

`fluentbit::config`

`config_dir`

`plugins_path`

`scripts_path`

`fluentbit::install`

`fluentbit::repo`

`fluentbit::repo::debian`

`key_fingerprint`

`key_location`

`flavour`

`release`

`fluentbit::repo::redhat`

`fluentbit::service`

`fluentbit::pipeline`

`pipeline`

`plugin`

`order`

`properties`