From 942e5ae762bd226ca0bad8e2d5b6a54cdd4c0849 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Fri, 13 Feb 2026 12:25:55 +0100 Subject: [PATCH] docs: buffering: document memrb storage type and improve coverage - Add memory ring buffer (memrb) as third buffering mode section - Update intro from "two modes" to "three modes" - Add memrb as accepted value for per-input storage.type - Add memrb configuration examples in YAML and classic formats - Document memrb_dropped_chunks and memrb_dropped_bytes metrics - Add note about storage.inherit and global storage.type usage - Sort per-input settings table alphabetically by key - Clarify mem_buf_limit applies to both memory-only and memrb modes - Add storage.inherit configuration example with YAML and classic formats - fix Log_Level capitalization in classic format examples Fixes #2373 Signed-off-by: Eric D. Schabell --- pipeline/buffering.md | 78 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 72 insertions(+), 6 deletions(-) diff --git a/pipeline/buffering.md b/pipeline/buffering.md index 68e26daf1..69150eff4 100644 --- a/pipeline/buffering.md +++ b/pipeline/buffering.md @@ -45,7 +45,7 @@ After marking a chunk as irrecoverable, Fluent Bit logs an error message and the ## Buffering modes -Fluent Bit offers two modes for storing buffered data. Both modes store buffered data in memory, but filesystem buffering is a hybrid method that stores an additional copy of buffered data in the filesystem. +Fluent Bit offers three modes for storing buffered data. Memory-only and memory ring buffer modes store data exclusively in memory, while filesystem buffering is a hybrid method that stores an additional copy of buffered data in the filesystem. You can set the buffering mode for each active [input plugin](#per-input-settings). @@ -61,6 +61,14 @@ When filesystem buffering is enabled, Fluent Bit stores each chunk of buffered d This buffering method is less efficient than memory-only buffering, and uses more system overhead, but is less prone to data loss. +### Memory ring buffer buffering + +When memory ring buffer (`memrb`) buffering is enabled, Fluent Bit stores buffered data in a fixed-size memory ring buffer. Unlike memory-only buffering, when the ring buffer is full, Fluent Bit automatically drops the oldest chunks to make room for new data instead of pausing the input plugin. Use this mode in scenarios where you want to keep the most recent data and can tolerate loss of older data. + +When chunks are dropped, Fluent Bit tracks the number of dropped chunks and bytes through the `memrb_dropped_chunks` and `memrb_dropped_bytes` metrics. You can monitor these metrics to understand how much data is being discarded. + +This buffering method prioritizes continuous data ingestion over data completeness, making it suitable for high-throughput scenarios where pausing the input isn't acceptable. + ## Configuration settings {% hint style="info" %} @@ -73,17 +81,66 @@ Use the information in this section to configure buffering settings in Fluent Bi In the [`service` section](../administration/configuring-fluent-bit/yaml/service-section.md) of Fluent Bit configuration files, several settings related to buffering are stored in the [`storage` key](../administration/configuring-fluent-bit/yaml/service-section.md#storage-configuration). These are global settings that affect all input and output plugins. +To set a default buffering mode for all input plugins, configure `storage.type` and enable `storage.inherit` in the `service` section. Input plugins that don't explicitly set their own `storage.type` will inherit the global value. The following example sets filesystem buffering as the default for all inputs, while allowing individual inputs to override the default: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +service: + flush: 1 + log_level: info + storage.path: /var/log/flb-storage/ + storage.type: filesystem + storage.inherit: on + storage.sync: normal + storage.max_chunks_up: 128 + +pipeline: + inputs: + - name: cpu + # Inherits storage.type: filesystem from service + + - name: mem + storage.type: memory # Overrides the inherited default +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[SERVICE] + flush 1 + Log_Level info + storage.path /var/log/flb-storage/ + storage.type filesystem + storage.inherit on + storage.sync normal + storage.max_chunks_up 128 + +[INPUT] + name cpu + # Inherits storage.type: filesystem from SERVICE + +[INPUT] + name mem + storage.type memory # Overrides the inherited default +``` + +{% endtab %} +{% endtabs %} + ### Per-input settings You can configure buffering settings for any input plugin by using these configuration parameters: | Key | Description | Default | | :--- | :--- | :--- | -| `storage.type` | Specifies the buffering mechanism to use for this input plugin. To enable filesystem buffering, a global [`storage.path`](../administration/configuring-fluent-bit/yaml/service-section.md#storage-configuration) value must be set in the `service` section of your configuration file. Accepted values: `memory`, `filesystem`. | `memory` | -| `mem_buf_limit` | If memory-only buffering is enabled, sets a limit for how much buffered data the plugin can write to memory. After this limit is reached, the plugin will pause until more memory becomes available. This value must follow [unit size](../administration/configuring-fluent-bit.md#unit-sizes) specifications. If unspecified, no limit is enforced. | `0` | +| `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory. With memory-only buffering, the plugin pauses until more memory becomes available after this limit is reached. With memory ring buffer (`memrb`) buffering, the oldest chunks are dropped to make room for new data instead of pausing. This value must follow [unit size](../administration/configuring-fluent-bit.md#unit-sizes) specifications. If unspecified, no limit is enforced. | `0` | | `storage.pause_on_chunks_overlimit` | If filesystem buffering is enabled, specifies how the input plugin should behave after the global `storage.max_chunks_up` limit is reached. When set to `off`, the plugin will stop buffering data to memory but continue buffering data to the filesystem. When set to `on`, the plugin will stop both memory buffering and filesystem buffering until more memory becomes available. Possible values: `on`, `off`. | `off` | +| `storage.type` | Specifies the buffering mechanism to use for this input plugin. To enable filesystem buffering, a global [`storage.path`](../administration/configuring-fluent-bit/yaml/service-section.md#storage-configuration) value must be set in the `service` section of your configuration file. When set to `memrb`, Fluent Bit uses a memory ring buffer that automatically drops the oldest chunks when the buffer is full to make room for new data. Accepted values: `memory`, `filesystem`, `memrb`. | `memory` | -The following configuration example sets global settings in `service` to support filesystem buffering, then configures one input plugin with filesystem buffering and one input plugin with memory-only buffering: +The following configuration example sets global settings in `service` to support filesystem buffering, then configures input plugins with filesystem buffering, memory-only buffering, and memory ring buffer buffering: {% tabs %} {% tab title="fluent-bit.yaml" %} @@ -105,6 +162,10 @@ pipeline: - name: mem storage.type: memory + + - name: dummy + storage.type: memrb + mem_buf_limit: 10M ``` {% endtab %} @@ -113,7 +174,7 @@ pipeline: ```text [SERVICE] flush 1 - log_Level info + Log_Level info storage.path /var/log/flb-storage/ storage.sync normal storage.checksum off @@ -127,6 +188,11 @@ pipeline: [INPUT] name mem storage.type memory + +[INPUT] + name dummy + storage.type memrb + mem_buf_limit 10M ``` {% endtab %} @@ -172,7 +238,7 @@ pipeline: ```text [SERVICE] flush 1 - log_Level info + Log_Level info storage.path /var/log/flb-storage/ storage.sync normal storage.checksum off