From 10fb23ffc7561c290644462f146ca24a1243b369 Mon Sep 17 00:00:00 2001 From: Kentaro Hayashi Date: Mon, 9 Mar 2026 16:48:31 +0900 Subject: [PATCH 1/6] buffer: emphasize limitation about remote file system Signed-off-by: Kentaro Hayashi --- buffer/file.md | 2 ++ buffer/file_single.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/buffer/file.md b/buffer/file.md index 573cc9b7..fe703128 100644 --- a/buffer/file.md +++ b/buffer/file.md @@ -160,7 +160,9 @@ This config outputs the buffer chunk files as follows. In this case, `path_suffi ## Limitation +{% hint style='warning' %} Caution: `file` buffer implementation depends on the characteristics of the local file system. Don't use `file` buffer on remote file systems e.g. NFS, GlusterFS, HDFS, etc. We observed major data loss by using the remote file system. +{% endhint %} If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. diff --git a/buffer/file_single.md b/buffer/file_single.md index 90d99fd0..acfa22b6 100644 --- a/buffer/file_single.md +++ b/buffer/file_single.md @@ -103,7 +103,9 @@ This limitation will be removed by adding the metadata header in the file. ### Remote File System is Not Supported +{% hint style='warning' %} Caution: `file_single` buffer implementation depends on the characteristics of the local file system. Don't use `file_single` buffer on remote file systems e.g. NFS, GlusterFS, HDFS and etc. We observed major data loss by using remote file system. +{% endhint %} If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. From 77d5ccba7fb8fb61fa39f5719664197bf2e75998 Mon Sep 17 00:00:00 2001 From: Kentaro Hayashi Date: Mon, 9 Mar 2026 17:00:01 +0900 Subject: [PATCH 2/6] buffer: explicitly note as "feature" Signed-off-by: Kentaro Hayashi --- buffer/file_single.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buffer/file_single.md b/buffer/file_single.md index acfa22b6..30e33ed1 100644 --- a/buffer/file_single.md +++ b/buffer/file_single.md @@ -99,7 +99,7 @@ With `auto`, the plugin decides the chunk format by `formatted_to_msgpack_binary # NG. multiple field keys are not supported ``` -This limitation will be removed by adding the metadata header in the file. +This limitation will be removed by adding the metadata header in the file in the future. ### Remote File System is Not Supported From ab24d3faee2dd5c01a2bd415eaa0c369454cb8cb Mon Sep 17 00:00:00 2001 From: Kentaro Hayashi Date: Mon, 9 Mar 2026 17:26:07 +0900 Subject: [PATCH 3/6] buffer: add use-case information Signed-off-by: Kentaro Hayashi --- configuration/buffer-section.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/configuration/buffer-section.md b/configuration/buffer-section.md index 5f4c5c82..bd5e052f 100644 --- a/configuration/buffer-section.md +++ b/configuration/buffer-section.md @@ -71,6 +71,9 @@ In case of no or blank chunk key, the output plugin writes all the matched event | 12:00:25 ssh.login {"key1":"yay","key2":100} --| ``` +{% hint style='info' %} +Blank chunk keys should be used for simple stream transfers (via forward or http plugin) or when the output target destination does not depend on the key. (e.g. stdout) +{% endhint %} ### Tag @@ -221,6 +224,11 @@ Buffer chunk keys may be specified empty by using `[]` as the `buffer` section a This is particularly useful when the output plugin has its own default chunk keys and it needs to disable those. +{% hint style='info' %} +Empty Keys might be useful for CI because it suppress chunk splitting and simplify internal flush behavior. +As a side effect, you can't specify placeholder which uses chunk metadata (e.g timekey, "${tag}" in path and so on) +{% endhint %} + ## Placeholders When the chunk keys are specified, these values can be extracted in configuration parameter values. It depends on whether the plugin applies a method\(`extract_placeholders`\) on configuration values or not. From 200d6f5defd56bf3f4cc43542dd4d7ef7c72e5ec Mon Sep 17 00:00:00 2001 From: Kentaro Hayashi Date: Tue, 10 Mar 2026 11:33:25 +0900 Subject: [PATCH 4/6] file: add note about chunk keys It is useful to explain the relationship between buffer section and chunk keys. Signed-off-by: Kentaro Hayashi --- output/file.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/output/file.md b/output/file.md index 10dd86dc..536cb004 100644 --- a/output/file.md +++ b/output/file.md @@ -66,6 +66,11 @@ The `path` parameter is used as ``'s `path` in this plugin. Initially, you may see a file which looks like `/path/to/file.%Y%m%d/buffer.b5692238db04045286097f56f361028db.log`. This is an intermediate buffer file \(`b5692238db04045286097f56f361028db` identifies the buffer\). Once the content of the buffer has been completely [flushed](../buffer/file.md), you will see the output file without the trailing identifier. +{% hint style='info' %} +If you want to specific `time`, `tag` or any `record` fields - placeholders in the path, you must specify them as `` chunk keys. +(Thus, blank chunk keys `` or empty keys `` is not allowed. It causes configuration error.) +{% endhint %} + ### `append` | type | default | version | From 44a2b56aa3c01b033548089da4982cf9dfff4480 Mon Sep 17 00:00:00 2001 From: Kentaro Hayashi Date: Tue, 10 Mar 2026 11:58:09 +0900 Subject: [PATCH 5/6] file: add limitation about path parameter https://github.com/fluent/fluentd/issues/4407 Signed-off-by: Kentaro Hayashi --- buffer/file.md | 7 +++++++ output/file.md | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/buffer/file.md b/buffer/file.md index fe703128..59072039 100644 --- a/buffer/file.md +++ b/buffer/file.md @@ -164,5 +164,12 @@ This config outputs the buffer chunk files as follows. In this case, `path_suffi Caution: `file` buffer implementation depends on the characteristics of the local file system. Don't use `file` buffer on remote file systems e.g. NFS, GlusterFS, HDFS, etc. We observed major data loss by using the remote file system. {% endhint %} +{% hint style='warning' %} +`path` is implicitly used as file buffer path if file buffer path is not explicitly configured in `` section. +In such a case, it must not contain `[` or `]` in the actually evaluated `path`. +If such a character was included in the path, it might be trouble (ignored) after restarting of Fluentd during resuming buffer handling process. +As a workaround, override `path` parameter in `` section which does not contain such a character. +{% endhint %} + If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. diff --git a/output/file.md b/output/file.md index 536cb004..a194f281 100644 --- a/output/file.md +++ b/output/file.md @@ -71,6 +71,12 @@ If you want to specific `time`, `tag` or any `record` fields - placeholders in t (Thus, blank chunk keys `` or empty keys `` is not allowed. It causes configuration error.) {% endhint %} +{% hint style='warning' %} +`path` is implicitly used as file buffer path if file buffer path is not explicitly configured in `` section. +In such a case, it must not contain `[` or `]` in the actually evaluated `path`. See [`` limitation](../buffer/file.md#limitation) about detail. + +{% endhint %} + ### `append` | type | default | version | From 43d95376e7b972adcc419e069bc397a253f86f9f Mon Sep 17 00:00:00 2001 From: Kentaro Hayashi Date: Tue, 10 Mar 2026 14:12:56 +0900 Subject: [PATCH 6/6] buffer: add note about the meaning of interval Signed-off-by: Kentaro Hayashi --- configuration/buffer-section.md | 1 + 1 file changed, 1 insertion(+) diff --git a/configuration/buffer-section.md b/configuration/buffer-section.md index bd5e052f..1ff2fc4f 100644 --- a/configuration/buffer-section.md +++ b/configuration/buffer-section.md @@ -462,6 +462,7 @@ Following are the flushing parameters for chunks to optimize performance \(laten into chunks * `flush_interval` \[time\] * Default: 60s + * The interval time to check whether buffer should be flushed or not. It depends on chunk key or flush mode whether the buffer will be flushed or not, so it doesn't mean always flushed in this duration if the condition is not matched to be flushed. * `flush_thread_count` \[integer\] * Default: 1 * The number of threads to flush/write chunks in parallel