diff --git a/buffer/file.md b/buffer/file.md index 573cc9b7..59072039 100644 --- a/buffer/file.md +++ b/buffer/file.md @@ -160,7 +160,16 @@ This config outputs the buffer chunk files as follows. In this case, `path_suffi ## Limitation +{% hint style='warning' %} Caution: `file` buffer implementation depends on the characteristics of the local file system. Don't use `file` buffer on remote file systems e.g. NFS, GlusterFS, HDFS, etc. We observed major data loss by using the remote file system. +{% endhint %} + +{% hint style='warning' %} +`path` is implicitly used as file buffer path if file buffer path is not explicitly configured in `` section. +In such a case, it must not contain `[` or `]` in the actually evaluated `path`. +If such a character was included in the path, it might be trouble (ignored) after restarting of Fluentd during resuming buffer handling process. +As a workaround, override `path` parameter in `` section which does not contain such a character. +{% endhint %} If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. diff --git a/buffer/file_single.md b/buffer/file_single.md index 90d99fd0..30e33ed1 100644 --- a/buffer/file_single.md +++ b/buffer/file_single.md @@ -99,11 +99,13 @@ With `auto`, the plugin decides the chunk format by `formatted_to_msgpack_binary # NG. multiple field keys are not supported ``` -This limitation will be removed by adding the metadata header in the file. +This limitation will be removed by adding the metadata header in the file in the future. ### Remote File System is Not Supported +{% hint style='warning' %} Caution: `file_single` buffer implementation depends on the characteristics of the local file system. Don't use `file_single` buffer on remote file systems e.g. NFS, GlusterFS, HDFS and etc. We observed major data loss by using remote file system. +{% endhint %} If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. diff --git a/configuration/buffer-section.md b/configuration/buffer-section.md index 5f4c5c82..1ff2fc4f 100644 --- a/configuration/buffer-section.md +++ b/configuration/buffer-section.md @@ -71,6 +71,9 @@ In case of no or blank chunk key, the output plugin writes all the matched event | 12:00:25 ssh.login {"key1":"yay","key2":100} --| ``` +{% hint style='info' %} +Blank chunk keys should be used for simple stream transfers (via forward or http plugin) or when the output target destination does not depend on the key. (e.g. stdout) +{% endhint %} ### Tag @@ -221,6 +224,11 @@ Buffer chunk keys may be specified empty by using `[]` as the `buffer` section a This is particularly useful when the output plugin has its own default chunk keys and it needs to disable those. +{% hint style='info' %} +Empty Keys might be useful for CI because it suppress chunk splitting and simplify internal flush behavior. +As a side effect, you can't specify placeholder which uses chunk metadata (e.g timekey, "${tag}" in path and so on) +{% endhint %} + ## Placeholders When the chunk keys are specified, these values can be extracted in configuration parameter values. It depends on whether the plugin applies a method\(`extract_placeholders`\) on configuration values or not. @@ -454,6 +462,7 @@ Following are the flushing parameters for chunks to optimize performance \(laten into chunks * `flush_interval` \[time\] * Default: 60s + * The interval time to check whether buffer should be flushed or not. It depends on chunk key or flush mode whether the buffer will be flushed or not, so it doesn't mean always flushed in this duration if the condition is not matched to be flushed. * `flush_thread_count` \[integer\] * Default: 1 * The number of threads to flush/write chunks in parallel diff --git a/output/file.md b/output/file.md index 10dd86dc..a194f281 100644 --- a/output/file.md +++ b/output/file.md @@ -66,6 +66,17 @@ The `path` parameter is used as ``'s `path` in this plugin. Initially, you may see a file which looks like `/path/to/file.%Y%m%d/buffer.b5692238db04045286097f56f361028db.log`. This is an intermediate buffer file \(`b5692238db04045286097f56f361028db` identifies the buffer\). Once the content of the buffer has been completely [flushed](../buffer/file.md), you will see the output file without the trailing identifier. +{% hint style='info' %} +If you want to specific `time`, `tag` or any `record` fields - placeholders in the path, you must specify them as `` chunk keys. +(Thus, blank chunk keys `` or empty keys `` is not allowed. It causes configuration error.) +{% endhint %} + +{% hint style='warning' %} +`path` is implicitly used as file buffer path if file buffer path is not explicitly configured in `` section. +In such a case, it must not contain `[` or `]` in the actually evaluated `path`. See [`` limitation](../buffer/file.md#limitation) about detail. + +{% endhint %} + ### `append` | type | default | version |