buffer: improve explanation about chunk keys and limitation (#615)

* buffer: emphasize limitation about remote file system

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>

* buffer: explicitly note as "feature"

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>

* buffer: add use-case information

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>

* file: add note about chunk keys

It is useful to explain the relationship between buffer section and
chunk keys.

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>

* file: add limitation about path parameter

fluent/fluentd#4407

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>

* buffer: add note about the meaning of interval

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>

---------

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>

Author: kenhys

buffer/file.md

@@ -160,7 +160,16 @@ This config outputs the buffer chunk files as follows. In this case, `path_suffi
 
 ## Limitation
 
+{% hint style='warning' %}
 Caution: `file` buffer implementation depends on the characteristics of the local file system. Don't use `file` buffer on remote file systems e.g. NFS, GlusterFS, HDFS, etc. We observed major data loss by using the remote file system.
+{% endhint %}
+
+{% hint style='warning' %}
+`path` is implicitly used as file buffer path if file buffer path is not explicitly configured in `<buffer>` section.
+In such a case, it must not contain `[` or `]` in the actually evaluated `path`.
+If such a character was included in the path, it might be trouble (ignored) after restarting of Fluentd during resuming buffer handling process.
+As a workaround, override `path` parameter in `<buffer>` section which does not contain such a character.
+{% endhint %}
 
 If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.
 

buffer/file_single.md

@@ -99,11 +99,13 @@ With `auto`, the plugin decides the chunk format by `formatted_to_msgpack_binary
 <buffer key1,key2> # NG. multiple field keys are not supported
 ```
 
-This limitation will be removed by adding the metadata header in the file.
+This limitation will be removed by adding the metadata header in the file in the future.
 
 ### Remote File System is Not Supported
 
+{% hint style='warning' %}
 Caution: `file_single` buffer implementation depends on the characteristics of the local file system. Don't use `file_single` buffer on remote file systems e.g. NFS, GlusterFS, HDFS and etc. We observed major data loss by using remote file system.
+{% endhint %}
 
 If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.
 

configuration/buffer-section.md

@@ -71,6 +71,9 @@ In case of no or blank chunk key, the output plugin writes all the matched event
                                                  |
 12:00:25 ssh.login  {"key1":"yay","key2":100}  --|
 ```
+{% hint style='info' %}
+Blank chunk keys should be used for simple stream transfers (via forward or http plugin) or when the output target destination does not depend on the key. (e.g. stdout)
+{% endhint %}
 
 ### Tag
 
@@ -221,6 +224,11 @@ Buffer chunk keys may be specified empty by using `[]` as the `buffer` section a
 
 This is particularly useful when the output plugin has its own default chunk keys and it needs to disable those.
 
+{% hint style='info' %}
+Empty Keys might be useful for CI because it suppress chunk splitting and simplify internal flush behavior.
+As a side effect, you can't specify placeholder which uses chunk metadata (e.g timekey, "${tag}" in path and so on)
+{% endhint %}
+
 ## Placeholders
 
 When the chunk keys are specified, these values can be extracted in configuration parameter values. It depends on whether the plugin applies a method\(`extract_placeholders`\) on configuration values or not.
@@ -454,6 +462,7 @@ Following are the flushing parameters for chunks to optimize performance \(laten
     into chunks
 * `flush_interval` \[time\]
   * Default: 60s
+  * The interval time to check whether buffer should be flushed or not. It depends on chunk key or flush mode whether the buffer will be flushed or not, so it doesn't mean always flushed in this duration if the condition is not matched to be flushed.
 * `flush_thread_count` \[integer\]
   * Default: 1
   * The number of threads to flush/write chunks in parallel

output/file.md

@@ -66,6 +66,17 @@ The `path` parameter is used as `<buffer>`'s `path` in this plugin.
 
 Initially, you may see a file which looks like `/path/to/file.%Y%m%d/buffer.b5692238db04045286097f56f361028db.log`. This is an intermediate buffer file \(`b5692238db04045286097f56f361028db` identifies the buffer\). Once the content of the buffer has been completely [flushed](../buffer/file.md), you will see the output file without the trailing identifier.
 
+{% hint style='info' %}
+If you want to specific `time`, `tag` or any `record` fields - placeholders in the path, you must specify them as `<buffer>` chunk keys.
+(Thus, blank chunk keys `<buffer>` or empty keys `<buffer []>` is not allowed. It causes configuration error.)
+{% endhint %}
+
+{% hint style='warning' %}
+`path` is implicitly used as file buffer path if file buffer path is not explicitly configured in `<buffer>` section.
+In such a case, it must not contain `[` or `]` in the actually evaluated `path`. See  [`<buffer>` limitation](../buffer/file.md#limitation) about detail.
+
+{% endhint %}
+
 ### `append`
 
 | type | default | version |