Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions buffer/file.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,16 @@ This config outputs the buffer chunk files as follows. In this case, `path_suffi

## Limitation

{% hint style='warning' %}
Caution: `file` buffer implementation depends on the characteristics of the local file system. Don't use `file` buffer on remote file systems e.g. NFS, GlusterFS, HDFS, etc. We observed major data loss by using the remote file system.
{% endhint %}

{% hint style='warning' %}
`path` is implicitly used as file buffer path if file buffer path is not explicitly configured in `<buffer>` section.
In such a case, it must not contain `[` or `]` in the actually evaluated `path`.
If such a character was included in the path, it might be trouble (ignored) after restarting of Fluentd during resuming buffer handling process.
As a workaround, override `path` parameter in `<buffer>` section which does not contain such a character.
{% endhint %}

If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

4 changes: 3 additions & 1 deletion buffer/file_single.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,11 +99,13 @@ With `auto`, the plugin decides the chunk format by `formatted_to_msgpack_binary
<buffer key1,key2> # NG. multiple field keys are not supported
```

This limitation will be removed by adding the metadata header in the file.
This limitation will be removed by adding the metadata header in the file in the future.

### Remote File System is Not Supported

{% hint style='warning' %}
Caution: `file_single` buffer implementation depends on the characteristics of the local file system. Don't use `file_single` buffer on remote file systems e.g. NFS, GlusterFS, HDFS and etc. We observed major data loss by using remote file system.
{% endhint %}

If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

9 changes: 9 additions & 0 deletions configuration/buffer-section.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ In case of no or blank chunk key, the output plugin writes all the matched event
|
12:00:25 ssh.login {"key1":"yay","key2":100} --|
```
{% hint style='info' %}
Blank chunk keys should be used for simple stream transfers (via forward or http plugin) or when the output target destination does not depend on the key. (e.g. stdout)
{% endhint %}

### Tag

Expand Down Expand Up @@ -221,6 +224,11 @@ Buffer chunk keys may be specified empty by using `[]` as the `buffer` section a

This is particularly useful when the output plugin has its own default chunk keys and it needs to disable those.

{% hint style='info' %}
Empty Keys might be useful for CI because it suppress chunk splitting and simplify internal flush behavior.
As a side effect, you can't specify placeholder which uses chunk metadata (e.g timekey, "${tag}" in path and so on)
{% endhint %}

## Placeholders

When the chunk keys are specified, these values can be extracted in configuration parameter values. It depends on whether the plugin applies a method\(`extract_placeholders`\) on configuration values or not.
Expand Down Expand Up @@ -454,6 +462,7 @@ Following are the flushing parameters for chunks to optimize performance \(laten
into chunks
* `flush_interval` \[time\]
* Default: 60s
* The interval time to check whether buffer should be flushed or not. It depends on chunk key or flush mode whether the buffer will be flushed or not, so it doesn't mean always flushed in this duration if the condition is not matched to be flushed.
* `flush_thread_count` \[integer\]
* Default: 1
* The number of threads to flush/write chunks in parallel
Expand Down
11 changes: 11 additions & 0 deletions output/file.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,17 @@ The `path` parameter is used as `<buffer>`'s `path` in this plugin.

Initially, you may see a file which looks like `/path/to/file.%Y%m%d/buffer.b5692238db04045286097f56f361028db.log`. This is an intermediate buffer file \(`b5692238db04045286097f56f361028db` identifies the buffer\). Once the content of the buffer has been completely [flushed](../buffer/file.md), you will see the output file without the trailing identifier.

{% hint style='info' %}
If you want to specific `time`, `tag` or any `record` fields - placeholders in the path, you must specify them as `<buffer>` chunk keys.
(Thus, blank chunk keys `<buffer>` or empty keys `<buffer []>` is not allowed. It causes configuration error.)
{% endhint %}

{% hint style='warning' %}
`path` is implicitly used as file buffer path if file buffer path is not explicitly configured in `<buffer>` section.
In such a case, it must not contain `[` or `]` in the actually evaluated `path`. See [`<buffer>` limitation](../buffer/file.md#limitation) about detail.

{% endhint %}

### `append`

| type | default | version |
Expand Down