⚠️ Repository Archived

Please note that fluent-plugin-scribe is no longer maintained and this repository has been archived.

The Facebook Scribe project has been officially archived by Facebook. As Scribe is no longer used in modern infrastructure, the role of this plugin has come to an end.

Thank you to everyone who has used and contributed to this plugin over the years!

Scribe input plugin for Fluentd

Overview

This is a plugin for fluentd data collector. This plugin adds the Scribe compatible interface to fluentd.

What’s Scribe?

Scribe is a server for aggregating log data streamed in real time from a large number of servers, developed at Facebook.

It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.

What’s Scribe plugin for fluent?

The Scribe plugin for fluentd, which enables fluentd to talk the Scribe protocol. Scribe protocol is defined as follows, in Thrift-IDL format:

enum ResultCode
{
  OK,
  TRY_LATER
}

struct LogEntry
{
  1:  string category,
  2:  string message
}

service scribe extends fb303.FacebookService
{
  ResultCode Log(1: list<LogEntry> messages);
}

The category field is used as fluentd ‘tag’.

How to use?

fluent-plugin-scribe contains both input and output.

Scribe Input

Please add the following configurations to fluent.conf. This allows your Scribe client to import logs through port 1463.

# Scribe input
<source>
  type scribe
  port 1463
</source>

These options are supported.

• port: port number (default: 1463)

• bind: bind address (default: 0.0.0.0)

• server_type: server architecture either in ‘simple’, ‘threaded’, ‘thread_pool’, ‘nonblocking’ (default: nonblocking)

• is_framed: use framed protocol or not (default: true)

• add_prefix: prefix string, added to the tag (default: nil)

• msg_format: format of the messages either in ‘text’, ‘json’, ‘url_param’ (default: text)

Scribe Output

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Scribe server. Note that fluentd conveys semi-structured data while Scribe conveys unstructured data, thus, ‘field_ref’ parameter is prepared to specify which field will be transferred.

# Scribe output
<match *>
  type scribe
  host scribed-host.local
  port 1463
  field_ref message
</match>

These options are supported.

• host: host name or address (default: localhost)

• port: port number (default: 1463)

• field_ref: field name which sent as scribe log message (default: message)

• timeout: thrift protocol timeout (default: 30)

• format_to_json: if true/yes, format entire record as json, and send as message (default: false)

• remove_prefix: prefix string, removed from the tag (default: nil)

For Developers

To run fluentd with this plugin on chaging,

$ bundle # (or 'bundle update')
$ bundle exec fluentd -v -v -v -c example.conf

Then please execute the sample client.

$ bundle exec bin/fluent-scribe-remote

Contributors

• Satoshi Tagomori

• Sadayuki Furuhashi

• Ken Robertson

Copyright

Copyright

Copyright © 2011 Treasure Data, Inc.

License

Apache License, Version 2.0