fluent-plugin-record-reformer

Fluentd plugin to add or replace fields of a event record

Installation

Use RubyGems:

gem install fluent-plugin-record-reformer

Configuration

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  
  tag reformed.${tag_prefix[-2]}
  <record>
    hostname ${hostname}
    input_tag ${tag}
    last_tag ${tag_parts[-1]}
    message ${message}, yay!
  </record>
</match>

Assume following input is coming (indented):

foo.bar {
  "remove_me":"bar",
  "not_remove_me":"bar",
  "message":"Hello world!"
}

then output becomes as below (indented):

reformed.foo {
  "not_remove_me":"bar",
  "hostname":"YOUR_HOSTNAME",
  "input_tag":"foo.bar",
  "last_tag":"bar",
  "message":"Hello world!, yay!",
}

Configuration (Classic Style)

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  tag reformed.${tag_prefix[-2]}
  
  hostname ${hostname}
  input_tag ${tag}
  last_tag ${tag_parts[-1]}
  message ${message}, yay!
</match>

This results in same, but please note that following option parameters are reserved, so can not be used as a record key.

Option Parameters

• output_tag (obsolete)

  The output tag name. This option is deprecated. Use tag option instead

• tag

  The output tag name.

• remove_keys

  Specify record keys to be removed by a string separated by , (comma) like

    remove_keys message,foo
  

• renew_record bool

  renew_record true creates an output record newly without extending (merging) the input record fields. Default is false.

• keep_keys

  You may want to remain some record fields although you specify renew_record true. Then, specify record keys to be kept by a string separated by , (comma) like

    keep_keys message,foo
  

• enable_ruby bool

  Enable to use ruby codes in placeholders. See Placeholders section. Default is true (just for lower version compatibility).

Placeholders

The keys of input json are available as placeholders. In the above example,

• ${remove_me}
• ${not_remove_me}
• ${message}

shall be available. In addition, following placeholders are reserved:

• ${hostname} Hostname of the running machine

• ${tag} Input tag

• ${time} Time of the event

• ${tags[N]} (Obsolete. Use tag_parts) Input tag splitted by '.'

• ${tag_parts[N]} Input tag splitted by '.' indexed with N such as ${tag_parts[0]}, ${tag_parts[-1]}.

• ${tag_prefix[N]} Tag parts before and on the index N. For example,

    Input tag: prefix.test.tag.suffix
    
    ${tag_prefix[0]}  => prefix
    ${tag_prefix[1]}  => prefix.test
    ${tag_prefix[-2]} => prefix.test.tag
    ${tag_prefix[-1]} => prefix.test.tag.suffix
  

• ${tag_suffix[N]} Tag parts after and on the index N. For example,

    Input tag: prefix.test.tag.suffix
  
    ${tag_suffix[0]}  => prefix.test.tag.suffix
    ${tag_suffix[1]}  => test.tag.suffix
    ${tag_suffix[-2]} => tag.suffix
    ${tag_suffix[-1]} => suffix
  

It is also possible to write a ruby code in placeholders if you set enable_ruby true option, so you may write some codes as

• ${time.strftime('%Y-%m-%dT%H:%M:%S%z')}
• ${tag_parts.last}

but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance.

Relatives

Following plugins look similar:

• fluent-plugin-record-modifier
• fluent-plugin-format
• fluent-plugin-add
• filter_record_transformer is a Fluentd v0.12 built-in plugin which is based on record-reformer.

ChangeLog

See CHANGELOG.md for details.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Copyright

Copyright (c) 2013 - 2015 Naotoshi Seo. See LICENSE for details.