Array-splitter Plugin for Fluentd

Overview

We are excited to announce the release of fluent-plugin-array-splitter, a new plugin to the Fluentd family, developed by our team. This innovation was inspired by the recurring need to decompose array values embedded in JSON formatted log records, which is often a bottleneck in data parsing and analysis workflows.

With the fluent-plugin-array-splitter, you can now effortlessly break down these arrays into individual records, paving the way for more streamlined data processing and analysis.

Installation

$ gem install fluent-plugin-array-splitter

Example Configuration

Case1: Basic Usage

The array_key option is essential for specifying the name of the field whose values you want to split. It is required for this plugin to work. Not specifying this option, or leaving it empty, will result in an error, as the plugin needs to know which field to target for splitting the array values.

For example, if your data contains an array in the field named events, you must set array_key to that field name to enable the splitting process.

<filter **>
  @type array_splitter
  array_key message
</filter>

Input:

{"message": ["value1", "value2"]}

Output:

{"message": "value1"}
{"message": "value2"}

It will output two separate records from the array values.

Make sure that array_key is set to the correct field name to avoid processing errors and to make the plugin work as expected.

Example 2: Renaming Key

The key_name option allows you to rename the field used to access the array. This is useful if the array elements you want to split do not conform to a key/value pair format. By using the key_name option, you can assign a new field name to the array, making it easier to identify within your data structure.

For example, if your original data structure has an array labeled data that does not contain key/value pairs, you can use the key_name option as follows

<filter **>
  @type array_splitter
  array_key message
  key_name new_key
</filter>

Input:

{"message": ["value1", "value2"]}

Output:

{"new_key": "value1"}
{"new_key": "value2"}

This configuration changes the field name from “message” to “new_key” for each split value, simplifying the data structure for further processing steps.

Example 3: Handling Key/Value Pairs in an Array

If the array contains elements that are key/value pairs, this plugin will separate each pair into its own message. 
This means that if your array consists of multiple key/value pairs, the splitter will process each as a separate,  individual record, allowing for more granular analysis.

For example, if your data structure has key/value pairs in an array, you can use the array_key option as follows

<filter **>
  @type array_splitter
  array_key message
</filter>

Input:

{"message": [{"key1": "value1"}, {"key2": "value2"}]}

Output:

{"key1": "value1"}
{"key2": "value2"}

Now, each key/value pair is in its own line, making it much simpler to read and use.

---

The fluent-plugin-array-splitter is now available and can be a valuable addition to your Fluentd setup, especially if you deal with log records containing array values. Give it a try and feel free to contribute or report issues on the GitHub repository.

---

Need some help? - We are here for you.

In the Fluentd Subscription Network, we will provide you consultancy and professional services to help you run Fluentd and Fluent Bit with confidence by solving your pains. Service desk is also available for your operation and the team is equipped with the Diagtool and the knowledge of tips running Fluent Bit/Fluentd in production. Contact us anytime if you would like to learn more about our service offerings!