Overview

Fluentd is a powerful tool for log collection and processing. One of its most useful features is the ability to parse logs using regular expressions (regex). This allows you to extract specific information from your logs and structure them in a way that makes them easier to analyze.

In this post, we'll go through some examples of how to use regex with Fluentd to parse logs.

System Environments for this Exercise

The system environment used in the exercise below is as following.

• Rocky Linux release 8.6

• td-agent 4.5.0 (fluentd 1.16.1)

Basic Regex Parsing

Let's start with a basic example. Suppose you have the following log message.

You can use the following Fluentd configuration to parse this log.

The output will look something like this.

This configuration uses the regexp parser to match the log message against the provided regex pattern. The (?<message>...) part of the pattern is a named capture group that extracts the entire log message and assigns it to the message field in the resulting record.

Handling Unmatched Logs

Sometimes your logs may contain messages that don't match your regex pattern. By default, Fluentd will display a warning when this happens. For example, if your logs contain the message "Hello World", Fluentd will issue the following warning.

You can suppress these warnings by adding the emit_invalid_record_to_error false option to your configuration.

Note: This option is not available in the source directive. Please add this option to the filter directive.

Parsing Multiline Logs

Fluentd also supports parsing multiline logs. This is useful when your logs contain messages that span multiple lines. For example, consider the following log message.

You can use the multiline parser to handle this kind of log.

As you can see, the output will look like this.

This configuration uses the multiline parser to match the first line of each log message against the format_firstline pattern. It then uses the format1 pattern to extract the entire message, including any additional lines.

Conclusion

Fluentd's regex parsing capabilities make it a powerful tool for processing logs. Whether you're dealing with simple single line messages or complex multiline logs, Fluentd can help you extract the information you need.

Happy logging!

Need more help? - We are here for you.

In the Fluentd Subscription Network, we will provide you consultancy and professional services to help you run Fluentd and Fluent Bit with confidence by solving your pains. Service desk is also available for your operation and the team is equipped with the Diagtool and the knowledge of tips running Fluent Bit/Fluentd in production. Contact us anytime if you would like to learn more about our service offerings.