[[filtering-and-enhancing-data]]
== Filter and enhance data with processors

++++
<titleabbrev>Processors</titleabbrev>
++++

Your use case might require only a subset of the data exported by {beatname_uc},
or you might need to enhance the exported data (for example, by adding
metadata). {beatname_uc} provides a couple of options for filtering and
enhancing exported data.

You can configure each input to include or exclude specific lines or files. This
allows you to specify different filtering criteria for each input. To do this,
you use the `include_lines`, `exclude_lines`, and `exclude_files` options under
the +{beatname_lc}.inputs+ section of the config file (see
<<configuration-{beatname_lc}-options>>). The disadvantage of this approach is
that you need to implement a configuration option for each filtering criteria
that you need.

Another approach (the one described here) is to define processors to configure
global processing across all data exported by {beatname_uc}.


[float]
[[using-processors]]
=== Processors

include::{libbeat-dir}/processors.asciidoc[]

[float]
[[drop-event-example]]
==== Drop event example

The following configuration drops all the DEBUG messages.

[source,yaml]
-----------------------------------------------------
processors:
  - drop_event:
      when:
        regexp:
          message: "^DBG:"
-----------------------------------------------------

To drop all the log messages coming from a certain log file:

[source,yaml]
----------------
processors:
  - drop_event:
      when:
        contains:
          source: "test"
----------------

[float]
[[decode-json-example]]
==== Decode JSON example

In the following example, the fields exported by {beatname_uc} include a
field, `inner`, whose value is a JSON object encoded as a string:

[source,json]
-----------------------------------------------------
{ "outer": "value", "inner": "{\"data\": \"value\"}" }
-----------------------------------------------------

The following configuration decodes the inner JSON object:

["source","yaml",subs="attributes"]
-----------------------------------------------------
{beatname_lc}.inputs:
- type: log
  paths:
    - input.json
  json.keys_under_root: true

processors:
  - decode_json_fields:
      fields: ["inner"]

output.console.pretty: true
-----------------------------------------------------

The resulting output looks something like this:

["source","json",subs="attributes"]
-----------------------------------------------------
{
  "@timestamp": "2016-12-06T17:38:11.541Z",
  "beat": {
    "hostname": "host.example.com",
    "name": "host.example.com",
    "version": "{version}"
  },
  "inner": {
    "data": "value"
  },
  "input": {
    "type": "log",
  },
  "offset": 55,
  "outer": "value",
  "source": "input.json",
  "type": "log"
}
-----------------------------------------------------

include::{libbeat-dir}/processors-using.asciidoc[]
