//////////////////////////////////////////////////////////////////////////
//// This content is shared by all Elastic Beats. Make sure you keep the
//// descriptions here generic enough to work for all Beats that include
//// this file. When using cross references, make sure that the cross
//// references resolve correctly for any files that include this one.
//// Use the appropriate variables defined in the index.asciidoc file to
//// resolve Beat names: beatname_uc and beatname_lc.
//// Use the following include to pull this content into a doc file:
//// include::../../libbeat/docs/shared-config-ingest.asciidoc[]
//////////////////////////////////////////////////////////////////////////

[[configuring-ingest-node]]
== Parse data using an ingest pipeline

When you use {es} for output, you can configure {beatname_uc} to use
an {ref}/ingest.html[ingest pipeline] to pre-process documents before the actual
indexing takes place in {es}.
ifndef::no-output-logstash[]
An ingest pipeline is a convenient processing option when you want to do some
extra processing on your data, but you do not require the full power of {ls}.
endif::[]
For example, you can create an ingest pipeline
in {es} that consists of one processor that removes a field in a
document followed by another processor that renames a field.

After defining the pipeline in {es}, you simply configure {beatname_uc}
to use the pipeline. To configure {beatname_uc}, you specify the pipeline ID in
the `parameters` option under `elasticsearch` in the +{beatname_lc}.yml+ file:

[source,yaml]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: my_pipeline_id
------------------------------------------------------------------------------

For example, let's say that you've defined the following pipeline in a file
named `pipeline.json`:

[source,json]
------------------------------------------------------------------------------
{
    "description": "Test pipeline",
    "processors": [
        {
            "lowercase": {
                "field": "agent.name"
            }
        }
    ]
}
------------------------------------------------------------------------------

To add the pipeline in {es}, you would run:

[source,shell]
------------------------------------------------------------------------------
curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_ingest/pipeline/test-pipeline' -d@pipeline.json
------------------------------------------------------------------------------

Then in the +{beatname_lc}.yml+ file, you would specify:

[source,yaml]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: "test-pipeline"
------------------------------------------------------------------------------

When you run {beatname_uc}, the value of `agent.name` is converted to lowercase before indexing.

For more information about defining a pre-processing pipeline, see the
{ref}/ingest.html[ingest pipeline] documentation.
