[[elasticsearch-output]]
=== Configure the Elasticsearch output

++++
<titleabbrev>Elasticsearch</titleabbrev>
++++

The Elasticsearch output sends events directly to Elasticsearch using the Elasticsearch HTTP API.

Example configuration:

["source","yaml",subs="attributes"]
----
output.elasticsearch:
  hosts: ["https://myEShost:9200"] <1>
----
<1> To enable SSL, add `https` to all URLs defined under __hosts__.

When sending data to a secured cluster through the `elasticsearch`
output, {beatname_uc} can use any of the following authentication methods:

* Basic authentication credentials (username and password).
* Token-based (API key) authentication.
* Public Key Infrastructure (PKI) certificates.

*Basic authentication:*

["source","yaml",subs="attributes,callouts"]
----
output.elasticsearch:
  hosts: ["https://myEShost:9200"]
  username: "{beat_default_index_prefix}_writer"
  password: "{pwd}"
----

*API key authentication:*

["source","yaml",subs="attributes,callouts"]
----
output.elasticsearch:
  hosts: ["https://myEShost:9200"]
  api_key: "KnR6yE41RrSowb0kQ0HWoA"
----

*PKI certificate authentication:*

["source","yaml",subs="attributes,callouts"]
----
output.elasticsearch:
  hosts: ["https://myEShost:9200"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key"
----

See <<securing-communication-elasticsearch>> for details on each authentication method.

==== Compatibility

This output works with all compatible versions of Elasticsearch. See the
https://www.elastic.co/support/matrix#matrix_compatibility[Elastic Support
Matrix].

==== Configuration options

You can specify the following options in the `elasticsearch` section of the +{beatname_lc}.yml+ config file:

===== `enabled`

The enabled config is a boolean setting to enable or disable the output. If set
to `false`, the output is disabled.

The default value is `true`.


[[hosts-option]]
===== `hosts`

The list of Elasticsearch nodes to connect to. The events are distributed to
these nodes in round robin order. If one node becomes unreachable, the event is
automatically sent to another node. Each Elasticsearch node can be defined as a `URL` or `IP:PORT`.
For example: `http://192.15.3.2`, `https://es.found.io:9230` or `192.24.3.2:9300`.
If no port is specified, `9200` is used.

NOTE: When a node is defined as an `IP:PORT`, the _scheme_ and _path_ are taken from the
<<protocol-option,`protocol`>> and <<path-option,`path`>> config options.

[source,yaml]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] <1>
  protocol: https
  path: /elasticsearch
------------------------------------------------------------------------------

In the previous example, the Elasticsearch nodes are available at `https://10.45.3.2:9220/elasticsearch` and
`https://10.45.3.1:9230/elasticsearch`.

===== `compression_level`

The gzip compression level. Setting this value to `0` disables compression.
The compression level must be in the range of `1` (best speed) to `9` (best compression).

Increasing the compression level will reduce the network usage but will increase the cpu usage.

The default value is `1`.

===== `escape_html`

Configure escaping of HTML in strings. Set to `true` to enable escaping.

The default value is `false`.


===== `worker`

The number of workers per configured host publishing events to Elasticsearch. This
is best used with load balancing mode enabled. Example: If you have 2 hosts and
3 workers, in total 6 workers are started (3 for each host).

The default value is `1`.

===== `api_key`

Instead of using a username and password, you can use API keys to secure communication
with {es}. The value must be the ID of the API key and the API key joined by a colon: `id:api_key`.

See <<beats-api-keys>> for more information.

===== `username`

The basic authentication username for connecting to Elasticsearch.

This user needs the privileges required to publish events to {es}.
To create a user like this, see <<privileges-to-publish-events>>.

===== `password`

The basic authentication password for connecting to Elasticsearch.

===== `parameters`

Dictionary of HTTP parameters to pass within the url with index operations.

[[protocol-option]]
===== `protocol`

The name of the protocol Elasticsearch is reachable on. The options are:
`http` or `https`. The default is `http`. However, if you specify a URL for
<<hosts-option,`hosts`>>, the value of `protocol` is overridden by whatever scheme you
specify in the URL.

[[path-option]]
===== `path`

An HTTP path prefix that is prepended to the HTTP API calls. This is useful for
the cases where Elasticsearch listens behind an HTTP reverse proxy that exports
the API under a custom prefix.

===== `headers`

Custom HTTP headers to add to each request created by the Elasticsearch output.
Example:

[source,yaml]
------------------------------------------------------------------------------
output.elasticsearch.headers:
  X-My-Header: Header contents
------------------------------------------------------------------------------

It is possible to specify multiple header values for the same header
name by separating them with a comma.


===== `proxy_disable`

If set to `true` all proxy settings, including `HTTP_PROXY` and `HTTPS_PROXY`
variables are ignored.


===== `proxy_url`

The URL of the proxy to use when connecting to the Elasticsearch servers. The
value must be a complete URL. If a value is not specified through the configuration file
then proxy environment variables are used. See the
https://golang.org/pkg/net/http/#ProxyFromEnvironment[Go documentation]
for more information about the environment variables.


===== `proxy_headers`

Additional headers to send to proxies during CONNECT requests.

[[index-option-es]]
===== `index`

// Begin exclude for APM Server docs
ifndef::apm-server[]
The index name to write events to when you're using daily indices. The default is
+"{beatname_lc}-%{[{beat_version_key}]}-%{+yyyy.MM.dd}"+, for example,
+"{beatname_lc}-{version}-{localdate}"+. If you change this setting, you also
need to configure the `setup.template.name` and `setup.template.pattern` options
(see <<configuration-template>>).

ifndef::no_dashboards[]
If you are using the pre-built Kibana
dashboards, you also need to set the `setup.dashboards.index` option (see
<<configuration-dashboards>>).
endif::no_dashboards[]

ifndef::no_ilm[]
When <<ilm,index lifecycle management (ILM)>> is enabled, the default `index` is
+"{beatname_lc}-%{[{beat_version_key}]}-%{+yyyy.MM.dd}-%{index_num}"+, for example,
+"{beatname_lc}-{version}-{localdate}-000001"+. Custom `index` settings are ignored
when ILM is enabled. If you’re sending events to a cluster that supports index
lifecycle management, see <<ilm>> to learn how to change the index name.
endif::no_ilm[]

You can set the index dynamically by using a format string to access any event
field. For example, this configuration uses a custom field, `fields.log_type`,
to set the index:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "%{[fields.log_type]}-%{[{beat_version_key}]}-%{+yyyy.MM.dd}" <1>
------------------------------------------------------------------------------

<1> We recommend including +{beat_version_key}+ in the name to avoid mapping issues
when you upgrade.

With this configuration, all events with `log_type: normal` are sent to an
index named +normal-{version}-{localdate}+, and all events with
`log_type: critical` are sent to an index named
+critical-{version}-{localdate}+.
endif::apm-server[]
// End exclude for APM Server docs

// Start include for APM Server docs
ifdef::apm-server[]
The index name to write events to when you're using daily indices. The default is
+"apm-%{[{beat_version_key}]}-{type}-%{+yyyy.MM.dd}"+ (for example,
+"apm-{version}-transaction-{localdate}"+). If you change this setting,
you need to configure the `setup.template.name` and `setup.template.pattern` options
(see <<configuration-template>>).

When <<ilm,index lifecycle management (ILM)>> is enabled, the default `index` is
+"apm-%{[{beat_version_key}]}-{type}-%{index_num}"+ (for example,
+"apm-{version}-transaction-000001"+). **Defining a custom `index` here will disable <<ilm>>**.

You can set the index dynamically by using a format string to access any event
field. For example, this configuration uses the field, `processor.event` to separate
events into different indices:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}\" <1>
------------------------------------------------------------------------------
<1> +{beat_version_key}+ is a field managed by Beats that is added to every document;
It holds the current version of APM Server. We recommend including
+{beat_version_key}+ in the index name to avoid mapping issues when you upgrade
{beatname_uc}.

endif::apm-server[]
// End include for APM Server docs

TIP: To learn how to add custom fields to events, see the
<<libbeat-configuration-fields,`fields`>> option.

See the <<indices-option-es,`indices`>> setting for other ways to set the index
dynamically.

[[indices-option-es]]
===== `indices`

An array of index selector rules. Each rule specifies the index to use for
events that match the rule. During publishing, {beatname_uc} uses the first
matching rule in the array. Rules can contain conditionals, format string-based
fields, and name mappings. If the `indices` setting is missing or no rule
matches, the <<index-option-es,`index`>> setting is used.

ifndef::no_ilm[]
Similar to `index`, defining custom `indices` will disable <<ilm>>.
endif::no_ilm[]

Rule settings:

*`index`*:: The index format string to use. If this string contains field
references, such as `%{[fields.name]}`, the fields must exist, or the rule fails.

*`mappings`*:: A dictionary that takes the value returned by `index` and maps it
to a new name.

*`default`*:: The default string value to use if `mappings` does not find a
match.

*`when`*:: A condition that must succeed in order to execute the current rule.
ifndef::no-processors[]
All the <<conditions,conditions>> supported by processors are also supported
here.
endif::no-processors[]

ifndef::apm-server[]
The following example sets the index based on whether the `message` field
contains the specified string:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  indices:
    - index: "warning-%{[{beat_version_key}]}-%{+yyyy.MM.dd}"
      when.contains:
        message: "WARN"
    - index: "error-%{[{beat_version_key}]}-%{+yyyy.MM.dd}"
      when.contains:
        message: "ERR"
------------------------------------------------------------------------------


This configuration results in indices named +warning-{version}-{localdate}+
and +error-{version}-{localdate}+ (plus the default index if no matches are
found).

The following example sets the index by taking the name returned by the `index`
format string and mapping it to a new name that's used for the index:

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  indices:
    - index: "%{[fields.log_type]}"
      mappings:
        critical: "sev1"
        normal: "sev2"
      default: "sev3"
------------------------------------------------------------------------------


This configuration results in indices named `sev1`, `sev2`, and `sev3`.

The `mappings` setting simplifies the configuration, but is limited to string
values. You cannot specify format strings within the mapping pairs.
endif::apm-server[]

ifdef::apm-server[]
The following example sets the index based on whether the `processor.event` field
contains the specified string:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  indices:
   - index: "apm-%{[observer.version]}-sourcemap"
      when.contains:
        processor.event: "sourcemap"

   - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"

   - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"

   - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"

   - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"

   - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"
------------------------------------------------------------------------------

NOTE: `observer` refers to {beatname_uc}. We recommend including
+{beat_version_key}+ in the name to avoid mapping issues when you upgrade
{beatname_uc}.

This is the default configuration for {beatname_uc} when ILM is disabled, and results in indices
named in the following format: +"apm-%{[{beat_version_key}]}-{type}-%{+yyyy.MM.dd}"+
For example: +"apm-{version}-transaction-{localdate}"+.

The following example sets the index by taking the name returned by the `index`
format string and mapping it to a new name that's used for the index:

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  indices:
    - index: "%{[processor.event]}"
      mappings:
        sourcemap:    "apm-sourcemap"
        error:        "apm-error"
        transaction:  "apm-transaction"
        span:         "apm-span"
        metric:       "apm-metric"
        onboarding:   "apm-onboarding"
      default:        "apm"
------------------------------------------------------------------------------

This configuration results in indices named `apm-sourcemap`, `apm-error`, etc.

The `mappings` setting simplifies the configuration, but is limited to string
values. You cannot specify format strings within the mapping pairs.
endif::apm-server[]

//TODO: MOVE ILM OPTIONS TO APPEAR LOGICALLY BASED ON LOCATION IN THE YAML FILE.

ifndef::no_ilm[]
[[ilm-es]]
===== `ilm`

Configuration options for index lifecycle management.

See <<ilm>> for more information.
endif::no_ilm[]

ifndef::no-pipeline[]
[[pipeline-option-es]]
===== `pipeline`

A format string value that specifies the ingest pipeline to write events to.

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipeline: my_pipeline_id
------------------------------------------------------------------------------

For more information, see <<configuring-ingest-node>>.

ifndef::apm-server[]
You can set the ingest pipeline dynamically by using a format string to
access any event field. For example, this configuration uses a custom field,
`fields.log_type`, to set the pipeline for each event:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipeline: "%{[fields.log_type]}_pipeline"
------------------------------------------------------------------------------

With this configuration, all events with `log_type: normal` are sent to a pipeline
named `normal_pipeline`, and all events with `log_type: critical` are sent to a
pipeline named `critical_pipeline`.
endif::apm-server[]

ifdef::apm-server[]
You can set the ingest pipeline dynamically by using a format string to
access any event field. For example, this configuration uses the field,
`processor.event`, to set the pipeline for each event:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipeline: "%{[processor.event]}_pipeline"
------------------------------------------------------------------------------

With this configuration, all events with `processor.event: transaction` are sent to a pipeline
named `transaction_pipeline`. Similarly, all events with `processor.event: error` are sent to a
pipeline named `error_pipeline`.

The default pipeline is `apm`. To disable this, or any other pipeline, set
`output.elasticsearch.pipeline: _none`.
endif::apm-server[]

TIP: To learn how to add custom fields to events, see the
<<libbeat-configuration-fields,`fields`>> option.

See the <<pipelines-option-es,`pipelines`>> setting for other ways to set the
ingest pipeline dynamically.

[[pipelines-option-es]]
===== `pipelines`

An array of pipeline selector rules. Each rule specifies the ingest
pipeline to use for events that match the rule. During publishing, {beatname_uc}
uses the first matching rule in the array. Rules can contain conditionals,
format string-based fields, and name mappings. If the `pipelines` setting is
missing or no rule matches, the <<pipeline-option-es,`pipeline`>> setting is
used.

Rule settings:

*`pipeline`*:: The pipeline format string to use. If this string contains field
references, such as `%{[fields.name]}`, the fields must exist, or the rule
fails.

*`mappings`*:: A dictionary that takes the value returned by `pipeline` and maps
it to a new name.

*`default`*:: The default string value to use if `mappings` does not find a
match.

*`when`*:: A condition that must succeed in order to execute the current rule.
ifndef::no-processors[]
All the <<conditions,conditions>> supported by processors are also supported
here.
endif::no-processors[]

ifndef::apm-server[]
The following example sends events to a specific pipeline based on whether the
`message` field contains the specified string:

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipelines:
    - pipeline: "warning_pipeline"
      when.contains:
        message: "WARN"
    - pipeline: "error_pipeline"
      when.contains:
        message: "ERR"
------------------------------------------------------------------------------


The following example sets the pipeline by taking the name returned by the
`pipeline` format string and mapping it to a new name that's used for the
pipeline:

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipelines:
    - pipeline: "%{[fields.log_type]}"
      mappings:
        critical: "sev1_pipeline"
        normal: "sev2_pipeline"
      default: "sev3_pipeline"
------------------------------------------------------------------------------


With this configuration, all events with `log_type: critical` are sent to
`sev1_pipeline`, all events with `log_type: normal` are sent to a
`sev2_pipeline`, and all other events are sent to `sev3_pipeline`.
endif::apm-server[]

ifdef::apm-server[]
The following example sends events to a specific pipeline based on whether the
`processor.event` field contains the specified string:

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipelines:
    - pipeline: "sourcemap_pipeline"
      when.contains:
        processor.event: "sourcemap"

    - pipeline: "error_pipeline"
      when.contains:
        processor.event: "error"

    - pipeline: "transaction_pipeline"
      when.contains:
        processor.event: "transaction"

    - pipeline: "span_pipeline"
      when.contains:
        processor.event: "span"

    - pipeline: "metric_pipeline"
      when.contains:
        processor.event: "metric"

    - pipeline: "onboarding_pipeline"
      when.contains:
        processor.event: "onboarding"
------------------------------------------------------------------------------


The following example sets the pipeline by taking the name returned by the
`pipeline` format string and mapping it to a new name that's used for the
pipeline:

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipelines:
    - pipeline: "%{[processor.event]}"
      mappings:
        sourcemap:    "sourcemap_pipeline"
        error:        "error_pipeline"
        transaction:  "transaction_pipeline"
        span:         "span_pipeline"
        metric:       "metric_pipeline"
        onboarding:   "onboarding_pipeline"
      default: "apm_pipeline"
------------------------------------------------------------------------------

With this configuration, all events with `processor.event: transaction` are sent to a pipeline
named `transaction_pipeline`, all events with `processor.event: error` are sent to a
pipeline named `error_pipeline`, etc.

NOTE: Defining any pipeline will deactivate the default `apm` pipeline.
endif::apm-server[]

For more information about ingest pipelines, see
<<configuring-ingest-node>>.

endif::[]

===== `max_retries`

ifdef::ignores_max_retries[]
{beatname_uc} ignores the `max_retries` setting and retries indefinitely.
endif::[]

ifndef::ignores_max_retries[]
The number of times to retry publishing an event after a publishing failure.
After the specified number of retries, the events are typically dropped.

Set `max_retries` to a value less than 0 to retry until all events are published.

The default is 3.
endif::[]


===== `bulk_max_size`

The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50.

Events can be collected into batches. When using the memory queue with `queue.mem.flush.min_events`
set to a value greater than `1`, the maximum batch is is the value of `queue.mem.flush.min_events`.
{beatname_uc} will split batches read from the queue which are larger than `bulk_max_size` into
multiple batches.

Specifying a larger batch size can improve performance by lowering the overhead of sending events.
However big batch sizes can also increase processing times, which might result in
API errors, killed connections, timed-out publishing requests, and, ultimately, lower
throughput.

Setting `bulk_max_size` to values less than or equal to 0 disables the
splitting of batches. When splitting is disabled, the queue decides on the
number of events to be contained in a batch.

===== `backoff.init`

The number of seconds to wait before trying to reconnect to Elasticsearch after
a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to
reconnect. If the attempt fails, the backoff timer is increased exponentially up
to `backoff.max`. After a successful connection, the backoff timer is reset. The
default is `1s`.


===== `backoff.max`

The maximum number of seconds to wait before attempting to connect to
Elasticsearch after a network error. The default is `60s`.

===== `timeout`

The http request timeout in seconds for the Elasticsearch request. The default is 90.

===== `ssl`

Configuration options for SSL parameters like the certificate authority to use
for HTTPS-based connections. If the `ssl` section is missing, the host CAs are used for HTTPS connections to
Elasticsearch.

See the <<securing-communication-elasticsearch,secure communication with {es}>> guide
or <<configuration-ssl,SSL configuration reference>> for more information.

===== `kerberos`

Configuration options for Kerberos authentication.

See <<configuration-kerberos>> for more information.

===== `non_indexable_policy`

Specifies the behavior when the elasticsearch cluster explicitly rejects documents, for example on mapping conflicts.

====== `drop`
The default behaviour, when an event is explicitly rejected by elasticsearch it is dropped.

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  non_indexable_policy.drop: ~
------------------------------------------------------------------------------

====== `dead_letter_index`

beta[]

On an explicit rejection, this policy will retry the event in the next batch. However, the target index will change
to index specified. In addition, the structure of the event will be change to the following fields:

message:: Contains the escaped json of the original event.
error.type:: Contains the status code
error.message:: Contains status returned by elasticsearch, describing the reason

`index`:: The index to send rejected events to.

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  non_indexable_policy.dead_letter_index:
    index: "my-dead-letter-index"
------------------------------------------------------------------------------
