[[add-kubernetes-metadata]]
=== Add Kubernetes metadata

++++
<titleabbrev>add_kubernetes_metadata</titleabbrev>
++++

The `add_kubernetes_metadata` processor annotates each event with relevant
metadata based on which Kubernetes pod the event originated from.
At startup, it detects an `in_cluster` environment and caches the
Kubernetes-related metadata. Events are only annotated if a valid configuration
is detected. If it's not able to detect a valid Kubernetes configuration,
the events are not annotated with Kubernetes-related metadata.

Each event is annotated with:

* Pod Name
* Pod UID
* Namespace
* Labels

In addition, the node and namespace metadata are added to the pod metadata.

The `add_kubernetes_metadata` processor has two basic building blocks:

* Indexers
* Matchers

Indexers use pod metadata to create unique identifiers for each one of the
pods. These identifiers help to correlate the metadata of the observed pods with
actual events. For example, the `ip_port` indexer can take a Kubernetes pod and
create identifiers for it based on all its `pod_ip:container_port` combinations.

Matchers use information in events to construct lookup keys that match the
identifiers created by the indexers. For example, when the `fields` matcher takes
`["metricset.host"]` as a lookup field, it would construct a lookup key with the
value of the field `metricset.host`. When one of these lookup keys matches with one
of the identifiers, the event is enriched with the metadata of the identified
pod.

ifdef::kubernetes_default_indexers[]
include::{kubernetes_default_indexers}[]
endif::kubernetes_default_indexers[]
ifndef::kubernetes_default_indexers[]
Each Beat can define its own default indexers and matchers which are enabled by
default. For example, Filebeat enables the `container` indexer, which identifies
pod metadata based on all container IDs, and a `logs_path` matcher, which takes
the `log.file.path` field, extracts the container ID, and uses it to retrieve
metadata.
endif::kubernetes_default_indexers[]

You can find more information about the available indexers and matchers, and some
examples in <<kubernetes-indexers-and-matchers>>.

The configuration below enables the processor when {beatname_lc} is run as a pod in
Kubernetes.

[source,yaml,subs="attributes+"]
-------------------------------------------------------------------------------
processors:
  - add_kubernetes_metadata:
ifndef::kubernetes_default_indexers[]
      # Defining indexers and matchers manually is required for {beatname_lc}, for instance:
      #indexers:
      #  - ip_port:
      #matchers:
      #  - fields:
      #      lookup_fields: ["metricset.host"]
endif::kubernetes_default_indexers[]
      #labels.dedot: true
      #annotations.dedot: true
-------------------------------------------------------------------------------

The configuration below enables the processor on a Beat running as a process on
the Kubernetes node.

[source,yaml,subs="attributes+"]
-------------------------------------------------------------------------------
processors:
  - add_kubernetes_metadata:
      host: <hostname>
      # If kube_config is not set, KUBECONFIG environment variable will be checked
      # and if not present it will fall back to InCluster
      kube_config: ${HOME}/.kube/config
ifndef::kubernetes_default_indexers[]
      # Defining indexers and matchers manually is required for {beatname_lc}, for instance:
      #indexers:
      #  - ip_port:
      #matchers:
      #  - fields:
      #      lookup_fields: ["metricset.host"]
endif::kubernetes_default_indexers[]
      #labels.dedot: true
      #annotations.dedot: true
-------------------------------------------------------------------------------

The configuration below has the default indexers and matchers disabled and
enables ones that the user is interested in.

[source,yaml]
-------------------------------------------------------------------------------
processors:
  - add_kubernetes_metadata:
      host: <hostname>
      # If kube_config is not set, KUBECONFIG environment variable will be checked
      # and if not present it will fall back to InCluster
      kube_config: ~/.kube/config
      default_indexers.enabled: false
      default_matchers.enabled: false
      indexers:
        - ip_port:
      matchers:
        - fields:
            lookup_fields: ["metricset.host"]
      #labels.dedot: true
      #annotations.dedot: true
-------------------------------------------------------------------------------

The `add_kubernetes_metadata` processor has the following configuration settings:

`host`:: (Optional) Specify the node to scope {beatname_lc} to in case it
cannot be accurately detected, as when running {beatname_lc} in host network
mode.
`scope`:: (Optional) Specify if the processor should have visibility at the node level or at the entire cluster
level. Possible values are `node` and `cluster`. Scope is `node` by default.
`namespace`:: (Optional) Select the namespace from which to collect the
metadata. If it is not set, the processor collects metadata from all namespaces.
It is unset by default.
`add_resource_metadata`:: (Optional) Specify labels and annotations filters for the extra metadata coming from node and namespace.
 `add_resource_metadata` can be done for `node` or `namespace`. By default all labels are included
while annotations are not. These settings are useful when storing labels and annotations
that require special handling to avoid overloading the storage output. The enrichment of `node` or `namespace` metadata
can be individually disabled by setting `enabled: false`.
Example:

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
      add_resource_metadata:
        namespace:
          include_labels: ["namespacelabel1"]
          #labels.dedot: true
          #annotations.dedot: true
        node:
          include_labels: ["nodelabel2"]
          include_annotations: ["nodeannotation1"]
          #labels.dedot: true
          #annotations.dedot: true
-------------------------------------------------------------------------------------
`kube_config`:: (Optional) Use given config file as configuration for Kubernetes
client. It defaults to `KUBECONFIG` environment variable if present.
`cleanup_timeout`:: (Optional) Specify the time of inactivity before stopping the
running configuration for a container. This is `60s` by default.
`sync_period`:: (Optional) Specify the timeout for listing historical resources.
`default_indexers.enabled`:: (Optional) Enable or disable default pod indexers when you want to specify your own.
`default_matchers.enabled`:: (Optional) Enable or disable default pod matchers when you want to specify your own.
`labels.dedot`:: (Optional) Default to be true. If set to true, then `.` in labels will be replaced with `_`.
`annotations.dedot`:: (Optional) Default to be true. If set to true, then `.` in labels will be replaced with `_`.
