[[configuration-autodiscover]]
== Autodiscover

When you run applications on containers, they become moving targets to the monitoring system. Autodiscover
allows you to track them and adapt settings as changes happen. By defining configuration templates, the
autodiscover subsystem can monitor services as they start running.

You define autodiscover settings in the  +{beatname_lc}.autodiscover+ section of the +{beatname_lc}.yml+
config file. To enable autodiscover, you specify a list of providers.

[float]
=== Providers

Autodiscover providers work by watching for events on the system and translating those events into internal autodiscover
events with a common format. When you configure the provider, you can optionally use fields from the autodiscover event
to set conditions that, when met, launch specific configurations.

On start, {beatname_uc} will scan existing containers and launch the proper configs for them. Then it will watch for new
start/stop events. This ensures you don't need to worry about state, but only define your desired configs.

[float]
===== Docker

The Docker autodiscover provider watches for Docker containers to start and stop.


These are the fields available within config templating. The `docker.*` fields will be available on each emitted event.
event:

  * host
  * port
  * docker.container.id
  * docker.container.image
  * docker.container.name
  * docker.container.labels


For example:

[source,yaml]
-------------------------------------------------------------------------------------
{
  "host": "10.4.15.9",
  "port": 6379,
  "docker": {
    "container": {
      "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51"
      "name": "redis",
      "image": "redis:3.2.11",
      "labels": {
        "io.kubernetes.pod.namespace": "default"
        ...
      }
    }
  }
}
-------------------------------------------------------------------------------------

You can define a set of configuration templates to be applied when the condition matches an event. Templates define
a condition to match on autodiscover events, together with the list of configurations to launch when this condition
happens.

Conditions match events from the provider. Providers use the same format for <<conditions>> that
processors use.

Configuration templates can contain variables from the autodiscover event. They can be accessed under the `data` namespace.
For example, with the example event, "`${data.port}`" resolves to `6379`.

include::../../{beatname_lc}/docs/autodiscover-docker-config.asciidoc[]


ifeval::["{beatname_lc}"=="filebeat"]
[WARNING]
=======================================
When using autodiscover, you have to be careful when defining config templates, especially if they are
reading from places holding information for several containers. For instance, under this file structure:

`/mnt/logs/<container_id>/*.log`

You can define a config template like this:

**Wrong settings**:

[source,yaml]
-------------------------------------------------------------------------------------
autodiscover.providers:
  - type: docker
    templates:
      - condition.contains:
          docker.container.image: nginx
        config:
          - type: log
            paths:
              - "/mnt/logs/*/*.log"
-------------------------------------------------------------------------------------

That would read all the files under the given path several times (one per nginx container). What you really
want is to scope your template to the container that matched the autodiscover condition. Good settings:

[source,yaml]
-------------------------------------------------------------------------------------
autodiscover.providers:
  - type: docker
    templates:
      - condition.contains:
          docker.container.image: nginx
        config:
          - type: log
            paths:
              - "/mnt/logs/${data.docker.container.id}/*.log"
-------------------------------------------------------------------------------------

It has the following settings:

`host`:: (Optional) Docker socket (UNIX or TCP socket). It uses
`unix:///var/run/docker.sock` by default.
`ssl`:: (Optional) SSL configuration to use when connecting to the Docker
socket.
`cleanup_timeout`:: (Optional) Specify the time of inactivity before stopping the
running configuration for a container,
ifeval::["{beatname_lc}"=="filebeat"]
 60s by default.
endif::[]
ifeval::["{beatname_lc}"!="filebeat"]
 disabled by default.
endif::[]
`labels.dedot`:: (Optional) Default to be false. If set to true, replace dots in
 labels with `_`.

=======================================
endif::[]

[float]
===== Kubernetes

The Kubernetes autodiscover provider watches for Kubernetes nodes, pods, services to start, update, and stop.

These are the fields available within config templating. The `kubernetes.*` fields will be available on each emitted event.

[float]
====== Generic fields:
  * host
  * port (if exposed)
  * kubernetes.labels
  * kubernetes.annotations

[float]
====== Pod specific:
  * kubernetes.container.id
  * kubernetes.container.image
  * kubernetes.container.name
  * kubernetes.namespace
  * kubernetes.node.name
  * kubernetes.pod.name
  * kubernetes.pod.uid

[float]
====== Node specific:
  * kubernetes.node.name
  * kubernetes.node.uid

[float]
====== Service specific:
  * kubernetes.namespace
  * kubernetes.service.name
  * kubernetes.service.uid
  * kubernetes.annotations

If the `include_annotations` config is added to the provider config, then the list of annotations present in the config
are added to the event.

If the `include_labels` config is added to the provider config, then the list of labels present in the config
will be added to the event.

If the `exclude_labels` config is added to the provider config, then the list of labels present in the config
will be excluded from the event.

if the `labels.dedot` config is set to be `true` in the provider config, then `.` in labels will be replaced with `_`.
By default it is `true`.

if the `annotations.dedot` config is set to be `true` in the provider config, then `.` in annotations will be replaced
with `_`. By default it is `true`.


For example:

[source,yaml]
-------------------------------------------------------------------------------------
{
  "host": "172.17.0.21",
  "port": 9090,
  "kubernetes": {
    "container": {
      "id": "bb3a50625c01b16a88aa224779c39262a9ad14264c3034669a50cd9a90af1527",
      "image": "prom/prometheus",
      "name": "prometheus"
    },
    "labels": {
      "project": "prometheus",
      ...
    },
    "namespace": "default",
    "node": {
      "name": "minikube"
    },
    "pod": {
      "name": "prometheus-2657348378-k1pnh"
    }
  },
}
-------------------------------------------------------------------------------------

The configuration of templates and conditions is similar to that of the Docker provider. Configuration templates can
contain variables from the autodiscover event. They can be accessed under data namespace.

The `kubernetes` autodiscover provider has the following configuration settings:

`node`:: (Optional) Specify the node to scope {beatname_lc} to in case it
  cannot be accurately detected, as when running {beatname_lc} in host network
  mode.
`namespace`:: (Optional) Select the namespace from which to collect the
  metadata. If it is not set, the processor collects metadata from all
  namespaces. It is unset by default. The namespace configuration only applies to
  kubernetes resources that are namespace scoped.
`cleanup_timeout`:: (Optional) Specify the time of inactivity before stopping the
running configuration for a container,
ifeval::["{beatname_lc}"=="filebeat"]
 60s by default.
endif::[]
ifeval::["{beatname_lc}"!="filebeat"]
 disabled by default.
endif::[]
`kube_config`:: (Optional) Use given config file as configuration for Kubernetes
  client. If kube_config is not set, KUBECONFIG environment variable will be
  checked and if not present it will fall back to InCluster.
`resource`:: (Optional) Select the resource to do discovery on. Currently supported
  Kubernetes resources are `pod`, `service` and `node`. If not configured `resource`
  defaults to `pod`.
`scope`:: (Optional) Specify at what level autodiscover needs to be done at. `scope` can
  either take `node` or `cluster` as values. `node` scope allows discovery of resources in
  the specified node. `cluster` scope allows cluster wide discovery. Only `pod` and `node` resources
  can be discovered at node scope.
`add_resource_metadata`:: (Optional) Specify labels and annotations filters for the extra metadata coming from Node and Namespace.
 `add_resource_metadata` can be done for `node` or `namespace`. By default all labels will be included
  while annotations are not added by default. This settings are useful when labels' and annotations'
  storing requires special handling to avoid overloading the storage output. The enrichment of `node` or `namespace` metadata
  can be individually disabled by setting `enabled: false`. If resource is `pod` and it is created from a `deployment`, by default
  the deployment name is added, this can be disabled by setting `deployment: false`.
  Example:

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
      add_resource_metadata:
        namespace:
          include_labels: ["namespacelabel1"]
        node:
          include_labels: ["nodelabel2"]
          include_annotations: ["nodeannotation1"]
        deployment: false
-------------------------------------------------------------------------------------

`unique`:: (Optional) Defaults to `false`. Marking an autodiscover provider as unique results into
  making the provider to enable the provided templates only when it will gain the leader lease.
  This setting can only be combined with `cluster` scope. When `unique` is enabled enabled, `resource`
  and `add_resource_metadata` settings are not taken into account.
`leader_lease`:: (Optional) Defaults to `{beatname_lc}-cluster-leader`. This will be name of the lock lease.
  One can monitor the status of the lease with `kubectl describe lease beats-cluster-leader`.
  Different Beats that refer to the same leader lease will be competitors in holding the lease
  and only one will be elected as leader each time.

ifeval::["{beatname_lc}"=="metricbeat"]
Example:

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
metricbeat.autodiscover:
  providers:
    - type: kubernetes
      scope: cluster
      node: ${NODE_NAME}
      unique: true
      identifier: leader-election-metricbeat
      templates:
        - config:
            - module: kubernetes
              hosts: ["kube-state-metrics:8080"]
              period: 10s
              add_metadata: true
              metricsets:
                - state_node
-------------------------------------------------------------------------------------

The above configuration when deployed on one or more Metribceat instances will enable `state_node`
metricset only for the Metricbeat instance that will gain the leader lease/lock. With this deployment
strategy we can ensure that cluster-wide metricsets are only enabled by one Beat instance when
deploying a Beat as DaemonSet.
endif::[]

include::../../{beatname_lc}/docs/autodiscover-kubernetes-config.asciidoc[]

ifdef::autodiscoverJolokia[]
[float]
===== Jolokia

The Jolokia autodiscover provider uses Jolokia Discovery to find agents running
in your host or your network.

Jolokia Discovery mechanism is supported by any Jolokia agent since version
1.2.0, it is enabled by default when Jolokia is included in the application as
a JVM agent, but disabled in other cases as the OSGI or WAR (Java EE) agents.
In any case, this feature is controlled with two properties:

  * `discoveryEnabled`, to enable the feature
  * `discoveryAgentUrl`, if set, this is the URL announced by the agent when
    being discovered, setting this parameter implicitly enables the feature

There are multiple ways of setting these properties, and they can vary from
application to application, please refer to the documentation of your
application to find the more suitable way to set them in your case.

Jolokia Discovery is based on UDP multicast requests. Agents join the multicast
group 239.192.48.84, port 24884, and discovery is done by sending queries to
this group. You have to take into account that UDP traffic between {beatname_uc}
and the Jolokia agents has to be allowed. Also notice that this multicast
address is in the 239.0.0.0/8 range, that is reserved for private use within an
organization, so it can only be used in private networks.

These are the available fields during within config templating. The `jolokia.*` fields will be available on each emitted event.

  * jolokia.agent.id
  * jolokia.agent.version
  * jolokia.secured
  * jolokia.server.product
  * jolokia.server.vendor
  * jolokia.server.version
  * jolokia.url

The configuration of this provider consists in a set of network interfaces, as
well as a set of templates as in other providers. The network interfaces will be
the ones used for discovery probes, they have these settings:

`name`:: the name of the interface (e.g. `br0`), it can contain a wildcard
  as suffix to apply the same settings to multiple network interfaces of
  the same type (e.g. `br*`).
`interval`:: time between probes (defaults to 10s)
 `grace_period`:: time since the last reply to consider an instance stopped
  (defaults to 30s)
`probe_timeout`:: max time to wait for responses since a probe is sent
  (defaults to 1s)

include::../../{beatname_lc}/docs/autodiscover-jolokia-config.asciidoc[]
endif::autodiscoverJolokia[]

ifdef::autodiscoverAWSELB[]
[float]
===== Amazon ELBs

*Note: This provider is experimental*

The Amazon ELB autodiscover provider discovers https://aws.amazon.com/elasticloadbalancing/[ELBs] and their listeners. This is useful when you don't want to connect
directly to a service, but rather to the ELB fronting a pool of services.

This provider will yield one config block per ELB Listener. So, if you have one ELB exposing both ports 80 and 443, it
will generate two configs, one for each port. Keep in mind that the beat will de-duplicate configs. So, if the generated
configs are the same only one will actually run.

This provider will load AWS credentials using the standard AWS environment variables and shared credentials files see https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html[Best Practices for Managing AWS Access Keys] for more information. If you do not wish to use these, you may explicitly set the `access_key_id` and `secret_access_key` variables.

These are the available fields during within config templating. The `aws.elb.*` fields will be available on each emitted event.

  * host
  * port

  * cloud.availability_zone
  * cloud.provider
  * cloud.region

  * aws.elb.listener_arn
  * aws.elb.load_balancer_arn
  * aws.elb.protocol
  * aws.elb.type
  * aws.elb.scheme
  * aws.elb.availability_zones
  * aws.elb.created
  * aws.elb.state.code
  * aws.elb.state.reason
  * aws.elb.ip_address_type
  * aws.elb.security_groups
  * aws.elb.vpc_id
  * aws.elb.ssl_policy

include::../../{beatname_lc}/docs/autodiscover-aws-elb-config.asciidoc[]

This autodiscover provider takes our standard <<aws-credentials-config,AWS credentials options>>.

[id="aws-credentials-config"]
include::../../x-pack/libbeat/docs/aws-credentials-config.asciidoc[]

endif::autodiscoverAWSELB[]

ifdef::autodiscoverNomad[]
[float]
===== Nomad

experimental[]

The Nomad autodiscover provider watches for Nomad jobs to start, update, and stop.

These are the available fields during config templating. The `nomad.*` fields will be available
on each emitted event.

* nomad.allocation.id
* nomad.allocation.name
* nomad.allocation.status
* nomad.datacenter
* nomad.job.name
* nomad.job.type
* nomad.namespace
* nomad.region
* nomad.task.name
* nomad.task.service.canary_tags
* nomad.task.service.name
* nomad.task.service.tags

If the `include_labels` config is added to the provider config, then the list of labels present in
the config will be added to the event.

If the `exclude_labels` config is added to the provider config, then the list of labels present in
the config will be excluded from the event.

if the `labels.dedot` config is set to be `true` in the provider config, then `.` in labels will be
replaced with `_`.

For example:

[source,yaml]
-------------------------------------------------------------------------------------
{
  ...
  "region": "europe",
  "allocation": {
    "name": "coffeshop.api[0]",
    "id": "35eba07f-e5e4-20ac-6def-85117bee6efb",
    "status": "running"
  },
  "datacenters": [
    "europe-west4"
  ],
  "namespace": "default",
  "job": {
    "type": "service",
    "name": "coffeshop"
  },
  "task": {
    "service": {
      "name": [
        "coffeshop"
      ],
      "tags": [
        "coffeshop",
        "nginx"
      ],
      "canary_tags": [
        "coffeshop"
      ]
    },
    "name": "api"
  },
  ...
}
-------------------------------------------------------------------------------------

The configuration of templates and conditions is similar to that of the Docker provider.
Configuration templates can contain variables from the autodiscover event. They can be accessed under
`data` namespace.

The `nomad` autodiscover provider has the following configuration settings:

`address`:: (Optional) Specify the address of the Nomad agent. By default it will try to talk to a
  Nomad agent running locally (`http://127.0.0.1:4646`).

`region`:: (Optional) Region to use. If not provided, the default agent region is used.

`namespace`:: (Optional) Namespace to use. If not provided the `default` namespace is used.

`secret_id`:: (Optional) SecretID to use if ACL is enabled in Nomad. This is an
example ACL policy to apply to the token.

[source,hcl]
----
namespace "*" {
  policy = "read"
}
node {
  policy = "read"
}
agent {
  policy = "read"
}
----

`node`:: (Optional) Specify the node to scope {beatname_lc} to in case it
  cannot be accurately detected when `node` scope is used.

`scope`:: (Optional) Specify at what level autodiscover needs to be done at. `scope` can
  either take `node` or `cluster` as values. `node` scope allows discovery of resources in
  the specified node. `cluster` scope allows cluster wide discovery. Defaults to `node`.

`wait_time`:: (Optional) Limits how long a Watch will block. If not specified (or set to `0`) the
  default configuration from the agent will be used.

`allow_stale`:: (Optional) allows any Nomad server (non-leader) to service a read. This normally
  means that the local node where filebeat is allocated will service filebeat's requests.
  Defaults to `true`.

include::../../{beatname_lc}/docs/autodiscover-nomad-config.asciidoc[]

endif::autodiscoverNomad[]

ifdef::autodiscoverAWSEC2[]
[float]
===== Amazon EC2s

experimental[]

The Amazon EC2 autodiscover provider discovers https://aws.amazon.com/ec2/[EC2 instances].
This is useful for users to launch {beatname_uc} modules to monitor services running on AWS EC2 instances.

ifeval::["{beatname_lc}"=="metricbeat"]
For example, you can use this provider to gather MySQL metrics from MySQL
servers running on EC2 instances that have a specific tag, `service: mysql`.
endif::[]

This provider will load AWS credentials using the standard AWS environment variables and shared credentials files
see https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html[Best Practices for Managing AWS Access Keys]
for more information. If you do not wish to use these, you may explicitly set the `access_key_id` and
`secret_access_key` variables.

These are the available fields during within config templating.
The `aws.ec2.*` fields and `cloud.*` fields will be available on each emitted event.

* cloud.availability_zone
* cloud.instance.id
* cloud.machine.type
* cloud.provider
* cloud.region

* aws.ec2.architecture
* aws.ec2.image.id
* aws.ec2.kernel.id
* aws.ec2.monitoring.state
* aws.ec2.private.dns_name
* aws.ec2.private.ip
* aws.ec2.public.dns_name
* aws.ec2.public.ip
* aws.ec2.root_device_name
* aws.ec2.state.code
* aws.ec2.state.name
* aws.ec2.subnet.id
* aws.ec2.tags
* aws.ec2.vpc.id

include::../../{beatname_lc}/docs/autodiscover-aws-ec2-config.asciidoc[]

This autodiscover provider takes our standard <<aws-credentials-config,AWS credentials options>>.

endif::autodiscoverAWSEC2[]

ifdef::autodiscoverHints[]
[[configuration-autodiscover-hints]]
=== Hints based autodiscover

include::../../{beatname_lc}/docs/autodiscover-hints.asciidoc[]
endif::autodiscoverHints[]

[[configuration-autodiscover-advanced]]
=== Advanced usage

////
Builders are not exposed in docs, as we only have `hints` builder by the moment,
and that's covered in previous section

[float]
==== Builders
Builders allow users to pass hints to autodiscover for it to be able to make decisions
on how and what kind of configuration should be generated. Each Beat can define its own
Builders that it can use. Hints are generated based on all information that is passed to
the provider using a prefix "co.elastic.*". The Kubernetes provider uses annotations and
the Docker provider uses labels to achieve the same.

////

[float]
==== Appenders
Appenders allow users to append configuration that is already built with the help of either templates
or builders. Appenders can be configured to be applied only when a required condition is matched. The kind
of configuration that is applied is specific to each appender.

[float]
===== Config
The config appender can apply a config on top of the config that was generated by
templates or builders. The config is applied whenever a provided condition is matched. It is always
applied if there is no condition provided.

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
{beatname_lc}.autodiscover:
  providers:
    - type: kubernetes
      templates:
        ...
      appenders:
        - type: config
          condition.equals:
            kubernetes.namespace: "prometheus"
          config:
            fields:
              type: monitoring
-------------------------------------------------------------------------------------
