{beatname_uc} supports autodiscover based on hints from the provider. The hints system looks for
hints in Kubernetes Pod annotations or Docker labels that have the prefix `co.elastic.logs`. As soon as
the container starts, {beatname_uc} will check if it contains any hints and launch the proper config for
it. Hints tell {beatname_uc} how to get logs for the given container. By default logs will be retrieved
from the container using the `container` input. You can use hints to modify this behavior. This is the full
list of supported hints:

[float]
===== `co.elastic.logs/enabled`

Filebeat gets logs from all containers by default, you can set this hint to `false` to ignore
the output of the container. Filebeat won't read or send logs from it. If default config is
disabled, you can use this annotation to enable log retrieval only for containers with this
set to `true`. If you are aiming to use this with Kubernetes, have in mind that annotation
values can only be of string type so you will need to explicitly define this as `"true"`
or `"false"` accordingly.

[float]
===== `co.elastic.logs/multiline.*`

Multiline settings. See <<multiline-examples>> for a full list of all supported options.

[float]
===== `co.elastic.logs/json.*`

JSON settings. See <<filebeat-input-log-config-json>> for a full list of all supported options.

[float]
===== `co.elastic.logs/include_lines`

A list of regular expressions to match the lines that you want {beatname_uc} to include.
See <<configuration-filebeat-options>> for more info.

[float]
===== `co.elastic.logs/exclude_lines`

A list of regular expressions to match the lines that you want {beatname_uc} to exclude.
See <<configuration-filebeat-options>> for more info.

[float]
===== `co.elastic.logs/module`

Instead of using raw `docker` input, specifies the module to use to parse logs from the container. See
<<filebeat-modules>> for the list of supported modules.

[float]
===== `co.elastic.logs/fileset`

When module is configured, map container logs to module filesets. You can either configure
a single fileset like this:

[source,yaml]
-----
co.elastic.logs/fileset: access
-----

Or configure a fileset per stream in the container (stdout and stderr):

[source,yaml]
-----
co.elastic.logs/fileset.stdout: access
co.elastic.logs/fileset.stderr: error
-----

[float]
===== `co.elastic.logs/raw`
When an entire input/module configuration needs to be completely set the `raw` hint can be used. You can provide a
stringified JSON of the input configuration. `raw` overrides every other hint and can be used to create both a single or
a list of configurations.

[source,yaml]
-----
co.elastic.logs/raw: "[{\"containers\":{\"ids\":[\"${data.container.id}\"]},\"multiline\":{\"negate\":\"true\",\"pattern\":\"^test\"},\"type\":\"docker\"}]"
-----

[float]
===== `co.elastic.logs/processors`

Define a processor to be added to the {beatname_uc} input/module configuration. See <<filtering-and-enhancing-data>> for the list
of supported processors.

If processors configuration uses list data structure, object fields must be enumerated.
For example, hints for the `rename` processor configuration below
[source,yaml]
-----
processors:
  - rename:
      fields:
        - from: "a.g"
          to: "e.d"
      fail_on_error: true
-----
will look like:
[source,yaml]
-----
co.elastic.logs/processors.rename.fields.0.from: "a.g"
co.elastic.logs/processors.rename.fields.1.to: "e.d"
co.elastic.logs/processors.rename.fail_on_error: 'true'
-----

If processors configuration uses map data structure, enumeration is not needed. For example, the equivalent to the `add_fields` configuration below
[source,yaml]
-----
processors:
  - add_fields:
      target: project
      fields:
        name: myproject
-----
is
[source,yaml]
-----
co.elastic.logs/processors.1.add_fields.target: "project"
co.elastic.logs/processors.1.add_fields.fields.name: "myproject"
-----

In order to provide ordering of the processor definition, numbers can be provided. If not, the hints builder will do
arbitrary ordering:

[source,yaml]
-----
co.elastic.logs/processors.1.dissect.tokenizer: "%{key1} %{key2}"
co.elastic.logs/processors.dissect.tokenizer: "%{key2} %{key1}"
-----

In the above sample the processor definition tagged with `1` would be executed first.

[float]
==== Kubernetes

Kubernetes autodiscover provider supports hints in Pod annotations. To enable it just set `hints.enabled`:

[source,yaml]
-----
filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
-----

You can configure the default config that will be launched when a new container is seen, like this:

[source,yaml]
-----
filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*-${data.container.id}.log  # CRI path
-----

You can also disable default settings entirely, so only Pods annotated like `co.elastic.logs/enabled: true`
will be retrieved:

[source,yaml]
-----
filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      hints.default_config.enabled: false
-----

You can annotate Kubernetes Pods with useful info to spin up {beatname_uc} inputs or modules:

[source,yaml]
-----
annotations:
  co.elastic.logs/multiline.pattern: '^\['
  co.elastic.logs/multiline.negate: true
  co.elastic.logs/multiline.match: after
-----


[float]
===== Multiple containers

When a pod has multiple containers, the settings are shared unless you put the container name in the
hint. For example, these hints configure multiline settings for all containers in the pod, but set a
specific `exclude_lines` hint for the container called `sidecar`.


[source,yaml]
-----
annotations:
  co.elastic.logs/multiline.pattern: '^\['
  co.elastic.logs/multiline.negate: true
  co.elastic.logs/multiline.match: after
  co.elastic.logs.sidecar/exclude_lines: '^DBG'
-----

[float]
===== Multiple sets of hints
When a container needs multiple inputs to be defined on it, sets of annotations can be provided with numeric prefixes.
If there are hints that don't have a numeric prefix then they get grouped together into a single configuration.

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
annotations:
  co.elastic.logs/exclude_lines: '^DBG'
  co.elastic.logs/1.include_lines: '^DBG'
  co.elastic.logs/1.processors.dissect.tokenizer: "%{key2} %{key1}"
-------------------------------------------------------------------------------------

The above configuration would generate two input configurations. The first input handles only debug logs and passes it through a dissect
tokenizer. The second input handles everything but debug logs.

[float]
=====  Namespace Defaults

Hints can be configured on the Namespace's annotations as defaults to use when Pod level annotations are missing.
The resultant hints are a combination of Pod annotations and Namespace annotations with the Pod's taking precedence. To
enable Namespace defaults configure the `add_resource_metadata` for Namespace objects as follows:

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      add_resource_metadata:
        namespace:
          include_annotations: ["nsannotation1"]
-------------------------------------------------------------------------------------



[float]
==== Docker

Docker autodiscover provider supports hints in labels. To enable it just set `hints.enabled`:

[source,yaml]
-----
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
-----

You can configure the default config that will be launched when a new container is seen, like this:

[source,yaml]
-----
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*-${data.container.id}.log  # CRI path
-----

You can also disable default settings entirely, so only containers labeled with `co.elastic.logs/enabled: true`
will be retrieved:

[source,yaml]
-----
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config.enabled: false
-----

You can label Docker containers with useful info to spin up {beatname_uc} inputs, for example:

[source,yaml]
-----
  co.elastic.logs/module: nginx
  co.elastic.logs/fileset.stdout: access
  co.elastic.logs/fileset.stderr: error
-----

The above labels configure {beatname_uc} to use the Nginx module to harvest logs for this container.
Access logs will be retrieved from stdout stream, and error logs from stderr.


You can label Docker containers with useful info to decode logs structured as JSON messages, for example:

[source,yaml]
-----
  co.elastic.logs/json.keys_under_root: true
  co.elastic.logs/json.add_error_key: true
  co.elastic.logs/json.message_key: log
-----



[float]
==== Nomad

Nomad autodiscover provider supports hints using the
https://www.nomadproject.io/docs/job-specification/meta.html[`meta` stanza]. To
enable it just set `hints.enabled`:

[source,yaml]
-----
filebeat.autodiscover:
  providers:
    - type: nomad
      hints.enabled: true
-----

You can configure the default config that will be launched when a new job is
seen, like this:

[source,yaml]
-----
filebeat.autodiscover:
  providers:
    - type: nomad
      hints.enabled: true
      hints.default_config:
        type: log
        paths:
          - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
-----

You can also disable the default config such that only logs from jobs explicitly
annotated with `"co.elastic.logs/enabled" = "true"` will be collected:

[source,yaml]
-----
filebeat.autodiscover:
  providers:
    - type: nomad
      hints.enabled: true
      hints.default_config:
        enabled: false
        type: log
        paths:
          - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
-----

You can annotate Nomad Jobs using the `meta` stanza with useful info to spin up
{beatname_uc} inputs or modules:

[source,hcl]
-----
meta {
  "co.elastic.logs/enabled"           = "true"
  "co.elastic.logs/multiline.pattern" = "^\\["
  "co.elastic.logs/multiline.negate"  = "true"
  "co.elastic.logs/multiline.match"   = "after"
}
-----

If you are using autodiscover then in most cases you will want to use the
<<add-nomad-metadata,`add_nomad_metadata`>> processor to enrich events with
Nomad metadata. This example configures {{beatname_uc}} to connect to the local
Nomad agent over HTTPS and adds the Nomad allocation ID to all events from the
input. Later in the pipeline the `add_nomad_metadata` processor will use that ID
to enrich the event.

[source,yaml]
-----
filebeat.autodiscover:
  providers:
    - type: nomad
      address: https://localhost:4646
      hints.enabled: true
      hints.default_config:
        enabled: false <1>
        type: log
        paths:
          - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
        processors:
          - add_fields: <2>
              target: nomad
              fields:
                allocation.id: ${data.nomad.allocation.id}

processors:
  - add_nomad_metadata: <3>
      when.has_fields.fields: [nomad.allocation.id]
      address: https://localhost:4646
      default_indexers.enabled: false
      default_matchers.enabled: false
      indexers:
        - allocation_uuid:
      matchers:
        - fields:
            lookup_fields:
              - 'nomad.allocation.id'
-----
<1> The default config is disabled meaning any task without the
`"co.elastic.logs/enabled" = "true"` metadata will be ignored.
<2> The `add_fields` processor populates the `nomad.allocation.id` field with
the Nomad allocation UUID.
<3> The `add_nomad_metadata` processor is configured at the global level so
that it is only instantiated one time which saves resources.
