[[running-on-kubernetes]]
=== Run {beatname_uc} on Kubernetes

You can use {beatname_uc} <<running-on-docker,Docker images>> on Kubernetes to
retrieve and ship container logs.

TIP: Running {ecloud} on Kubernetes? See {eck-ref}/k8s-beat.html[Run {beats} on ECK].

ifeval::["{release-state}"=="unreleased"]

However, version {version} of {beatname_uc} has not yet been
released, so no Docker image is currently available for this version.

endif::[]


[float]
==== Kubernetes deploy manifests

You deploy {beatname_uc} as a https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/[DaemonSet]
to ensure there's a running instance on each node of the cluster.

The Docker logs host folder (`/var/lib/docker/containers`) is mounted on the
{beatname_uc} container. {beatname_uc} starts an input for the files and
begins harvesting them as soon as they appear in the folder.

Everything is deployed under the `kube-system` namespace by default. To change
the namespace, modify the manifest file.

To download the manifest file, run:

["source", "sh", subs="attributes"]
------------------------------------------------
curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/kubernetes/filebeat-kubernetes.yaml
------------------------------------------------

[WARNING]
=======================================
*If you are using Kubernetes 1.7 or earlier:* {beatname_uc} uses a hostPath volume to persist internal data. It's located
under +/var/lib/{beatname_lc}-data+. The manifest uses folder autocreation (`DirectoryOrCreate`), which was introduced in
Kubernetes 1.8. You need to remove `type: DirectoryOrCreate` from the manifest and create the host folder yourself.
=======================================

[float]
==== Settings

By default, {beatname_uc} sends events to an existing Elasticsearch deployment,
if present. To specify a different destination, change the following parameters
in the manifest file:

[source,yaml]
------------------------------------------------
- name: ELASTICSEARCH_HOST
  value: elasticsearch
- name: ELASTICSEARCH_PORT
  value: "9200"
- name: ELASTICSEARCH_USERNAME
  value: elastic
- name: ELASTICSEARCH_PASSWORD
  value: changeme
------------------------------------------------

[float]
===== Red Hat OpenShift configuration

If you are using Red Hat OpenShift, you need to specify additional settings in
the manifest file and enable the container to run as privileged.

. Modify the `DaemonSet` container spec in the manifest file:
+
[source,yaml]
-----
  securityContext:
    runAsUser: 0
    privileged: true
-----

. Grant the `filebeat` service account access to the privileged SCC:
+
[source,shell]
-----
oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:filebeat
-----
+
This command enables the container to be privileged as an administrator for
OpenShift.

. Override the default node selector for the `kube-system` namespace (or your
custom namespace) to allow for scheduling on any node:
+
[source,shell]
----
oc patch namespace kube-system -p \
'{"metadata": {"annotations": {"openshift.io/node-selector": ""}}}'
----
+
This command sets the node selector for the project to an empty string. If you
don't run this command, the default node selector will skip master nodes.

[float]
==== Load {kib} dashboards

{beatname_uc} comes packaged with various pre-built {kib} dashboards
that you can use to visualize logs from your Kubernetes environment.

If these dashboards are not already loaded into {kib}, you must <<{beatname_lc}-installation-configuration,install {beatname_uc}>>
on any system that can connect to the {stack}, and then run the `setup` command to load the dashboards.
To learn how, see <<load-kibana-dashboards,Load {kib} dashboards>>.

The `setup` command does not load the ingest pipelines used to parse log lines. By default, ingest pipelines
are set up automatically the first time you run {beatname_uc} and connect to {es}.

[IMPORTANT]
=======================================
If you are using a different output other than {es}, such as {ls}, you
need to:

* <<load-template-manually>>
* <<load-kibana-dashboards>>
* <<load-ingest-pipelines>>
=======================================

[float]
==== Deploy

To deploy {beatname_uc} to Kubernetes, run:

["source", "sh", subs="attributes"]
------------------------------------------------
kubectl create -f filebeat-kubernetes.yaml
------------------------------------------------

To check the status, run:

["source", "sh", subs="attributes"]
------------------------------------------------
$ kubectl --namespace=kube-system get ds/filebeat

NAME       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR   AGE
filebeat   32        32        0         32           0           <none>          1m
------------------------------------------------

Log events should start flowing to Elasticsearch. The events are annotated with
metadata added by the <<add-kubernetes-metadata>> processor.


[float]
==== Parsing json logs

It is common case when collecting logs from workloads running on Kubernetes that these
applications are logging in json format. In these case, special handling can be applied so as to
parse these json logs properly and decode them into fields. Bellow there are provided 2 different ways
of configuring <<configuration-autodiscover, filebeat's autodiscover>> so as to identify and parse json logs.
We will use an example of one Pod with 2 containers where only one of these logs in json format.

Example log:
```
{"type":"log","@timestamp":"2020-11-16T14:30:13+00:00","tags":["warning","plugins","licensing"],"pid":7,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
```


. Using `json.*` options with templates
+
[source,yaml]
------------------------------------------------
filebeat.autodiscover:
  providers:
      - type: kubernetes
        node: ${NODE_NAME}
        templates:
          - condition:
              contains:
                kubernetes.container.name: "no-json-logging"
            config:
              - type: container
                paths:
                  - "/var/log/containers/*-${data.kubernetes.container.id}.log"
          - condition:
              contains:
                kubernetes.container.name: "json-logging"
            config:
              - type: container
                paths:
                  - "/var/log/containers/*-${data.kubernetes.container.id}.log"
                json.keys_under_root: true
                json.add_error_key: true
                json.message_key: message
------------------------------------------------

. Using `json.*` options with hints
+
Key part here is to properly annotate the Pod to only parse logs of the correct container
as json logs. In this, annotation should be constructed like this:
+
`co.elastic.logs.<container_name>/json.keys_under_root: "true"`
+
Autodiscovery configuration:
+
[source,yaml]
------------------------------------------------
filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log
------------------------------------------------
+
Then annotate the pod properly:
+
[source,yaml]
------------------------------------------------
annotations:
    co.elastic.logs.json-logging/json.keys_under_root: "true"
    co.elastic.logs.json-logging/json.add_error_key: "true"
    co.elastic.logs.json-logging/json.message_key: "message"
------------------------------------------------

[float]
==== Logrotation

According to https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level[kubernetes documentation]
_Kubernetes is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that_.
Different logrotation strategies can cause issues that might make Filebeat losing events or even duplicating events.
Users can find more information about Filebeat's logrotation best practises at Filebeat's
<<file-log-rotation,log rotation specific documentation>>
