[[running-on-kubernetes]]
=== Run Metricbeat on Kubernetes

You can use {beatname_uc} <<running-on-docker,Docker images>> on Kubernetes to
retrieve cluster metrics.

TIP: Running {ecloud} on Kubernetes? See {eck-ref}/k8s-beat.html[Run {beats} on ECK].

ifeval::["{release-state}"=="unreleased"]

However, version {version} of {beatname_uc} has not yet been
released, so no Docker image is currently available for this version.

endif::[]


[float]
==== Kubernetes deploy manifests

You deploy {beatname_uc} as a https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/[DaemonSet]
to ensure that there's a running instance on each node of the cluster. These
instances are used to retrieve most metrics from the host, such as system
metrics, Docker stats, and metrics from all the services running on top of
Kubernetes.

In addition, one of the Pods in the DaemonSet will constantly hold a _leader lock_ which makes it responsible for
handling cluster-wide monitoring.
This instance is used to retrieve metrics that are unique for the whole
cluster, such as Kubernetes events or
https://github.com/kubernetes/kube-state-metrics[kube-state-metrics].
You can find more information about leader election configuration options at <<configuration-autodiscover>>.

Note: If you are upgrading from older versions, please make sure there are no redundant parts
as left-overs from the old manifests. Deployment specification and its ConfigMaps might be the case.

Everything is deployed under the `kube-system` namespace by default. To change
the namespace, modify the manifest file.

To download the manifest file, run:

["source", "sh", subs="attributes"]
------------------------------------------------
curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/kubernetes/metricbeat-kubernetes.yaml
------------------------------------------------

[WARNING]
=======================================
*If you are using Kubernetes 1.7 or earlier:* {beatname_uc} uses a hostPath volume to persist internal data. It's located
under +/var/lib/{beatname_lc}-data+. The manifest uses folder autocreation (`DirectoryOrCreate`), which was introduced in
Kubernetes 1.8. You need to remove `type: DirectoryOrCreate` from the manifest and create the host folder yourself.
=======================================

[float]
==== Settings

By default, {beatname_uc} sends events to an existing Elasticsearch deployment,
if present. To specify a different destination, change the following parameters
in the manifest file:

[source,yaml]
------------------------------------------------
- name: ELASTICSEARCH_HOST
  value: elasticsearch
- name: ELASTICSEARCH_PORT
  value: "9200"
- name: ELASTICSEARCH_USERNAME
  value: elastic
- name: ELASTICSEARCH_PASSWORD
  value: changeme
------------------------------------------------

[float]
===== Red Hat OpenShift configuration

If you are using Red Hat OpenShift, you need to specify additional settings in
the manifest file and enable the container to run as privileged.

. Modify the `DaemonSet` container spec in the manifest file:
+
[source,yaml]
-----
  securityContext:
    runAsUser: 0
    privileged: true
-----

. In the manifest file, edit the `metricbeat-daemonset-modules` ConfigMap, and
specify the following settings under `kubernetes.yml` in the `data` section:
+
[source,yaml]
-----
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: ["https://${NODE_NAME}:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.certificate_authorities:
        - /path/to/kubelet-service-ca.crt
-----
NOTE: `kubelet-service-ca.crt` can be any CA bundle that contains the issuer of the certificate used in the Kubelet API.
According to each specific installation of Openshift this can be found either in `secrets` or in `configmaps`.
In some installations it can be available as part of the service account secret, in
`/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt`.
In case of using https://github.com/openshift/installer/blob/master/docs/user/gcp/install.md[Openshift installer]
for GCP then the following `configmap` can be mounted in Metricbeat Pod and use `ca-bundle.crt`
in `ssl.certificate_authorities`:
+
[source,shell]
-----
Name:         kubelet-serving-ca
Namespace:    openshift-kube-apiserver
Labels:       <none>
Annotations:  <none>

Data
====
ca-bundle.crt:
-----

. Under the `metricbeat` ClusterRole, add the following resources:
+
[source,yaml]
-----
  - nodes/metrics
  - nodes/stats
-----

. Grant the `metricbeat` service account access to the privileged SCC:
+
[source,shell]
-----
oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:metricbeat
-----
+
This command enables the container to be privileged as an administrator for
OpenShift.

. Override the default node selector for the `kube-system` namespace (or your
custom namespace) to allow for scheduling on any node:
+
[source,shell]
----
oc patch namespace kube-system -p \
'{"metadata": {"annotations": {"openshift.io/node-selector": ""}}}'
----
+
This command sets the node selector for the project to an empty string. If you
don't run this command, the default node selector will skip master nodes.

[float]
==== Load {kib} dashboards

{beatname_uc} comes packaged with various pre-built {kib} dashboards
that you can use to visualize metrics about your Kubernetes environment.

If these dashboards are not already loaded into {kib}, you must <<{beatname_lc}-installation-configuration,install {beatname_uc}>>
on any system that can connect to the {stack}, and then run the `setup` command to load the dashboards. To learn how,
see <<load-kibana-dashboards,Load {kib} dashboards>>.

[IMPORTANT]
=======================================
If you are using a different output other than {es}, such as {ls}, you
need to <<load-template-manually>> and <<load-kibana-dashboards>>.
=======================================

[float]
==== Deploy

Metricbeat gets some metrics from https://github.com/kubernetes/kube-state-metrics#usage[kube-state-metrics].
If `kube-state-metrics` is not already running, deploy it now (see the
https://github.com/kubernetes/kube-state-metrics#kubernetes-deployment[Kubernetes
deployment] docs).

To deploy {beatname_uc} to Kubernetes, run:

["source", "sh", subs="attributes"]
------------------------------------------------
kubectl create -f metricbeat-kubernetes.yaml
------------------------------------------------

To check the status, run:

["source", "sh", subs="attributes"]
------------------------------------------------
$ kubectl --namespace=kube-system  get ds/metricbeat

NAME       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR   AGE
metricbeat   32        32        0         32           0           <none>          1m
------------------------------------------------

Metrics should start flowing to Elasticsearch.


[float]
==== Deploying Metricbeat to collect cluster-level metrics in large clusters

The size and the number of nodes in a Kubernetes cluster can be fairly large at times, and in such cases
the Pod that will be collecting cluster level metrics might face performance issues due to
resources limitations. In this case users might consider to avoid using the leader election strategy
and instead run a dedicated, standalone {beatname_uc} instance using a Deployment in addition to the DaemonSet.
