////
This file is generated! See scripts/mage/docs_collector.go
////

:modulename: aws

[[metricbeat-module-aws]]
[role="xpack"]
== AWS module

:libbeat-xpack-dir: ../../../x-pack/libbeat

include::{libbeat-dir}/shared/integration-link.asciidoc[]

:modulename!:

This module periodically fetches monitoring metrics from AWS CloudWatch using
https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for AWS services.

All metrics are enabled by default.

IMPORTANT: Extra AWS charges on CloudWatch API requests will be generated by this
module. Please see <<aws-api-requests,AWS API requests>> for more details.

[float]
== Module-specific configuration notes

* *AWS Credentials*

The `aws` module requires AWS credentials configuration in order to make AWS API calls.
Users can either use `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and/or
`AWS_SESSION_TOKEN`, or use shared AWS credentials file.
Please see <<aws-credentials-config,AWS credentials options>> for more details.

* *regions*

This module also accepts optional configuration `regions` to specify which
AWS regions to query metrics from. If the `regions` parameter is not set in the
config file, then by default, the `aws` module will query metrics from all available
AWS regions. If `endpoint` is specified, `regions` becomes a required config parameter.

* *latency*

Some AWS services send monitoring metrics to CloudWatch with a latency to
process larger than Metricbeat collection period. This will cause data points missing
or none get collected by Metricbeat. In this case, please specify a `latency`
parameter so collection start time and end time will be shifted by the given
latency amount.

* *endpoint*

Most AWS services offer a regional endpoint that can be used to make requests.
The general syntax of a regional endpoint is `protocol://service-code.region-code.endpoint-code`.
Some services, such as IAM, do not support regions. The endpoints for these
services do not include a region. In `aws` module, `endpoint` config is to set
the `endpoint-code` part, such as `amazonaws.com`, `amazonaws.com.cn`, `c2s.ic.gov`,
`sc2s.sgov.gov`.

If endpoint is specified, `regions` config becomes required. For example:

* *tags_filter*

The tags to filter against. If tags are given in config, then only
collect metrics from resources that have tag key and tag value matches the filter.
For example, if tags parameter is given as `Organization=Engineering` under
`AWS/ELB` namespace, then only collect metrics from ELBs with tag name equals to
`Organization` and tag value equals to `Engineering`.

Note: tag filtering only works for metricsets with `resource_type` specified in the 
metricset-specific configuration.

[source,yaml]
----
- module: aws
  period: 5m
  endpoint: amazonaws.com.cn
  regions: cn-north-1
  metricsets:
    - ec2
  tags_filter:
    - key: "Organization"
      value: "Engineering"
----

The aws module comes with a predefined dashboard. For example:

image::./images/metricbeat-aws-overview.png[]

[float]
== Metricsets

Currently, we have `billing`, `cloudwatch`, `dynamodb`, `ebs`, `ec2`, `elb`, `kinesis`
`lambda`, `mtest`, `natgateway`, `rds`, `s3_daily_storage`, `s3_request`, `sns`, `sqs`,
`transitgateway`, `usage` and `vpn` metricset in `aws` module.

[float]
=== `billing`
Billing metric data includes the estimated charges for every service in the AWS
account and the estimated overall total charge for the AWS account. The estimated
charges are calculated and sent several times daily to CloudWatch. Therefore,
`period` in aws module configuration is set to `12h`.

The billing metricset comes with a predefined dashboard:

image::./images/metricbeat-aws-billing-overview.png[]

[float]
=== `cloudwatch`
This metricset allows users to query metrics from AWS CloudWatch with any given
namespaces or specific instance with a given period.
Please see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html[AWS Services That Publish CloudWatch Metrics]
for a list of AWS services that publish metrics to CloudWatch.

[float]
=== `dynamodb`
DynamoDB sends metrics to CloudWatch periodically for better monitoring how web
application or service is performing.

The dynamodb metricset comes with a predefined dashboard:

image::./images/metricbeat-aws-dynamodb-overview.png[]

[float]
=== `ebs`
For basic monitoring in AWS EBS volumes, data is available automatically in
5-minute periods at no charge. This includes data for the root device volumes
for EBS-backed instances. User can also enable detailed monitoring for
provisioned IOPS SSD (io1) volumes to automatically send one-minute metrics to
CloudWatch. Default period in aws module configuration is set to `5m` for ebs
metricset.

The ebs metricset comes with a predefined dashboard:

image::./images/metricbeat-aws-ebs-overview.png[]

[float]
=== `ec2`
By default, Amazon EC2 sends metric data to CloudWatch every 5 minutes. With this basic monitoring, `period` in aws module
configuration should be larger or equal than `300s`. If `period` is set to be less than `300s`, the same cloudwatch metrics
will be collected more than once which will cause extra fees without getting more granular metrics. For example, in `US East (N. Virginia)` region, it costs
$0.01/1000 metrics requested using GetMetricData. Please see https://aws.amazon.com/cloudwatch/pricing/[AWS CloudWatch Pricing]
for more details. To avoid unnecessary charges, `period` is preferred to be set to `300s` or multiples of `300s`, such as
`600s` and `900s`. For more granular monitoring data you can enable detailed monitoring on the instance to get metrics every 1 minute. Please see
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html[Enabling Detailed Monitoring] for instructions
on how to enable detailed monitoring. With detailed monitoring enabled, `period` in aws module configuration can be any number
larger than `60s`. Since AWS sends metric data to CloudWatch in 1-minute periods, setting metricbeat module `period` less
than `60s` will cause extra API requests which means extra charges on AWS. To avoid unnecessary charges, `period` is
preferred to be set to `60s` or multiples of `60s`, such as `120s` and `180s`.

The ec2 metricset comes with a predefined dashboard:

image::./images/metricbeat-aws-ec2-overview.png[]

[float]
=== `elb`
elb metricset collects CloudWatch metrics from classic load balancer, application
load balancer and network load balancer.

All three kinds of elastic load balancing reports metrics to Cloudwatch only when requests are flowing
through the load balancer. If there are requests flowing through the load balancer,
Elastic Load Balancing measures and sends its metrics in 60-second intervals.
If there are no requests flowing through the load balancer or no data for a metric,
the metric is not reported.
Therefore, `period` in aws module configuration is set to `1m`.

The elb metricset comes with a predefined dashboard:

image::./images/metricbeat-aws-elb-overview.png[]

[float]
=== `lambda`
When an invocation completes, Lambda sends a set of metrics to CloudWatch for that invocation.
Default period in aws module configuration is set to `5m` for lambda metricset.
The lambda metricset comes with a predefined dashboard:

image::./images/metricbeat-aws-lambda-overview.png[]

[float]
=== `natgateway`
CloudWatch collects information from NAT gateways and creates readable, near real-time metrics.
This metricset enables users to collect these metrics from CloudWatch to monitor and troubleshoot
their NAT gateway. NAT gateway metric data is provided at 1-minute intervals and therefore,
`period` for `natgateway` metricset is recommended to be `1m` or multiples of `1m`.

[float]
=== `rds`
`period` for `rds` metricset is recommended to be `60s` or multiples of `60s` because Amazon RDS sends metrics and
dimensions to Amazon CloudWatch every minute.

The rds metricset comes with a predefined dashboard:

image::./images/metricbeat-aws-rds-overview.png[]

[float]
=== `s3_daily_storage`
Daily storage metrics for S3 buckets are reported once per day with no additional cost. Since they are daily metrics,
`period` for `s3_daily_storage` metricset is recommended to be `86400s` or multiples of `86400s`.

[float]
=== `s3_request`
Request metrics are available
at 1-minute intervals with additional charges. The s3_request metricset will give more
granular data to track S3 bucket usage. The `period` for `s3_request` metricset can be set to `60s` or multiples of `60s`.
But because of the extra charges for querying these metrics, the `period` is recommended to set to `86400s`. The user can
always adjust this to the granularity they want. Request metrics are not enabled by default for S3 buckets. Please see
https://docs.aws.amazon.com/AmazonS3/latest/user-guide/configure-metrics.html[How to
Configure Request Metrics for S3] for instructions on how to enable request metrics for
each S3 bucket.

The s3_daily_storage and s3_request metricset comes with a predefined combined dashboard:

image::./images/metricbeat-aws-s3-overview.png[]

[float]
=== `sqs`
CloudWatch metrics for Amazon SQS queues are automatically collected and pushed to CloudWatch every 5 minutes,
the `period` for `sqs` metricset is recommended to be `300s` or multiples of `300s`.

The sqs metricset comes with a predefined dashboard:

image::./images/metricbeat-aws-sqs-overview.png[]

[float]
=== `transitgateway`
Amazon VPC reports metrics to CloudWatch only when requests are flowing through
the transit gateway. If there are requests flowing through the transit gateway,
Amazon VPC measures and sends its metrics in 60-second intervals. `period` for
`transitgateway` metricset is recommended to be `1m` or multiples of `1m`.

[float]
=== `usage`
CloudWatch collects metrics that track the usage of some AWS resources. These
metrics correspond to AWS service quotas. Tracking these metrics can help
proactively manage quotas. Service quota usage metrics are in the AWS/Usage
namespace and are collected every minute. Therefore, `period` in aws module
configuration for usage metricset is set to `1m`.

The usage metricset comes with a predefined dashboard:

image::./images/metricbeat-aws-usage-overview.png[]

[float]
=== `vpn`
CloudWatch collects and processes raw data from the VPN service into readable, near
real-time metrics for users to better understand the performance of their web
applications and services.

[float]
[[aws-api-requests]]
== AWS API requests count
This session is to document what are the AWS API called made by each metricset
in `aws` module. This will be useful for users to estimate costs for using `aws`
module.

Note: some AWS APIs need pagination like ListMetrics and GetMetricData. Count
value is depends on the number of results.

ListMetrics max page size: 500, based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_ListMetrics.html[AWS API ListMetrics]

GetMetricData max page size: 100, based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[AWS API GetMetricData]

[float]
|===
| AWS API Name | AWS API Count | Frequency
| IAM ListAccountAliases | 1 | Once on startup
| STS GetCallerIdentity | 1 | Once on startup
| EC2 DescribeRegions| 1 | Once on startup
| CloudWatch ListMetrics | Total number of results / ListMetrics max page size | Per region per namespace per collection period
| CloudWatch GetMetricData | Total number of results / GetMetricData max page size | Per region per namespace per collection period
|===
`billing`, `ebs`, `elb`, `sns`, `usage` and `lambda` are the same as `cloudwatch` metricset.

[id="aws-credentials-config"]
include::{libbeat-xpack-dir}/docs/aws-credentials-config.asciidoc[]

[float]
== Running on EKS

* *WebIdentity authentication flow*

See documentation in order to create a IAM Role for Service account:
https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html

Once you have create the IRSA you can annotate `metricbeat` service account with it
[source,yaml]
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<IRSA_ID>
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat

In order to enable WebIdentity authentication flow you need to add a trust relationship
to the IRSA:
[source,json]
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<OIDC_PROVIDER_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.REGION.amazonaws.com/id/<OIDC_PROVIDER_ID>:sub": "system:serviceaccount:kube-system:metricbeat",
          "oidc.eks.REGION.amazonaws.com/id/<OIDC_PROVIDER_ID>:aud": "sts.amazonaws.com"
        }
      }
    }

In this case there's no need to add `role_arn` to modules config.


[float]
=== Example configuration

The AWS module supports the standard configuration options that are described
in <<configuration-metricbeat>>. Here is an example configuration:

[source,yaml]
----
metricbeat.modules:
- module: aws
  period: 300s
  credential_profile_name: test-mb
  metricsets:
    - ec2
  tags_filter:
    - key: "Organization"
      value: "Engineering"
- module: aws
  period: 300s
  credential_profile_name: test-mb
  metricsets:
    - sqs
  regions:
    - us-west-1
- module: aws
  period: 86400s
  metricsets:
    - s3_request
    - s3_daily_storage
  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
  session_token: '${AWS_SESSION_TOKEN:""}'
- module: aws
  period: 300s
  credential_profile_name: test-mb
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EC2
      name: ["CPUUtilization"]
      dimensions:
        - name: InstanceId
          value: i-0686946e22cf9494a
    - namespace: AWS/EBS
    - namespace: AWS/ELB
      resource_type: elasticloadbalancing
      tags:
        - key: "Organization"
          value: "Engineering"
- module: aws
  period: 60s
  credential_profile_name: test-mb
  tags_filter:
    - key: "dept"
      value: "eng"
  metricsets:
    - elb
    - natgateway
    - rds
    - transitgateway
    - usage
    - vpn
----

[float]
=== Metricsets

The following metricsets are available:

* <<metricbeat-metricset-aws-billing,billing>>

* <<metricbeat-metricset-aws-cloudwatch,cloudwatch>>

* <<metricbeat-metricset-aws-dynamodb,dynamodb>>

* <<metricbeat-metricset-aws-ebs,ebs>>

* <<metricbeat-metricset-aws-ec2,ec2>>

* <<metricbeat-metricset-aws-elb,elb>>

* <<metricbeat-metricset-aws-kinesis,kinesis>>

* <<metricbeat-metricset-aws-lambda,lambda>>

* <<metricbeat-metricset-aws-natgateway,natgateway>>

* <<metricbeat-metricset-aws-rds,rds>>

* <<metricbeat-metricset-aws-s3_daily_storage,s3_daily_storage>>

* <<metricbeat-metricset-aws-s3_request,s3_request>>

* <<metricbeat-metricset-aws-sns,sns>>

* <<metricbeat-metricset-aws-sqs,sqs>>

* <<metricbeat-metricset-aws-transitgateway,transitgateway>>

* <<metricbeat-metricset-aws-usage,usage>>

* <<metricbeat-metricset-aws-vpn,vpn>>

include::aws/billing.asciidoc[]

include::aws/cloudwatch.asciidoc[]

include::aws/dynamodb.asciidoc[]

include::aws/ebs.asciidoc[]

include::aws/ec2.asciidoc[]

include::aws/elb.asciidoc[]

include::aws/kinesis.asciidoc[]

include::aws/lambda.asciidoc[]

include::aws/natgateway.asciidoc[]

include::aws/rds.asciidoc[]

include::aws/s3_daily_storage.asciidoc[]

include::aws/s3_request.asciidoc[]

include::aws/sns.asciidoc[]

include::aws/sqs.asciidoc[]

include::aws/transitgateway.asciidoc[]

include::aws/usage.asciidoc[]

include::aws/vpn.asciidoc[]

