[role="xpack"]

:libbeat-xpack-dir: ../../../../x-pack/libbeat

:type: aws-cloudwatch

[id="{beatname_lc}-input-{type}"]
=== AWS CloudWatch input

++++
<titleabbrev>AWS CloudWatch</titleabbrev>
++++

`aws-cloudwatch` input can be used to retrieve all logs from all log streams in a
specific log group. `filterLogEvents` AWS API is used to list log events from
the specified log group. Amazon CloudWatch Logs can be used to store log files
from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sources.

A log group is a group of log streams that share the same retention, monitoring,
and access control settings. You can define log groups and specify which streams
to put into each group. There is no limit on the number of log streams that can
belong to one log group.

A log stream is a sequence of log events that share the same source. Each
separate source of logs in CloudWatch Logs makes up a separate log stream.

["source","yaml",subs="attributes"]
----
{beatname_lc}.inputs:
- type: aws-cloudwatch
  log_group_arn: arn:aws:logs:us-east-1:428152502467:log-group:test:*
  scan_frequency: 1m
  credential_profile_name: elastic-beats
  start_position: beginning
----

The `aws-cloudwatch` input supports the following configuration options plus the
<<{beatname_lc}-input-{type}-common-options>> described later.

[float]
==== `log_group_arn`
ARN of the log group to collect logs from.

[float]
==== `log_group_name`
Name of the log group to collect logs from. Note: `region_name` is required when
log_group_name is given.

[float]
==== `log_group_name_prefix`
The prefix for a group of log group names. Note: `region_name` is required when
log_group_name_prefix is given. `log_group_name` and `log_group_name_prefix`
cannot be given at the same time. The number of workers that will process the
log groups under this prefix is set through the `number_of_workers` config.

[float]
==== `region_name`
Region that the specified log group or log group prefix belongs to.

[float]
==== `number_of_workers`
Number of workers that will process the log groups with the given `log_group_name_prefix`.
Default value is 1.

[float]
==== `log_streams`
A list of strings of log streams names that Filebeat collect log events from.

[float]
==== `log_stream_prefix`
A string to filter the results to include only log events from log streams
that have names starting with this prefix.

[float]
==== `start_position`
`start_position` allows user to specify if this input should read log files from
the `beginning` or from the `end`.

* `beginning`: reads from the beginning of the log group (default).
* `end`: read only new messages from current time minus `scan_frequency` going forward

For example, with `scan_frequency` equals to `30s` and current timestamp is
`2020-06-24 12:00:00`:

* with `start_position = beginning`:
** first iteration: startTime=0, endTime=2020-06-24 12:00:00
** second iteration: startTime=2020-06-24 12:00:00, endTime=2020-06-24 12:00:30

* with `start_position = end`:
** first iteration: startTime=2020-06-24 11:59:30, endTime=2020-06-24 12:00:00
** second iteration: startTime=2020-06-24 12:00:00, endTime=2020-06-24 12:00:30

[float]
==== `scan_frequency`
This config parameter sets how often Filebeat checks for new log events from the
specified log group. Default `scan_frequency` is 1 minute, which means Filebeat
will sleep for 1 minute before querying for new logs again.

[float]
==== `api_timeout`
The maximum duration of AWS API can take. If it exceeds the timeout, AWS API
will be interrupted. The default AWS API timeout for a message is 120 seconds.
The minimum is 0 seconds.

[float]
==== `api_sleep`
This is used to sleep between AWS `FilterLogEvents` API calls inside the same
collection period. `FilterLogEvents` API has a quota of 5 transactions per
second (TPS)/account/Region. By default, `api_sleep` is 200 ms. This value should
only be adjusted when there are multiple Filebeats or multiple Filebeat inputs
collecting logs from the same region and AWS account.

[float]
==== `latency`
Some AWS services send logs to CloudWatch with a latency to process larger than
`aws-cloudwatch` input `scan_frequency`. This case, please specify a `latency`
parameter so collection start time and end time will be shifted by the given
latency amount.

[float]
==== `aws credentials`
In order to make AWS API calls, `aws-cloudwatch` input requires AWS credentials.
Please see <<aws-credentials-config,AWS credentials options>> for more details.

[float]
=== AWS Permissions
Specific AWS permissions are required for IAM user to access aws-cloudwatch:
----
cloudwatchlogs:DescribeLogGroups
logs:FilterLogEvents
----

[id="{beatname_lc}-input-{type}-common-options"]
include::../../../../filebeat/docs/inputs/input-common-options.asciidoc[]

:type!:
