////
This file is generated! See scripts/docs_collector.py
////

[[filebeat-module-aws]]
[role="xpack"]

:libbeat-xpack-dir: ../../../x-pack/libbeat

:modulename: aws
:has-dashboards: true

== AWS module

include::{libbeat-dir}/shared/integration-link.asciidoc[]

This is a module for aws logs. It uses filebeat s3 input to get log files from
AWS S3 buckets with SQS notification or directly polling list of S3 objects in an S3 bucket.
The use of SQS notification is preferred: polling list of S3 objects is expensive
in terms of performance and costs, and cannot scale horizontally without ingestion duplication,
and should be preferably used only when no SQS notification can be attached to the S3 buckets.

This module supports reading S3 server access logs with `s3access` fileset,
ELB access logs with `elb` fileset, VPC flow logs with `vpcflow` fileset,
and CloudTrail logs with `cloudtrail` fileset.

Access logs contain detailed information about the requests made to these
services. VPC flow logs captures information about the IP traffic going to and
from network interfaces in AWS VPC. ELB access logs captures detailed information
about requests sent to the load balancer. CloudTrail logs contain events
that represent actions taken by a user, role or AWS service.

The `aws` module requires AWS credentials configuration in order to make AWS API calls.
Users can either use `access_key_id`, `secret_access_key` and/or
`session_token`, or use `role_arn` AWS IAM role, or use shared AWS credentials file.

Please see <<aws-credentials-options,AWS credentials options>> for more details.

include::../include/gs-link.asciidoc[]

[float]
=== Module configuration

Example config:

[source,yaml]
----
- module: aws
  cloudtrail:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.bucket_arn: 'arn:aws:s3:::mybucket'
    #var.bucket_list_prefix: 'prefix'
    #var.bucket_list_interval: 300s
    #var.number_of_workers: 5
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb
    #var.proxy_url: http://proxy:8080

  cloudwatch:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.bucket_arn: 'arn:aws:s3:::mybucket'
    #var.bucket_list_prefix: 'prefix'
    #var.bucket_list_interval: 300s
    #var.number_of_workers: 5
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb
    #var.proxy_url: http://proxy:8080

  ec2:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.bucket_arn: 'arn:aws:s3:::mybucket'
    #var.bucket_list_prefix: 'prefix'
    #var.bucket_list_interval: 300s
    #var.number_of_workers: 5
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb
    #var.proxy_url: http://proxy:8080

  elb:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.bucket_arn: 'arn:aws:s3:::mybucket'
    #var.bucket_list_prefix: 'prefix'
    #var.bucket_list_interval: 300s
    #var.number_of_workers: 5
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb
    #var.proxy_url: http://proxy:8080

  s3access:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.bucket_arn: 'arn:aws:s3:::mybucket'
    #var.bucket_list_prefix: 'prefix'
    #var.bucket_list_interval: 300s
    #var.number_of_workers: 5
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb
    #var.proxy_url: http://proxy:8080

  vpcflow:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.bucket_arn: 'arn:aws:s3:::mybucket'
    #var.bucket_list_prefix: 'prefix'
    #var.bucket_list_interval: 300s
    #var.number_of_workers: 5
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb
    #var.proxy_url: http://proxy:8080
----

*`var.queue_url`*::

AWS SQS queue url (Required when `var.bucket_arn` is not set).

*`var.visibility_timeout`*::

The duration that the received messages are hidden from ReceiveMessage request.
Default to be 300 seconds.

*`var.api_timeout`*::

The maximum duration of the AWS API call. If it exceeds the timeout, the AWS API
call will be interrupted. The default AWS API timeout is `120s`.

The API timeout must be longer than the `sqs.wait_time` value.

*`var.bucket_arn`*::

AWS S3 bucket ARN (Required when `var.queue_url` is not set).

*`var.number_of_workers`*::

Number of workers that will process the S3 objects listed (Required when `var.bucket_arn` is set).
Use to vertically scale the input.

*`var.bucket_list_interval`*::

Wait interval between completion of a list request to the S3 bucket and beginning of the next one. Default to be 120 seconds.

*`var.bucket_list_prefix`*::

Prefix to apply for the list request to the S3 bucket. Default empty.

*`var.endpoint`*::

Custom endpoint used to access AWS APIs.

*`var.shared_credential_file`*::

Filename of AWS credential file.

*`var.credential_profile_name`*::

AWS credential profile name.

*`var.access_key_id`*::
First part of access key.

*`var.secret_access_key`*::
Second part of access key.

*`var.session_token`*::
Required when using temporary security credentials.

*`var.role_arn`*::
AWS IAM Role to assume.

[float]
=== config behaviour
Beware that in case both `var.queue_url` and `var.bucket_arn` are not set
instead of failing to start Filebeat with a config validation error, only the
specific fileset input will be stopped and a warning printed:
```
2021-08-26T14:33:03.661-0600 WARN [aws-s3] awss3/config.go:54 neither queue_url nor bucket_arn were provided, input aws-s3 will stop
2021-08-26T14:33:10.668-0600 INFO [input.aws-s3] compat/compat.go:111 Input aws-s3 starting {"id": "29F3565F5B2A7070"}
2021-08-26T14:33:10.668-0600 INFO [input.aws-s3] compat/compat.go:124 Input 'aws-s3' stopped {"id": "29F3565F5B2A7070"}
```

This behaviour is required in order to reduce destruction of existing Filebeat setup
where not all AWS module's filesets are defined and will change in next major release.

Setting `enabled: false` in the unused fileset will silence the warning and it is
the suggested setup. For example (assuming `cloudtrail` as unused fileset):
```
- module: aws
  cloudtrail:
    enabled: false

```

[float]
=== cloudtrail fileset

CloudTrail monitors events for the account. If user creates a trail, it
delivers those events as log files to a specific Amazon S3 bucket.
The `cloudtrail` fileset does not read the CloudTrail Digest files
that are delivered to the S3 bucket when Log File Integrity is turned
on, it only reads the CloudTrail logs.

[role="screenshot"]
image::./images/filebeat-aws-cloudtrail.png[]

[float]
=== cloudwatch fileset

Users can use Amazon CloudWatch Logs to monitor, store, and access log files
from different sources. Export logs from log groups to an Amazon S3 bucket which
has SQS notification setup already. This fileset will parse these logs into
`timestamp` and `message` field.

[float]
=== ec2 fileset

This fileset is specifically for EC2 logs stored in AWS CloudWatch. Export logs
from log groups to Amazon S3 bucket which has SQS notification setup already.
With this fileset, EC2 logs will be parsed into fields like  `ip`
and `program_name`. For logs from other services, please use `cloudwatch` fileset.

[float]
=== elb fileset

Elastic Load Balancing provides access logs that capture detailed information
about requests sent to the load balancer. Each log contains information such
as the time the request was received, the client's IP address, latencies,
request paths, and server responses. Users can use these access logs to analyze
traffic patterns and to troubleshoot issues.

Please follow https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-access-logs.html[enable access logs for classic load balancer]
for sending Classic ELB access logs to S3 bucket.
For application load balancer, please follow https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#enable-access-logging[enable access log for application load balancer].
For network load balancer, please follow https://docs.aws.amazon.com/elasticloadbalancing/latest//network/load-balancer-access-logs.html[enable access log for network load balancer].

This fileset comes with a predefined dashboard:

[role="screenshot"]
image::./images/filebeat-aws-elb-overview.png[]

[float]
=== s3access fileset

Server access logging provides detailed records for the requests that are made
to a bucket. Server access logs are useful for many applications. For example,
access log information can be useful in security and access audits. It can also
help you learn about customer base and understand Amazon S3 bill.

Please follow https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html#server-access-logging-overview[how to enable server access logging]
for sending server access logs to S3 bucket.

This fileset comes with a predefined dashboard:

[role="screenshot"]
image::./images/filebeat-aws-s3access-overview.png[]

[float]
=== vpcflow fileset

VPC Flow Logs is a feature in AWS that enables users to capture information
about the IP traffic going to and from network interfaces in VPC. Flow log data
needs to be published to Amazon S3 in order for `vpcflow` fileset to retrieve.
Flow logs can help users to monitor traffic that is reaching each instance and
determine the direction of the traffic to and from the network interfaces.

This fileset comes with a predefined dashboard:

[role="screenshot"]
image::./images/filebeat-aws-vpcflow-overview.png[]

[id="aws-credentials-options"]
include::{libbeat-xpack-dir}/docs/aws-credentials-config.asciidoc[]


[float]
=== Fields

For a description of each field in the module, see the
<<exported-fields-aws,exported fields>> section.

