[id="{beatname_lc}-overview"]
[role="xpack"]
== {beatname_uc} overview

{beatname_uc} is an Elastic https://www.elastic.co/beats[Beat] that you
deploy as a function in your serverless environment to collect data from cloud
services and ship it to the {stack}.

Version {version} supports deploying {beatname_uc} as an AWS Lambda service. It
responds to triggers defined for the following event sources:

* https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html[Amazon CloudWatch Logs]
* https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html[Amazon Simple Queue Service (SQS)]
* https://docs.aws.amazon.com/kinesis/latest/APIReference/Welcome.html[Amazon Kinesis]

image::./images/diagram-functionbeat-architecture.svg["{beatname_uc} collects events generated by cloud services"]

include::{libbeat-dir}/shared-libbeat-description.asciidoc[]

The following sections explore some common use cases for {beatname_uc}:

* <<monitor-cloud-deployments>>
* <<event-drive-processing>>

*Want to ship logs from Google Cloud?* Use our Google Cloud Dataflow templates
to ship Google Pub/Sub and Google Cloud Storage logs directly from the Google
Cloud Console. To learn more, refer to
{observability-guide}/gcp-dataflow.html[GCP Dataflow templates].

[float]
[[monitor-cloud-deployments]]
=== Monitor cloud deployments

You can deploy {beatname_uc} on your serverless environment to collect logs and
metrics generated by cloud services and stream the data to the {stack} for
centralized analytics.

[float]
==== Monitor AWS services with CloudWatch logs

You can deploy {beatname_uc} as a Lambda function on AWS to receive events from
a Cloudwatch Log group, extract and structure the relevant fields, then stream
the events to {es}.

The processing pipeline for this use case typically looks like this:

. {beatname_uc} runs as a Lambda function on AWS and reads the data stream from
a Cloudwatch Log group.

. {beats} processors, such as <<dissect,dissect>> and <<drop-fields,drop_fields>>,
filter and structure the events.

. Optional {ref}/ingest.html[ingest pipelines] in {es} further enhance the
data.

. The structured events are indexed into an {es} cluster.

image::./images/functionbeat-pipeline-cloudwatchlogs.png["{beatname_uc} collects events generated by CloudWatch logs"]

/////
The following is commented out until this architecture is fully enabled:
If you have multiple instances of CloudWatch logs and want to centralize your
logging environment in Amazon Kinesis, you can send the logs to Kinesis, then
use {beatname_uc} to receive events from Kinesis, process them, and then stream
them to {es} for further analysis.
/////

[float]
[[event-drive-processing]]
=== Perform event-driven processing

You can use {beatname_uc} to implement event-driven processing workflows with
cloud messaging queues and the {stack}. {beatname_uc} responds to event triggers
from AWS Kinesis and SQS.

[float]
==== Analyze application data from SQS

For applications that send JSON-encoded events to an SQS queue, {beatname_uc}
can listen for, ingest, and decode JSON events prior to
shipping them to {es}, where you can analyze the streaming data.

The processing pipeline for this use case typically looks like this:

. {beatname_uc} runs as a serverless shipper and listens to an SQS queue for
application events.

. The {beats} <<decode-json-fields,decode_json_fields>> processor
decodes JSON strings and replaces them with valid JSON objects.

. Optional {ref}/ingest.html[ingest pipelines] in {es} further enhance the
data.

. The events are indexed into an {es} cluster.

image::./images/functionbeat-pipeline-sqs.png["{beatname_uc} application events triggered by SQS"]
