[role="xpack"]

:type: httpjson

[id="{beatname_lc}-input-{type}"]
=== HTTP JSON input

++++
<titleabbrev>HTTP JSON</titleabbrev>
++++

Use the `httpjson` input to read messages from an HTTP API with JSON payloads.

This input supports:

* Auth
** Basic
** OAuth2
* Retrieval at a configurable interval
* Pagination
* Retries
* Rate limiting
* Proxying
* Request transformations
* Response transformations

Example configurations:

["source","yaml",subs="attributes"]
----
filebeat.inputs:
# Fetch your public IP every minute.
- type: httpjson
  config_version: 2
  interval: 1m
  request.url: https://api.ipify.org/?format=json
  processors:
    - decode_json_fields:
        fields: ["message"]
        target: "json"
----

["source","yaml",subs="attributes"]
----
filebeat.inputs:
- type: httpjson
  config_version: 2
  request.url: http://localhost:9200/_search?scroll=5m
  request.method: POST
  response.split:
    target: body.hits.hits
  response.pagination:
    - set:
        target: url.value
        value: http://localhost:9200/_search/scroll
    - set:
        target: url.params.scroll_id
        value: '[[.last_response.body._scroll_id]]'
    - set:
        target: body.scroll
        value: 5m
----

Additionally, it supports authentication via Basic auth, HTTP Headers or oauth2.

Example configurations with authentication:

["source","yaml",subs="attributes"]
----
filebeat.inputs:
- type: httpjson
  config_version: 2
  request.url: http://localhost
  request.transforms:
    - set:
        target: header.Authorization
        value: 'Basic aGVsbG86d29ybGQ='
----

["source","yaml",subs="attributes"]
----
filebeat.inputs:
- type: httpjson
  config_version: 2
  auth.oauth2:
    client.id: 12345678901234567890abcdef
    client.secret: abcdef12345678901234567890
    token_url: http://localhost/oauth2/token
  request.url: http://localhost
----

[[input-state]]
==== Input state

The `httpjson` input keeps a runtime state between requests. This state can be accessed by some configuration options and transforms.

The state has the following elements:

- `last_response.url.value`: The full URL with params and fragments from the last request with a successful response.
- `last_response.url.params`: A https://pkg.go.dev/net/url#Values[`url.Values`] of the params from the URL in `last_response.url.value`.  Can be queried with the https://pkg.go.dev/net/url#Values.Get[`Get`] function.
- `last_response.header`: A map containing the headers from the last successful response.
- `last_response.body`: A map containing the parsed JSON body from the last successful response. This is the response as it comes from the remote server.
- `last_response.page`: A number indicating the page number of the last response. It starts with the value `0` at every interval.
- `first_event`: A map representing the first event sent to the output (result from applying transforms to `last_response.body`).
- `last_event`: A map representing the last event sent to the output (result from applying transforms to `last_response.body`).
- `url`: The last requested URL as a raw https://pkg.go.dev/net/url#URL[`url.URL`] Go type.
- `header`: A map containing the headers. References the next request headers when used in <<request-transforms>> or <<response-pagination>> configuration sections, and to the last response headers when used in <<response-transforms>>, <<response-split>>, or <<request-rate-limit>> configuration sections.
- `body`: A map containing the body. References the next request body when used in <<request-transforms>> or <<response-pagination>> configuration sections, and to the last response body when used in <<response-transforms>> or <<response-split>> configuration sections.
- `cursor`: A map containing any data the user configured to be stored between restarts (See <<cursor>>).

All of the mentioned objects are only stored at runtime, except `cursor`, which has values that are persisted between restarts.

[[transforms]]
==== Transforms

A transform is an action that lets the user modify the <<input-state,input state>>. Depending on where the transform is defined, it will have access for reading or writing different elements of the <<input-state,state>>.

The access limitations are described in the corresponding configuration sections.

[float]
==== `append`

Appends a value to an array. If the field does not exist, the first entry will create a new array. If the field exists, the value is appended to the existing field and converted to a list.

["source","yaml",subs="attributes"]
----
- append:
    target: body.foo.bar
    value: '[[.cursor.baz]]'
    default: "a default value"
----

- `target` defines the destination field where the value is stored.
- `value` defines the value that will be stored and it is a <<value-templates,value template>>.
- `default` defines the fallback value whenever `value` is empty or the template parsing fails. Default templates do not have access to any state, only to functions.
- `value_type` defines the type of the resulting value. Possible values are: `string`, `json`, and `int`. Default is `string`.
- `fail_on_template_error` if set to `true` an error will be returned and the request will be aborted when the template evaluation fails. Default is `false`.

[float]
==== `delete`

Deletes the target field.

["source","yaml",subs="attributes"]
----
- delete:
    target: body.foo.bar
----

- `target` defines the destination field to delete. If `target` is a list and not a single element, the complete list will be deleted.

[float]
==== `set`

Sets a value.

["source","yaml",subs="attributes"]
----
- set:
    target: body.foo.bar
    value: '[[.cursor.baz]]'
    default: "a default value"
----

- `target` defines the destination field where the value is stored.
- `value` defines the value that will be stored and it is a <<value-templates,value template>>.
- `default` defines the fallback value whenever `value` is empty or the template parsing fails. Default templates do not have access to any state, only to functions.
- `value_type` defines how the resulting value will be treated. Possible values are: `string`, `json`, and `int`. Default is `string`.
- `fail_on_template_error` if set to `true` an error will be returned and the request will be aborted when the template evaluation fails. Default is `false`.

[[value-templates]]
==== Value templates

Some configuration options and transforms can use value templates. Value templates are Go templates with access to the input state and to some built-in functions.
Please note that delimiters are changed from the default `{{ }}` to `[[ ]]` to improve interoperability with other templating mechanisms.

To see which <<input-state,state elements>> and operations are available, see the documentation for the option or <<transforms,transform>> where you want to use a value template.

A value template looks like:

["source","yaml",subs="attributes"]
----
- set:
    target: body.foo.bar
    value: '[[.cursor.baz]] more data'
    default: "a default value"
----

The content inside the brackets `[[` `]]` is evaluated. For more information on Go templates please refer to https://golang.org/pkg/text/template[the Go docs].

Some built-in helper functions are provided to work with the input state inside value templates:

- `parseDuration`: parses duration strings and returns `time.Duration`. Example: `[[parseDuration "1h"]]`.
- `now`: returns the current `time.Time` object in UTC. Optionally, it can receive a `time.Duration` as a parameter. Example: `[[now (parseDuration "-1h")]]` returns the time at 1 hour before now.
- `parseTimestamp`: parses a timestamp in seconds and returns a `time.Time` in UTC. Example: `[[parseTimestamp 1604582732]]` returns `2020-11-05 13:25:32 +0000 UTC`.
- `parseTimestampMilli`: parses a timestamp in milliseconds and returns a `time.Time` in UTC. Example: `[[parseTimestamp 1604582732000]]` returns `2020-11-05 13:25:32 +0000 UTC`.
- `parseTimestampNano`: parses a timestamp in nanoseconds and returns a `time.Time` in UTC. Example: `[[parseTimestamp 1604582732000000000]]` returns `2020-11-05 13:25:32 +0000 UTC`.
- `parseDate`: parses a date string and returns a `time.Time` in UTC. By default the expected layout is `RFC3339` but optionally can accept any of the Golang predefined layouts or a custom one. Example: `[[ parseDate "2020-11-05T12:25:32Z" ]]`, `[[ parseDate "2020-11-05T12:25:32.1234567Z" "RFC3339Nano" ]]`, `[[ (parseDate "Thu Nov  5 12:25:32 +0000 2020" "Mon Jan _2 15:04:05 -0700 2006").UTC ]]`.
- `formatDate`: formats a `time.Time`. By default the format layout is `RFC3339` but optionally can accept any of the Golang predefined layouts or a custom one. It will default to UTC timezone when formatting, but you can specify a different timezone. If the timezone is incorrect, it will default to UTC. Example: `[[ formatDate (now) "UnixDate" ]]`, `[[ formatDate (now) "UnixDate" "America/New_York" ]]`.
- `getRFC5988Link`: extracts a specific relation from a list of https://tools.ietf.org/html/rfc5988[RFC5988] links. It is useful when parsing header values for pagination. Example: `[[ getRFC5988Link "next" .last_response.header.Link ]]`.
- `toInt`: converts a value of any type to an integer when possible. Returns 0 if the conversion fails.
- `add`: adds a list of integers and returns their sum.
- `mul`: multiplies two integers.
- `div`: does the integer division of two integer values.
- `hmac`: calculates the hmac signature of a list of strings concatenated together. Returns a hex encoded signature. Supports sha1 or sha256. Example `[[hmac "sha256" "secret" "string1" "string2" (formatDate (now) "RFC1123")]]`
- `base64Encode`: Joins and base64 encodes all supplied strings. Example `[[base64Encode "string1" "string2"]]`
- `base64EncodeNoPad`: Joins and base64 encodes all supplied strings without padding. Example `[[base64EncodeNoPad "string1" "string2"]]`
- `base64Decode`: Decodes the base64 string. Any binary output will be converted to a UTF8 string.
- `base64DecodeNoPad`: Decodes the base64 string without padding. Any binary output will be converted to a UTF8 string.
- `join`: joins a list using the specified separator. Example: `[[join .body.arr ","]]`
- `sprintf`: formats according to a format specifier and returns the resulting string. Refer to https://pkg.go.dev/fmt#Sprintf[the Go docs] for usage. Example: `[[sprintf "%d:%q" 34 "quote this"]]`
- `hmacBase64`: calculates the hmac signature of a list of strings concatenated together. Returns a base64 encoded signature. Supports sha1 or sha256. Example `[[hmac "sha256" "secret" "string1" "string2" (formatDate (now) "RFC1123")]]`
- `uuid`: returns a random UUID such as `a11e8780-e3e7-46d0-8e76-f66e75acf019` Example: `[[ uuid ]]`

In addition to the provided functions, any of the native functions for https://golang.org/pkg/time/#Time[`time.Time`], https://golang.org/pkg/net/http/#Header[`http.Header`], and https://golang.org/pkg/net/url/#Values[`url.Values`] types can be used on the corresponding objects. Examples: `[[(now).Day]]`, `[[.last_response.header.Get "key"]]`

==== Configuration options

The `httpjson` input supports the following configuration options plus the
<<{beatname_lc}-input-{type}-common-options>> described later.

[float]
==== `config_version`

Defines the configuration version. Current supported versions are: `1` and `2`. Default: `1`.

NOTE: This setting defaults to `1` to avoid breaking current configurations. V1 configuration is deprecated and will be unsupported in future releases. Any new configuration should use `config_version: 2`.

[float]
==== `interval`

Duration between repeated requests. It may make additional pagination requests in response to the initial request if pagination is enabled. Default: `60s`.

[float]
==== `auth.basic.enabled`

When set to `false`, disables the basic auth configuration. Default: `true`.

NOTE: Basic auth settings are disabled if either `enabled` is set to `false` or
the `auth.basic` section is missing.

[float]
==== `auth.basic.user`

The user to authenticate with.

[float]
==== `auth.basic.password`

The password to use.

[float]
==== `auth.oauth2.enabled`

When set to `false`, disables the oauth2 configuration. Default: `true`.

NOTE: OAuth2 settings are disabled if either `enabled` is set to `false` or
the `auth.oauth2` section is missing.

[float]
==== `auth.oauth2.provider`

Used to configure supported oauth2 providers.
Each supported provider will require specific settings. It is not set by default.
Supported providers are: `azure`, `google`.

[float]
==== `auth.oauth2.client.id`

The client ID used as part of the authentication flow. It is always required
except if using `google` as provider. Required for providers: `default`, `azure`.

[float]
==== `auth.oauth2.client.secret`

The client secret used as part of the authentication flow. It is always required
except if using `google` as provider. Required for providers: `default`, `azure`.

[float]
==== `auth.oauth2.scopes`

A list of scopes that will be requested during the oauth2 flow.
It is optional for all providers.

[float]
==== `auth.oauth2.token_url`

The endpoint that will be used to generate the tokens during the oauth2 flow. It is required if no provider is specified.

NOTE: For `azure` provider either `token_url` or `azure.tenant_id` is required.

[float]
==== `auth.oauth2.endpoint_params`

Set of values that will be sent on each request to the `token_url`. Each param key can have multiple values.
Can be set for all providers except `google`.

["source","yaml",subs="attributes"]
----
- type: httpjson
  config_version: 2
  auth.oauth2:
    endpoint_params:
      Param1:
        - ValueA
        - ValueB
      Param2:
        - Value
----

[float]
==== `auth.oauth2.azure.tenant_id`

Used for authentication when using `azure` provider.
Since it is used in the process to generate the `token_url`, it can't be used in
combination with it. It is not required.

For information about where to find it, you can refer to
https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal.

[float]
==== `auth.oauth2.azure.resource`

The accessed WebAPI resource when using `azure` provider.
It is not required.

[float]
==== `auth.oauth2.google.credentials_file`

The credentials file for Google.

NOTE: Only one of the credentials settings can be set at once. If none is provided, loading
default credentials from the environment will be attempted via ADC. For more information about
how to provide Google credentials, please refer to https://cloud.google.com/docs/authentication.

[float]
==== `auth.oauth2.google.credentials_json`

Your credentials information as raw JSON.

NOTE: Only one of the credentials settings can be set at once. If none is provided, loading
default credentials from the environment will be attempted via ADC. For more information about
how to provide Google credentials, please refer to https://cloud.google.com/docs/authentication.

[float]
==== `auth.oauth2.google.jwt_file`

The JWT Account Key file for Google.

NOTE: Only one of the credentials settings can be set at once. If none is provided, loading
default credentials from the environment will be attempted via ADC. For more information about
how to provide Google credentials, please refer to https://cloud.google.com/docs/authentication.

[float]
==== `auth.oauth2.google.delegated_account`

Email of the delegated account used to create the credentials (usually an admin).

[float]
==== `request.url`

The URL of the HTTP API. Required.

[float]
==== `request.method`

HTTP method to use when making requests. `GET` or `POST` are the options. Default: `GET`.

[float]
==== `request.encode_as`

ContentType used for encoding the request body. If set it will force the encoding in the specified format regardless of the `Content-Type` header value, otherwise it will honor it if possible or fallback to `application/json`. By default the requests are sent with `Content-Type: application/json`. Supported values: `application/json` and `application/x-www-form-urlencoded`. `application/x-www-form-urlencoded` will url encode the `url.params` and set them as the body. It is not set by default.

[float]
==== `request.body`

An optional HTTP POST body. The configuration value must be an object, and it
will be encoded to JSON. This is only valid when `request.method` is `POST`.
Defaults to `null` (no HTTP body).

["source","yaml",subs="attributes"]
----
- type: httpjson
  config_version: 2
  request.method: POST
  request.body:
    query:
      bool:
        filter:
          term:
            type: authentication
----

[float]
==== `request.timeout`

Duration before declaring that the HTTP client connection has timed out. Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Default: `30s`.

[float]
==== `request.ssl`

This specifies SSL/TLS configuration. If the ssl section is missing, the host's
CAs are used for HTTPS connections. See <<configuration-ssl>> for more
information.

[float]
==== `request.proxy_url`

This specifies proxy configuration in the form of `http[s]://<user>:<password>@<server name/ip>:<port>`

["source","yaml",subs="attributes"]
----
filebeat.inputs:
# Fetch your public IP every minute.
- type: httpjson
  config_version: 2
  interval: 1m
  request.url: https://api.ipify.org/?format=json
  request.proxy_url: http://proxy.example:8080
----

[float]
==== `request.keep_alive.disable`

This specifies whether to disable keep-alives for HTTP end-points. Default: `true`.

[float]
==== `request.keep_alive.max_idle_connections`

The maximum number of idle connections across all hosts. Zero means no limit. Default: `0`.

[float]
==== `request.keep_alive.max_idle_connections_per_host`

The maximum idle connections to keep per-host. If zero, defaults to two. Default: `0`.

[float]
==== `request.keep_alive.idle_connection_timeout`

The maximum amount of time an idle connection will remain idle before closing itself. Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Zero means no limit. Default: `0s`.

[float]
==== `request.retry.max_attempts`

The maximum number of retries for the HTTP client. Default: `5`.

[float]
==== `request.retry.wait_min`

The minimum time to wait before a retry is attempted. Default: `1s`.

[float]
==== `request.retry.wait_max`

The maximum time to wait before a retry is attempted. Default: `60s`.

[float]
==== `request.redirect.forward_headers`

When set to `true` request headers are forwarded in case of a redirect. Default: `false`.

[float]
==== `request.redirect.headers_ban_list`

When `redirect.forward_headers` is set to `true`, all headers __except__ the ones defined in this list will be forwarded. Default: `[]`.

[float]
==== `request.redirect.max_redirects`

The maximum number of redirects to follow for a request. Default: `10`.

[[request-rate-limit]]
[float]
==== `request.rate_limit.limit`

The value of the response that specifies the total limit. It is defined with a Go template value.
Can read state from: [`.last_response.header`]

[float]
==== `request.rate_limit.remaining`

The value of the response that specifies the remaining quota of the rate limit.
It is defined with a Go template value. Can read state from: [`.last_response.header`]
If the `remaining` header is missing from the Response, no rate-limiting will occur.

[float]
==== `request.rate_limit.reset`

The value of the response that specifies the epoch time when the rate limit will reset.
It is defined with a Go template value. Can read state from: [`.last_response.header`]

[[request-transforms]]
[float]
==== `request.rate_limit.early_limit`

Optionally start rate-limiting prior to the value specified in the Response.

Under the default behavior, Requests will continue while the `remaining` value is non-zero.
Specifying an `early_limit` will mean that rate-limiting will occur prior to reaching `0`.

* If the value specified for `early_limit` is less than `1`,
the value is treated as a percentage of the Response provided `limit`.
e.g. specifying `0.9` will mean that Requests will continue until reaching 90% of the rate-limit --
for a `limit` value of `120`, the rate-limit starts when the `remaining` reaches `12`.
If the `limit` header is missing from the Response, default rate-limiting will occur (when `remaining` reaches `0`).
* If the value specified for `early_limit` is greater than or equal to `1`,
the value is treated as the target value for `remaining`.
e.g. instead of rate-limiting when `remaining` hits `0`, rate-limiting will occur when `remaining` hits the value specified.

It is not set by default (by default the rate-limiting as specified in the Response is followed).

[float]
==== `request.transforms`

List of transforms to apply to the request before each execution.

Available transforms for request: [`append`, `delete`, `set`].

Can read state from: [`.last_response.*`, `.last_event.*`, `.cursor.*`, `.header.*`, `.url.*`, `.body.*`].

Can write state to: [`body.*`, `header.*`, `url.*`].

["source","yaml",subs="attributes"]
----
filebeat.inputs:
- type: httpjson
  config_version: 2
  request.url: http://localhost:9200/_search?scroll=5m
  request.method: POST
  request.transforms:
    - set:
        target: body.from
        value: '[[now (parseDuration "-1h")]]'
----

[float]
==== `response.decode_as`

ContentType used for decoding the response body. If set it will force the decoding in the specified format regardless of the `Content-Type` header value, otherwise it will honor it if possible or fallback to `application/json`. Supported values: `application/json, application/x-ndjson`. It is not set by default.

[[response-transforms]]
[float]
==== `response.transforms`

List of transforms to apply to the response once it is received.

Available transforms for response: [`append`, `delete`, `set`].

Can read state from: [`.last_response.*`, `.last_event.*`, `.cursor.*`, `.header.*`, `.url.*`].

Can write state to: [`body.*`].

["source","yaml",subs="attributes"]
----
filebeat.inputs:
- type: httpjson
  config_version: 2
  request.url: http://localhost:9200/_search?scroll=5m
  request.method: POST
  response.transforms:
    - delete:
        target: body.very_confidential
  response.split:
    target: body.hits.hits
  response.pagination:
    - set:
        target: url.value
        value: http://localhost:9200/_search/scroll
    - set:
        target: url.params.scroll_id
        value: '[[.last_response.body._scroll_id]]'
    - set:
        target: body.scroll
        value: 5m
----

[[response-split]]
[float]
==== `response.split`

Split operation to apply to the response once it is received. A split can convert a map, array, or string into multiple events.

[float]
==== `response.split[].target`

Defines the target field upon the split operation will be performed.

[float]
==== `response.split[].type`

Defines the field type of the target. Allowed values: `array`, `map`, `string`. `string` requires the use of the `delimiter` options to specify what characters to split the string on.  `delimiter` always behaves as if `keep_parent` is set to `true`. Default: `array`.

[float]
==== `response.split[].transforms`

A set of transforms can be defined. This list will be applied after `response.transforms` and after the object has been modified based on `response.split[].keep_parent` and `response.split[].key_field`.

Available transforms for response: [`append`, `delete`, `set`].

Can read state from: [`.last_response.*`, `.first_event.*`, `.last_event.*`, `.cursor.*`, `.header.*`, `.url.*`].

Can write state to: [`body.*`].

NOTE: in this context, `body.*` will be the result of all the previous transformations.

[float]
==== `response.split[].keep_parent`

If set to true, the fields from the parent document (at the same level as `target`) will be kept. Otherwise a new document will be created using `target` as the root. Default: `false`.

[float]
==== `response.split[].delimiter`

Required if using split type of `string`.  This is the sub string used to split the string.  For example if `delimiter` was "\n" and the string was "line 1\nline 2", then the split would result in "line 1" and "line 2".

[float]
==== `response.split[].key_field`

Valid when used with `type: map`. When not empty, defines a new field where the original key value will be stored.

[float]
==== `response.split[].ignore_empty_value`

If set to true, empty or missing value will be ignored and processing will pass on to the next nested split operation instead of failing with an error. Default: `false`.

[float]
==== `response.split[].split`

Nested split operation. Split operations can be nested at will. An event won't be created until the deepest split operation is applied.

[float]
==== `response.request_body_on_pagination`

If set to true, the values in `request.body` are sent for pagination requests. Default: `false`.

[[response-pagination]]
[float]
==== `response.pagination`

List of transforms that will be applied to the response to every new page request. All the transforms from `request.transform` will be executed and then `response.pagination` will be added to modify the next request as needed. For subsequent responses, the usual `response.transforms` and `response.split` will be executed normally.

Available transforms for pagination: [`append`, `delete`, `set`].

Can read state from: [`.last_response.*`, `.first_event.*`, `.last_event.*`, `.cursor.*`, `.header.*`, `.url.*`, `.body.*`].

Can write state to: [`body.*`, `header.*`, `url.*`].

Examples using split:

- We have a response with two nested arrays, and we want a document for each of the elements of the inner array:

+
["source","json",subs="attributes"]
----
{
  "this": "is kept",
  "alerts": [
    {
      "this_is": "also kept",
      "entities": [
        {
          "something": "something"
        },
        {
          "else": "else"
        }
      ]
    },
    {
      "this_is": "also kept 2",
      "entities": [
        {
          "something": "something 2"
        },
        {
          "else": "else 2"
        }
      ]
    }
  ]
}
----

+
The config will look like:

+
["source","yaml",subs="attributes"]
----
filebeat.inputs:
- type: httpjson
  config_version: 2
  interval: 1m
  request.url: https://example.com
  response.split:
    target: body.alerts
    type: array
    keep_parent: true
    split:
      # paths in nested splits need to represent the state of body, not only their current level of nesting
      target: body.alerts.entities
      type: array
      keep_parent: true
----

+
This will output:

+
["source","json",subs="attributes"]
----
[
  {
    "this": "is kept",
    "alerts": {
      "this_is": "also kept",
      "entities": {
        "something": "something"
      }
    }
  },
  {
    "this": "is kept",
    "alerts": {
      "this_is": "also kept",
      "entities": {
        "else": "else"
      }
    }
  },
  {
    "this": "is kept",
    "alerts": {
      "this_is": "also kept 2",
      "entities": {
        "something": "something 2"
      }
    }
  },
  {
    "this": "is kept",
    "alerts": {
      "this_is": "also kept 2",
      "entities": {
        "else": "else 2"
      }
    }
  }
]
----

- We have a response with an array with two objects, and we want a document for each of the object keys while keeping the keys values:

+
["source","json",subs="attributes"]
----
{
  "this": "is not kept",
  "alerts": [
    {
      "this_is": "kept",
      "entities": {
        "id1": {
          "something": "something"
        }
      }
    },
    {
      "this_is": "kept 2",
      "entities": {
        "id2": {
          "something": "something 2"
        }
      }
    }
  ]
}
----

+
The config will look like:

+
["source","yaml",subs="attributes"]
----
filebeat.inputs:
- type: httpjson
  config_version: 2
  interval: 1m
  request.url: https://example.com
  response.split:
    target: body.alerts
    type: array
    keep_parent: false
    split:
      # this time alerts will not exist because previous keep_parent is false
      target: body.entities
      type: map
      keep_parent: true
      key_field: id
----

+
This will output:

+
["source","json",subs="attributes"]
----
[
  {
    "this_is": "kept",
    "entities": {
      "id": "id1",
      "something": "something"
    }
  },
  {
    "this_is": "kept 2",
    "entities": {
      "id": "id2",
      "something": "something 2"
    }
  }
]
----

- We have a response with an array with two objects, and we want a document for each of the object keys while applying a transform to each:

+
["source","json",subs="attributes"]
----
{
  "this": "is not kept",
  "alerts": [
    {
      "this_is": "also not kept",
      "entities": {
        "id1": {
          "something": "something"
        }
      }
    },
    {
      "this_is": "also not kept",
      "entities": {
        "id2": {
          "something": "something 2"
        }
      }
    }
  ]
}
----

+
The config will look like:

+
["source","yaml",subs="attributes"]
----
filebeat.inputs:
- type: httpjson
  config_version: 2
  interval: 1m
  request.url: https://example.com
  response.split:
    target: body.alerts
    type: array
    split:
      transforms:
        - set:
            target: body.new
            value: will be added to each
      target: body.entities
      type: map
----

+
This will output:

+
["source","json",subs="attributes"]
----
[
  {
    "something": "something",
    "new": "will be added for each"
  },
  {
    "something": "something 2",
    "new": "will be added for each"
  }
]
----

- We have a response with a keys whose value is a string.  We want the string to be split on a delimiter and a document for each sub strings.

+
["source","json",subs="attributes"]
----
{
  "this": "is kept",
  "lines": "Line 1\nLine 2\nLine 3"
}
----

+
The config will look like:

+
["source","yaml",subs="attributes"]
----
filebeat.inputs:
- type: httpjson
  config_version: 2
  interval: 1m
  request.url: https://example.com
  response.split:
    target: body.lines
    type: string
    delimiter: "\n"
----

+
This will output:

+
["source","json",subs="attributes"]
----
[
  {
    "this": "is kept",
    "lines": "Line 1"
  },
  {
    "this": "is kept",
    "lines": "Line 2"
  },
  {
    "this": "is kept",
    "lines": "Line 3"
  }
]
----

[[cursor]]
[float]
==== `cursor`

Cursor is a list of key value objects where arbitrary values are defined. The values are interpreted as  <<value-templates,value templates>> and a default template can be set. Cursor state is kept between input restarts and updated once all the events for a request are published.

Each cursor entry is formed by:

- A `value` template, which will define the value to store when evaluated.
- A `default` template, which will define the value to store when the value template fails or is empty.
- An `ignore_empty_value` flag. When set to `true`, will not store empty values, preserving the previous one, if any. Default: `true`.

Can read state from: [`.last_response.*`, `.first_event.*`, `.last_event.*`].

NOTE: Default templates do not have access to any state, only to functions.

["source","yaml",subs="attributes"]
----
filebeat.inputs:
- type: httpjson
  config_version: 2
  interval: 1m
  request.url: https://api.ipify.org/?format=json
  response.transforms:
    - set:
        target: body.last_requested_at
        value: '[[.cursor.last_requested_at]]'
        default: "[[now]]"
  cursor:
    last_requested_at:
      value: '[[now]]'
  processors:
    - decode_json_fields:
        fields: ["message"]
        target: "json"
----

[float]
==== `api_key`

Deprecated, use `request.transforms`.

[float]
==== `http_client_timeout`

Deprecated, use `request.timeout`.

[float]
==== `http_headers`

Deprecated, use `request.transforms`.

[float]
==== `http_method`

Deprecated, use `request.method`.

[float]
==== `http_request_body`

Deprecated, use `request.body`.

[float]
==== `json_objects_array`

Deprecated, use `request.split`.

[float]
==== `split_events_by`

Deprecated, use `request.split`.

[float]
==== `no_http_body`

Deprecated.

[float]
==== `pagination.*`

Deprecated, use `response.pagination`.

[float]
==== `rate_limit.*`

Deprecated, use `request.rate_limit.*`.

[float]
==== `retry.*`

Deprecated, use `request.retry.*`.

[float]
==== `ssl`

Deprecated, use `request.ssl`.

[float]
==== `url`

Deprecated, use `request.url`.

[float]
==== `oauth2.*`

Deprecated, use `auth.oauth2.*`.

==== Request life cycle

image:images/input-httpjson-lifecycle.png[Request lifecycle]

. At every defined interval a new request is created.
. The request is transformed using the configured `request.transforms`.
. The resulting transformed request is executed.
. The server responds (here is where any retry or rate limit policy takes place when configured).
. The response is transformed using the configured `response.transforms` and `response.split`.
. Each resulting event is published to the output.
. If a `response.pagination` is configured and there are more pages, a new request is created using it, otherwise the process ends until the next interval.

[id="{beatname_lc}-input-{type}-common-options"]
include::../../../../filebeat/docs/inputs/input-common-options.asciidoc[]

:type!:
