[[migrate-to-filestream]]
== Migrate `log` input configurations to `filestream`

The `filestream` input has been generally available since 7.14. So it is high time you
migrate your existing `log` input configurations.  The `filestream` input comes with many
improvements over the old input, such as configurable order for parsers and more.

While we do not plan to remove the `log` input from Filebeat, we are not fixing
new issues or adding any enhancements to the input. Our focus is on `filestream`.

In this guide, you learn how to migrate an existing `log` input configuration.
The following example shows three `log` inputs:

[source,yaml]
----
filebeat.inputs:
 - type: log
   enabled: true
   paths:
     - /var/log/java-exceptions*.log
   multiline:
    pattern: '^\['
    negate: true
    match: after
  close_removed: true
  close_renamed: true

- type: log
  enabled: true
  paths:
    - /var/log/my-application*.json
  scan_frequency: 1m
  json.keys_under_root: true

- type: log
  enabled: true
  paths:
    - /var/log/my-old-files*.log
  tail_files: true
----

For this example, let's assume that the `log` input is used to collect logs from the following files. The progress of data collection is shown for each file.
["source","sh",subs="attributes"]
----
/var/log/java-exceptions1.log (100%)
/var/log/java-exceptions2.log (100%)
/var/log/java-exceptions3.log (75%)
/var/log/java-exceptions4.log (0%)
/var/log/java-exceptions5.log (0%)
/var/log/my-application1.json (100%)
/var/log/my-application2.json (5%)
/var/log/my-application3.json (0%)
/var/log/my-old-files1.json (0%)
----

=== Step 1: Set an identifier for each `filestream` input

All `filestream` inputs require an ID. So make sure you set a unique identifier for every input. 

IMPORTANT: Never change the ID of an input, or you will end up with duplicate events.

[source,yaml]
----
filebeat.inputs:
- type: filestream
  enabled: true
  id: my-java-collector
  paths:
    - /var/log/java-exceptions*.log

- type: filestream
  enabled: true
  id: my-application-input
  paths:
    - /var/log/my-application*.json

- type: filestream
  enabled: true
  id: my-old-files
  paths:
    - /var/log/my-old-files*.log
----

=== Step 2: Exclude all processed files

Filebeat does not provide access to the state information of different inputs.
Hence, the `filestream` input cannot access the state information of a `log` input in the
Filebeat registry. You must exclude the files the `log` input has processed
or is processing. If you do not exclude those files, you will end up with
duplicate events in the output.

Given the file list and ingestion progress shown earlier, 
you should run the `log` and `filestream` inputs simultaneously until everything
collected by the `log` input has made it to the output.
After the files collected by the `log` input are shipped and the files are deleted,
you can delete the `log` inputs and the `exlude_files` settings from `filestream` input.

[source,yaml]
----
filebeat.inputs:
 - type: log
   enabled: true
   paths:
     - /var/log/java-exceptions*.log
   multiline:
    pattern: '^\['
    negate: true
    match: after
  close_removed: true
  close_renamed: true
  exclude_files: java-exceptions[4-5]{1}.log

- type: log
  enabled: true
  paths:
    - /var/log/my-application*.json
  scan_frequency: 1m
  json.keys_under_root: true
  exclude_files: my-application3.log

- type: filestream
  enabled: true
  id: my-java-collector
  paths:
    - /var/log/java-exceptions*.log
  prospector.scanner.exclude_files: java-exceptions[1-3]{1}.log

- type: filestream
  enabled: true
  id: my-application-input
  paths:
    - /var/log/my-application*.json
  prospector.scanner.exclude_files: my-application[1-2]{1}.log

- type: filestream
  enabled: true
  id: my-old-files
  paths:
    - /var/log/my-old-files*.log
----


=== Step 3: Use new option names

Several options are renamed in `filestream`. You can find a table with all of the
changed configuration names at the end of this guide.

The most significant change you have to know about is in parsers. The configuration of
`multiline`, `json`, and other parsers has changed. Now the ordering is
configurable, so `filestream` expects a list of parsers. Furthermore, the `json`
parser was renamed to `ndjson`.

The example configuration shown earlier needs to be adjusted as well:

[source,yaml]
----
- type: filestream
  enabled: true
  id: my-java-collector
  paths:
    - /var/log/java-exceptions*.log
  parsers:
    - multiline:
        pattern: '^\['
        negate: true
        match: after
  close.on_state_change.removed: true
  close.on_state_change.renamed: true

- type: filestream
  enabled: true
  id: my-application-input
  paths:
    - /var/log/my-application*.json
  prospector.scanner.check_interval: 1m
  parsers:
    - ndjson:
        keys_under_root: true

- type: filestream
  enabled: true
  id: my-old-files
  paths:
    - /var/log/my-old-files*.log
  ignore_inactive: since_last_start
----

[cols="1,1"]
|===
|Option name in log input
|Option name in filestream input

|recursive_glob.enabled
|prospector.scanner.recursive_glob

|harvester_buffer_size
|buffer_size

|max_bytes
|message_max_bytes

|json
|parsers.n.ndjson

|multiline
|parsers.n.mutiline

|exclude_files
|prospector.scanner.exclude_files

|close_inactive
|close.on_state_change.inactive

|close_removed
|close.on_state_change.removed

|close_eof
|close.reader.on_eof

|close_timeout
|close.reader.after_interval

|close_inactive
|close.on_state_change.inactive

|scan_frequency
|prospector.scanner.check_interval

|tail_files
|ignore_inactive.since_last_start

|symlinks
|prospector.scanner.symlinks

|backoff
|backoff.init

|backoff_max
|backoff.max
|===



