[[file-log-rotation]]
=== Log rotation results in lost or duplicate events

{beatname_uc} supports reading from rotating log files. However, some log
rotation strategies can result in lost or duplicate events when using
{beatname_uc} to forward messages. To resolve this issue:

* *Avoid log rotation strategies that copy and truncate log files* 
+
Log rotation strategies that copy and truncate the input log file can result in
{beatname_uc} sending duplicate events. This happens because {beatname_uc}
identifies files by inode and device name. During log rotation, lines that
{beatname_uc} has already processed are moved to a new file. When
{beatname_uc} encounters the new file, it reads from the beginning because the
previous state information (the offset and read timestamp) is associated with the
inode and device name of the old file.
+
Furthermore, strategies that copy and truncate the input log file can result in
lost events if lines are written to the log file after it's copied, but before
it's truncated.

* *Make sure {beatname_uc} is configured to read from all rotated logs*
+
When an input log file is moved or renamed during log rotation, {beatname_uc} is
able to recognize that the file has already been read. After the file is
rotated, a new log file is created, and the application continues logging.
{beatname_uc} picks up the new file during the next scan. Because the file
has a new inode and device name, {beatname_uc} starts reading it from the
beginning.
+
To avoid missing events from a rotated file, configure the input to read from
the log file and all the rotated files. For examples, see
<<log-rotate-example>>.

If you're using Windows, also see <<log-rotation-windows>>.

[float]
[[log-rotate-example]]
==== Example configurations

This section shows a typical configuration for logrotate, a popular tool for
doing log rotation on Linux, followed by a {beatname_uc} configuration that
reads all the rotated logs.

[float]
[[log-rotate-example-logrotate]]
===== logrotate.conf

In this example, {beatname_uc} reads web server log. The logs are rotated every
day, and the new file is created with the specified permissions.

[source,yaml]
-----------------------------------------------------
/var/log/my-server/my-server.log {
    daily
    missingok
    rotate 7
    notifempty
    create 0640 www-data www-data
}
-----------------------------------------------------

[float]
[[log-rotate-example-filebeat]]
===== filebeat.yml

In this example, {beatname_uc} is configured to read all log files to make
sure it does not miss any events.

[source,yaml]
-----------------------------------------------------
filebeat.inputs:
- type: filestream
  id: my-server-filestream-id
  paths:
  - /var/log/my-server/my-server.log*
-----------------------------------------------------

[float]
[[log-rotation-windows]]
==== More about log rotation on Windows

On Windows, log rotation schemes that delete old files and rename newer
files to old filenames might get blocked if the old files are being processed by
{beatname_uc}. This happens because Windows does not delete files and file
metadata until the last process has closed the file. Unlike most *nix
filesystems, a Windows filename cannot be reused until all processes accessing
the file have closed the deleted file.

To avoid this problem, use dates in rotated filenames. The file will never
be renamed to an older filename, and the log writer and log rotator will always
be able to open the file. This approach also highly reduces the chance of
log writing, rotation, and collection interfering with each other.

Because log rotation is typically handled by the logging application, we are
not providing an example configuration for Windows.

Also read <<windows-file-rotation>>.
