# repmgrd setup and configuration

&repmgrd; is a daemon process which runs on each PostgreSQL node, monitoring the local node, and (unless it's the primary node) the upstream server (the primary server or with cascading replication, another standby) which it's connected to.

&repmgrd; can be configured to provide failover capability in case the primary or upstream node becomes unreachable, and/or provide monitoring data to the &repmgr; metadatabase.

From &repmgr; 4.4, when running on the primary node, &repmgrd; can also monitor standby disconnections/reconnections (see repmgrd-primary-child-disconnection).

## repmgrd configuration

To use &repmgrd;, its associated function library must be included via `postgresql.conf` with:

            shared_preload_libraries = 'repmgr'

Changing this setting requires a restart of PostgreSQL; for more details see the PostgreSQL documentation.

The following configuraton options apply to &repmgrd; in all circumstances:

`monitor_interval_secs`

- The interval (in seconds, default: `2`) to check the availability of the upstream node.

`connection_check_type`

- The option `connection_check_type` is used to select the method &repmgrd; uses to determine whether the upstream node is available.

  Possible values are:

  - `ping` (default) - uses `PQping()` to determine server availability

  - `connection` - determines server availability by attempting to make a new connection to the upstream node

  - `query` - determines server availability by executing an SQL statement on the node via the existing connection

    The query is a minimal throwaway query - `SELECT 1` - which is used to determine that the server can accept queries.

`reconnect_attempts`

- The number of attempts (default: `6`) will be made to reconnect to an unreachable upstream node before initiating a failover.

  There will be an interval of `reconnect_interval` seconds between each reconnection attempt.

`reconnect_interval`

- Interval (in seconds, default: `10`) between attempts to reconnect to an unreachable upstream node.

  The number of reconnection attempts is defined by the parameter `reconnect_attempts`.

`degraded_monitoring_timeout`

- Interval (in seconds) after which &repmgrd; will terminate if either of the servers (local node and or upstream node) being monitored is no longer available (degraded monitoring mode).

  `-1` (default) disables this timeout completely.

See also `repmgr.conf.sample` for an annotated sample configuration file.

### Required configuration for automatic failover

The following &repmgrd; options must be set in `repmgr.conf`:

- `failover`

- `promote_command`

- `follow_command`

Example:

              failover=automatic
              promote_command='/usr/bin/repmgr standby promote -f /etc/repmgr.conf --log-to-file'
              follow_command='/usr/bin/repmgr standby follow -f /etc/repmgr.conf --log-to-file --upstream-node-id=%n'

Details of each option are as follows:

`failover`

- `failover` can be one of `automatic` or `manual`.

  If `failover` is set to `manual`, &repmgrd; will not take any action if a failover situation is detected, and the node may need to be modified manually (e.g. by executing `repmgr standby follow`).

`promote_command`

- The program or script defined in `promote_command` will be executed in a failover situation when &repmgrd; determines that the current node is to become the new primary node.

  Normally `promote_command` is set as &repmgr;'s `repmgr standby promote` command.

  When invoking `repmgr standby promote` (either directly via the `promote_command`, or in a script called via `promote_command`), `--siblings-follow` must not be included as a command line option for `repmgr standby promote`.

  It is also possible to provide a shell script to e.g. perform user-defined tasks before promoting the current node. In this case the script must at some point execute `repmgr standby promote` to promote the node; if this is not done, &repmgr; metadata will not be updated and &repmgr; will no longer function reliably.

  Example:

                      promote_command='/usr/bin/repmgr standby promote -f /etc/repmgr.conf --log-to-file'

  Note that the `--log-to-file` option will cause output generated by the &repmgr; command, when executed by &repmgrd;, to be logged to the same destination configured to receive log output for &repmgrd;.

  &repmgr; will not apply `pg_bindir` when executing `promote_command` or `follow_command`; these can be user-defined scripts so must always be specified with the full path.

`follow_command`

- The program or script defined in `follow_command` will be executed in a failover situation when &repmgrd; determines that the current node is to follow the new primary node.

  Normally `follow_command` is set as &repmgr;'s `repmgr standby follow` command.

  The `follow_command` parameter should provide the `--upstream-node-id=%n` option to `repmgr standby follow`; the `%n` will be replaced by &repmgrd; with the ID of the new primary node. If this is not provided, `repmgr standby follow` will attempt to determine the new primary by itself, but if the original primary comes back online after the new primary is promoted, there is a risk that `repmgr standby follow` will result in the node continuing to follow the original primary.

  It is also possible to provide a shell script to e.g. perform user-defined tasks before promoting the current node. In this case the script must at some point execute `repmgr standby follow` to promote the node; if this is not done, &repmgr; metadata will not be updated and &repmgr; will no longer function reliably.

  Example:

                follow_command='/usr/bin/repmgr standby follow -f /etc/repmgr.conf --log-to-file --upstream-node-id=%n'

  Note that the `--log-to-file` option will cause output generated by the &repmgr; command, when executed by &repmgrd;, to be logged to the same destination configured to receive log output for &repmgrd;.

  &repmgr; will not apply `pg_bindir` when executing `promote_command` or `follow_command`; these can be user-defined scripts so must always be specified with the full path.

### Optional configuration for automatic failover

The following configuraton options can be used to fine-tune automatic failover:

`priority`

- Indicates a preferred priority (default: `100`) for promoting nodes.

  Note that the priority setting is only applied if two or more nodes are determined as promotion candidates; in that case the node with the higher priority is selected.

  A value of zero will always prevent the node being promoted to primary, even if there is no other promotion candidate.

`failover_validation_command`

- User-defined script to execute for an external mechanism to validate the failover decision made by &repmgrd;.

  This option must be identically configured on all nodes.

  One or more of the following parameter placeholders may be provided, which will be replaced by repmgrd with the appropriate value:

  - `%n`: node ID

  - `%a`: node name

  - `%v`: number of visible nodes

  - `%u`: number of shared upstream nodes

  - `%t`: total number of nodes

  See also: Failover validation.

`primary_visibility_consensus`

- If `true`, only continue with failover if no standbys (or the witness server, if present) have seen the primary node recently.

  This option must be identically configured on all nodes.

`always_promote`

- Default: `false`.

  If `true`, promote the local node even if its &repmgr; metadata is not up-to-date.

  Normally &repmgr; expects its metadata (stored in the `repmgr.nodes` table) to be up-to-date so &repmgrd; can take the correct action during a failover. However it's possible that updates made on the primary may not have propagated to the standby (promotion candidate). In this case &repmgrd; will default to not promoting the standby. This behaviour can be overridden by setting `always_promote` to `true`.

`standby_disconnect_on_failover`

- In a failover situation, disconnect the local node's WAL receiver.

  This option is available from PostgreSQL 9.5 and later.

  This option must be identically configured on all nodes.

  Additionally the &repmgr; user must be a superuser for this option.

  &repmgrd; will refuse to start if this option is set but either of these prerequisites is not met.

  See also: Standby disconnection on failover.

`repmgrd_exit_on_inactive_node`

- This parameter is available in &repmgr; 5.3 and later.

  If a node was marked as inactive but is running, and this option is set to `true`, &repmgrd; will abort on startup.

  By default, `repmgrd_exit_on_inactive_node` is set to `false`, in which case &repmgrd; will set the node record to active on startup.

  Setting this parameter to `true` causes &repmgrd; to behave in the same way it did in &repmgr; 5.2 and earlier.

The following options can be used to further fine-tune failover behaviour. In practice it's unlikely these will need to be changed from their default values, but are available as configuration options should the need arise.

`election_rerun_interval`

- If `failover_validation_command` is set, and the command returns an error, pause the specified amount of seconds (default: 15) before rerunning the election.

`sibling_nodes_disconnect_timeout`

- If `standby_disconnect_on_failover` is `true`, the maximum length of time (in seconds, default: `30`) to wait for other standbys to confirm they have disconnected their WAL receivers.

### Configuring &repmgrd; and pgbouncer to fence a failed primary node

For further details and a reference implementation, see the separate document Fencing a failed master node with repmgrd and PgBouncer.

### PostgreSQL service configuration

If using automatic failover, currently &repmgrd; will need to execute `repmgr standby follow` to restart PostgreSQL on standbys to have them follow a new primary.

To ensure this happens smoothly, it's essential to provide the appropriate system/service restart command appropriate to your operating system via `service_restart_command` in `repmgr.conf`. If you don't do this, &repmgrd; will default to using `pg_ctl`, which can result in unexpected problems, particularly on systemd-based systems.

For more details, see configuration-file-service-commands.

### repmgrd service configuration

If you are intending to use the `repmgr daemon start` and `repmgr daemon stop` commands, the following parameters must be set in `repmgr.conf`:

- `repmgrd_service_start_command`

- `repmgrd_service_stop_command`

Example (for &repmgr; with PostgreSQL 12 on CentOS 7):

    repmgrd_service_start_command='sudo systemctl repmgr12 start'
    repmgrd_service_stop_command='sudo systemctl repmgr12 stop'

For more details see the reference page for each command.

### Monitoring configuration

To enable monitoring, set:

              monitoring_history=yes

in `repmgr.conf`.

Monitoring data is written at the interval defined by the option `monitor_interval_secs` (see above).

For more details on monitoring, see repmgrd-monitoring. For information on monitoring standby disconnections, see repmgrd-primary-child-disconnection.

### Applying configuration changes to repmgrd

To apply configuration file changes to a running &repmgrd; daemon, execute the operating system's &repmgrd; service reload command (see appendix-packages for examples), or for instances which were manually started, execute `kill -HUP`, e.g. `kill -HUP `cat /tmp/repmgrd.pid``.

Check the &repmgrd; log to see what changes were applied, or if any issues were encountered when reloading the configuration.

Note that only the following subset of configuration file parameters can be changed on a running &repmgrd; daemon:

- `async_query_timeout`

- `child_nodes_check_interval`

- `child_nodes_connected_include_witness`

- `child_nodes_connected_min_count`

- `child_nodes_disconnect_command`

- `child_nodes_disconnect_min_count`

- `child_nodes_disconnect_timeout`

- `connection_check_type`

- `conninfo`

- `degraded_monitoring_timeout`

- `event_notification_command`

- `event_notifications`

- `failover_validation_command`

- `failover`

- `follow_command`

- `log_facility`

- `log_file`

- `log_level`

- `log_status_interval`

- `monitor_interval_secs`

- `monitoring_history`

- `primary_notification_timeout`

- `primary_visibility_consensus`

- `always_promote`

- `promote_command`

- `reconnect_attempts`

- `reconnect_interval`

- `retry_promote_interval_secs`

- `repmgrd_standby_startup_timeout`

- `sibling_nodes_disconnect_timeout`

- `standby_disconnect_on_failover`

The following set of configuration file parameters must be updated via `repmgr standby register --force`, as they require changes to the `repmgr.nodes` table so they are visible to all nodes in the replication cluster:

- `node_id`

- `node_name`

- `data_directory`

- `location`

- `priority`

After executing `repmgr standby register --force`, &repmgrd; must be restarted for the changes to take effect.

## repmgrd daemon

If installed from a package, the &repmgrd; can be started via the operating system's service command, e.g. in systemd using `systemctl`.

See appendix appendix-packages for details of service commands for different distributions.

The commands `repmgr daemon start` and `repmgr daemon stop` can be used as convenience wrappers to start and stop &repmgrd; on the local node.

`repmgr daemon start` and `repmgr daemon stop` require that the appropriate start/stop commands are configured as `repmgrd_service_start_command` and `repmgrd_service_stop_command` in `repmgr.conf`.

&repmgrd; can be started manually like this:

            repmgrd -f /etc/repmgr.conf --pid-file /tmp/repmgrd.pid

and stopped with `kill `cat /tmp/repmgrd.pid``. Adjust paths as appropriate.

### repmgrd's PID file

&repmgrd; will generate a PID file by default.

This is a behaviour change from previous versions (earlier than 4.1), where the PID file had to be explicitly specified with the command line parameter `--pid-file`.

The PID file can be specified in `repmgr.conf` with the configuration parameter `repmgrd_pid_file`.

It can also be specified on the command line (as in previous versions) with the command line parameter `--pid-file`. Note this will override any value set in `repmgr.conf` with `repmgrd_pid_file`. `--pid-file` may be deprecated in future releases.

If a PID file location was specified by the package maintainer, &repmgrd; will use that. This only applies if &repmgr; was installed from a package and the package maintainer has specified the PID file location.

If none of the above apply, &repmgrd; will create a PID file in the operating system's temporary directory (as determined by the environment variable `TMPDIR`, or if that is not set, will use `/tmp`).

To prevent a PID file being generated at all, provide the command line option `--no-pid-file`.

To see which PID file &repmgrd; would use, execute &repmgrd; with the option `--show-pid-file`. &repmgrd; will not start if this option is provided. Note that the value shown is the file &repmgrd; would use next time it starts, and is not necessarily the PID file currently in use.

### repmgrd daemon configuration on Debian/Ubuntu

If &repmgr; was installed from Debian/Ubuntu packages, additional configuration is required before &repmgrd; is started as a daemon.

This is done via the file `/etc/default/repmgrd`, which by default looks like this:

    # default settings for repmgrd. This file is source by /bin/sh from
    # /etc/init.d/repmgrd

    # disable repmgrd by default so it won't get started upon installation
    # valid values: yes/no
    REPMGRD_ENABLED=no

    # configuration file (required)
    #REPMGRD_CONF="/path/to/repmgr.conf"

    # additional options
    REPMGRD_OPTS="--daemonize=false"

    # user to run repmgrd as
    #REPMGRD_USER=postgres

    # repmgrd binary
    #REPMGRD_BIN=/usr/bin/repmgrd

    # pid file
    #REPMGRD_PIDFILE=/var/run/repmgrd.pid

Set `REPMGRD_ENABLED` to `yes`, and `REPMGRD_CONF` to the `repmgr.conf` file you are using.

See packages-debian-ubuntu for details of the Debian/Ubuntu packages and typical file locations (including `repmgr.conf`).

From &repmgrd; 4.1, ensure `REPMGRD_OPTS` includes `--daemonize=false`, as daemonization is handled by the service command.

If using systemd, you may need to execute `systemctl daemon-reload`. Also, if you attempted to start &repmgrd; using `systemctl start repmgrd`, you'll need to execute `systemctl stop repmgrd`. Because that's how systemd rolls.

### repmgrd daemon monitoring

The command `repmgr service status` provides an overview of the &repmgrd; daemon status (including pause status) on all nodes in the cluster.

From &repmgr; 5.3, `repmgr node check --repmgrd` can be used to check the status of &repmgrd; (including pause status) on the local node.

## repmgrd connection settings

In addition to the &repmgr; configuration settings, parameters in the `conninfo` string influence how &repmgr; makes a network connection to PostgreSQL. In particular, if another server in the replication cluster is unreachable at network level, system network settings will influence the length of time it takes to determine that the connection is not possible.

In particular explicitly setting a parameter for `connect_timeout` should be considered; the effective minimum value of `2` (seconds) will ensure that a connection failure at network level is reported as soon as possible, otherwise depending on the system settings (e.g. `tcp_syn_retries` in Linux) a delay of a minute or more is possible.

For further details on `conninfo` network connection parameters, see the PostgreSQL documentation.

## repmgrd log rotation

To ensure the current &repmgrd; logfile (specified in `repmgr.conf` with the parameter `log_file`) does not grow indefinitely, configure your system's `logrotate` to regularly rotate it.

Sample configuration to rotate logfiles weekly with retention for up to 52 weeks and rotation forced if a file grows beyond 100Mb:

        /var/log/repmgr/repmgrd.log {
            missingok
            compress
            rotate 52
            maxsize 100M
            weekly
            create 0600 postgres postgres
            postrotate
                /usr/bin/killall -HUP repmgrd
            endscript
        }