# Cloning standbys

## Cloning a standby from Barman

repmgr-standby-clone can use EDB's Barman application to clone a standby (and also as a fallback source for WAL files).

Barman (aka PgBarman) should be considered as an integral part of any PostgreSQL replication cluster. For more details see: https://www.pgbarman.org/.

Barman support provides the following advantages:

- the primary node does not need to perform a new backup every time a new standby is cloned

- a standby node can be disconnected for longer periods without losing the ability to catch up, and without causing accumulation of WAL files on the primary node

- WAL management on the primary becomes much easier as there's no need to use replication slots, and `wal_keep_segments` (PostgreSQL 13 and later: `wal_keep_size`) does not need to be set.

Currently &repmgr;'s support for cloning from Barman is implemented by using rsync to clone from the Barman server.

It is therefore not able to make use of Barman's parallel restore facility, which is executed on the Barman server and clones to the target server.

Barman's parallel restore facility can be used by executing it manually on the Barman server and configuring replication on the resulting cloned standby using `repmgr standby clone --replication-conf-only`.

### Prerequisites for cloning from Barman

In order to enable Barman support for `repmgr standby clone`, following prerequisites must be met:

- the Barman catalogue must include at least one valid backup for this server;

- the `barman_host` setting in `repmgr.conf` is set to the SSH hostname of the Barman server;

- the `barman_server` setting in `repmgr.conf` is the same as the server configured in Barman.

For example, assuming Barman is located on the host "`barmansrv`" under the "`barman`" user account, `repmgr.conf` should contain the following entries:

        barman_host='barman@barmansrv'
        barman_server='pg'

Here `pg` corresponds to a section in Barman's configuration file for a specific server backup configuration, which would look something like:

    [pg]
    description = "Main cluster"
    ...

More details on Barman configuration can be found in the Barman documentation's configuration section.

To use a non-default Barman configuration file on the Barman server, specify this in `repmgr.conf` with `barman_config`:

          barman_config='/path/to/barman.conf'

We also recommend configuring the `restore_command` setting in `repmgr.conf` to use the `barman-wal-restore` script (see section cloning-from-barman-restore-command below).

If you have a non-default SSH configuration on the Barman server, e.g. using a port other than 22, then you can set those parameters in a dedicated Host section in `~/.ssh/config` corresponding to the value of `barman_host` in `repmgr.conf`. See the `Host` section in `man 5 ssh_config` for more details.

If you wish to place WAL files in a location outside the main PostgreSQL data directory, set `--waldir` (PostgreSQL 9.6 and earlier: `--xlogdir`) in `pg_basebackup_options` to the target directory (must be an absolute filepath). &repmgr; will create and symlink to this directory in exactly the same way pg_basebackup would.

It's now possible to clone a standby from Barman, e.g.:

        $ repmgr -f /etc/repmgr.conf -h node1 -U repmgr -d repmgr standby clone
        NOTICE: destination directory "/var/lib/postgresql/data" provided
        INFO: connecting to Barman server to verify backup for "test_cluster"
        INFO: checking and correcting permissions on existing directory "/var/lib/postgresql/data"
        INFO: creating directory "/var/lib/postgresql/data/repmgr"...
        INFO: connecting to Barman server to fetch server parameters
        INFO: connecting to source node
        DETAIL: current installation size is 30 MB
        NOTICE: retrieving backup from Barman...
        (...)
        NOTICE: standby clone (from Barman) complete
        NOTICE: you can now start your PostgreSQL server
        HINT: for example: pg_ctl -D /var/lib/postgresql/data start

Barman support is automatically enabled if `barman_server` is set. Normally it is good practice to use Barman, for instance when fetching a base backup while cloning a standby; in any case, Barman mode can be disabled using the `--without-barman` command line option.

### Using Barman as a WAL file source

As a fallback in case streaming replication is interrupted, PostgreSQL can optionally retrieve WAL files from an archive, such as that provided by Barman. This is done by setting `restore_command` in the replication configuration to a valid shell command which can retrieve a specified WAL file from the archive.

`barman-wal-restore` is a Python script provided as part of the `barman-cli` package (Barman 2.0 ~ 2.7) or as part of the core Barman distribution (Barman 2.8 and later).

To use `barman-wal-restore` with &repmgr;, assuming Barman is located on the host "`barmansrv`" under the "`barman`" user account, and that `barman-wal-restore` is located as an executable at `/usr/bin/barman-wal-restore`, `repmgr.conf` should include the following lines:

        barman_host='barman@barmansrv'
        barman_server='pg'
        restore_command='/usr/bin/barman-wal-restore barmansrv pg %f %p'

`barman-wal-restore` supports command line switches to control parallelism (`--parallel=N`) and compression (`--bzip2`, `--gzip`).

### Using Barman through its API (pg-backup-api)

You can find information on how to install and setup pg-backup-api in the pg-backup-api documentation.

This mode (`pg-backupapi`) was introduced in v5.4.0 as a way to further integrate with Barman letting Barman handle the restore. This also reduces the ssh keys that need to share between the backup and postgres nodes. As long as you have access to the API service by HTTP calls, you could perform recoveries right away. You just need to instruct Barman through the API which backup you need and on which node the backup needs to to be restored on.

In order to enable `pg_backupapi mode` support for `repmgr standby clone`, you need the following lines in repmgr.conf:

- pg_backupapi_host: Where pg-backup-api is hosted

- pg_backupapi_node_name: Name of the server as understood by Barman

- pg_backupapi_remote_ssh_command: How Barman will be connecting as to the node

- pg_backupapi_backup_id: ID of the existing backup you need to restore

This is an example of how repmgr.conf would look like:

            pg_backupapi_host = '192.168.122.154'
            pg_backupapi_node_name = 'burrito'
            pg_backupapi_remote_ssh_command = 'ssh john_doe@192.168.122.1'
            pg_backupapi_backup_id = '20230223T093201'

`pg_backupapi_host` is the variable name that enables this mode, and when you set it, all the rest of the above variables are required. Also, remember that this service is just an interface between Barman and repmgr, hence if something fails during a recovery, you should check Barman's logs upon why the process couldn't finish properly.

Despite in Barman you can define shortcuts like "lastest" or "oldest", they are not supported for the time being in pg-backup-api. These shortcuts will be supported in a future release.

This is a real example of repmgr's output cloning with the API. Note that during this operation, we stopped the service for a little while and repmgr had to retry but that doesn't affect the final outcome. The primary is listening on localhost's port 6001:

    $ repmgr -f ~/nodes/node_3/repmgr.conf standby clone -U repmgr -p 6001 -h localhost
    NOTICE: destination directory "/home/mario/nodes/node_3/data" provided
    INFO: Attempting to use `pg_backupapi` new restore mode
    INFO: connecting to source node
    DETAIL: connection string is: user=repmgr port=6001 host=localhost
    DETAIL: current installation size is 8541 MB
    DEBUG: 1 node records returned by source node
    DEBUG: connecting to: "user=repmgr dbname=repmgr host=localhost port=6001 connect_timeout=2 fallback_application_name=repmgr options=-csearch_path="
    DEBUG: upstream_node_id determined as 1
    INFO: Attempting to use `pg_backupapi` new restore mode
    INFO: replication slot usage not requested;  no replication slot will be set up for this standby
    NOTICE: starting backup (using pg_backupapi)...
    INFO: Success creating the task: operation id '20230309T150647'
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    Incorrect reply received for that operation ID.
    INFO: Retrying...
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status IN_PROGRESS
    INFO: status DONE
    NOTICE: standby clone (from pg_backupapi) complete
    NOTICE: you can now start your PostgreSQL server
    HINT: for example: pg_ctl -D /home/mario/nodes/node_3/data start
    HINT: after starting the server, you need to register this standby with "repmgr standby register"

## Cloning and replication slots

Replication slots were introduced with PostgreSQL 9.4 and are designed to ensure that any standby connected to the primary using a replication slot will always be able to retrieve the required WAL files. This removes the need to manually manage WAL file retention by estimating the number of WAL files that need to be maintained on the primary using `wal_keep_segments` (PostgreSQL 13 and later: `wal_keep_size`). Do however be aware that if a standby is disconnected, WAL will continue to accumulate on the primary until either the standby reconnects or the replication slot is dropped.

To enable &repmgr; to use replication slots, set the boolean parameter `use_replication_slots` in `repmgr.conf`:

           use_replication_slots=true

Replication slots must be enabled in `postgresql.conf` by setting the parameter `max_replication_slots` to at least the number of expected standbys (changes to this parameter require a server restart).

When cloning a standby, &repmgr; will automatically generate an appropriate slot name, which is stored in the `repmgr.nodes` table, and create the slot on the upstream node:

        repmgr=# SELECT node_id, upstream_node_id, active, node_name, type, priority, slot_name
                   FROM repmgr.nodes ORDER BY node_id;
         node_id | upstream_node_id | active | node_name |  type   | priority |   slot_name
        ---------+------------------+--------+-----------+---------+----------+---------------
               1 |                  | t      | node1     | primary |      100 | repmgr_slot_1
               2 |                1 | t      | node2     | standby |      100 | repmgr_slot_2
               3 |                1 | t      | node3     | standby |      100 | repmgr_slot_3
         (3 rows)

        repmgr=# SELECT slot_name, slot_type, active, active_pid FROM pg_replication_slots ;
           slot_name   | slot_type | active | active_pid
        ---------------+-----------+--------+------------
         repmgr_slot_2 | physical  | t      |      23658
         repmgr_slot_3 | physical  | t      |      23687
        (2 rows)

Note that a slot name will be created by default for the primary but not actually used unless the primary is converted to a standby using e.g. `repmgr standby switchover`.

Further information on replication slots in the PostgreSQL documentation: https://www.postgresql.org/docs/current/interactive/warm-standby.html#STREAMING-REPLICATION-SLOTS

While replication slots can be useful for streaming replication, it's recommended to monitor for inactive slots as these will cause WAL files to build up indefinitely, possibly leading to server failure.

As an alternative we recommend using EDB's Barman, which offloads WAL management to a separate server, removing the requirement to use a replication slot for each individual standby to reserve WAL. See section cloning-from-barman for more details on using &repmgr; together with Barman.

## Cloning and cascading replication

Cascading replication, introduced with PostgreSQL 9.2, enables a standby server to replicate from another standby server rather than directly from the primary, meaning replication changes "cascade" down through a hierarchy of servers. This can be used to reduce load on the primary and minimize bandwidth usage between sites. For more details, see the PostgreSQL cascading replication documentation.

&repmgr; supports cascading replication. When cloning a standby, set the command-line parameter `--upstream-node-id` to the `node_id` of the server the standby should connect to, and &repmgr; will create a replication configuration file to point to it. Note that if `--upstream-node-id` is not explicitly provided, &repmgr; will set the standby's replication configuration to point to the primary node.

To demonstrate cascading replication, first ensure you have a primary and standby set up as shown in the quickstart. Then create an additional standby server with `repmgr.conf` looking like this:

        node_id=3
        node_name=node3
        conninfo='host=node3 user=repmgr dbname=repmgr'
        data_directory='/var/lib/postgresql/data'

Clone this standby (using the connection parameters for the existing standby), ensuring `--upstream-node-id` is provide with the `node_id` of the previously created standby (if following the example, this will be `2`):

        $ repmgr -h node2 -U repmgr -d repmgr -f /etc/repmgr.conf standby clone --upstream-node-id=2
        NOTICE: using configuration file "/etc/repmgr.conf"
        NOTICE: destination directory "/var/lib/postgresql/data" provided
        INFO: connecting to upstream node
        INFO: connected to source node, checking its state
        NOTICE: checking for available walsenders on upstream node (2 required)
        INFO: sufficient walsenders available on upstream node (2 required)
        INFO: successfully connected to source node
        DETAIL: current installation size is 29 MB
        INFO: creating directory "/var/lib/postgresql/data"...
        NOTICE: starting backup (using pg_basebackup)...
        HINT: this may take some time; consider using the -c/--fast-checkpoint option
        INFO: executing: 'pg_basebackup -l "repmgr base backup" -D /var/lib/postgresql/data -h node2 -U repmgr -X stream '
        NOTICE: standby clone (using pg_basebackup) complete
        NOTICE: you can now start your PostgreSQL server
        HINT: for example: pg_ctl -D /var/lib/postgresql/data start

then register it (note that `--upstream-node-id` must be provided here too):

         $ repmgr -f /etc/repmgr.conf standby register --upstream-node-id=2
         NOTICE: standby node "node2" (ID: 2) successfully registered

After starting the standby, the cluster will look like this, showing that `node3` is attached to `node2`, not the primary (`node1`).

        $ repmgr -f /etc/repmgr.conf cluster show
         ID | Name  | Role    | Status    | Upstream | Location | Connection string
        ----+-------+---------+-----------+----------+----------+--------------------------------------
         1  | node1 | primary | * running |          | default  | host=node1 dbname=repmgr user=repmgr
         2  | node2 | standby |   running | node1    | default  | host=node2 dbname=repmgr user=repmgr
         3  | node3 | standby |   running | node2    | default  | host=node3 dbname=repmgr user=repmgr

Under some circumstances when setting up a cascading replication cluster, you may wish to clone a downstream standby whose upstream node does not yet exist. In this case you can clone from the primary (or another upstream node); provide the parameter `--upstream-conninfo` to explicitly set the upstream's `primary_conninfo` string in the replication configuration.

## Advanced cloning options

### pg_basebackup options when cloning a standby

As &repmgr; uses `pg_basebackup` to clone a standby, it's possible to provide additional parameters for `pg_basebackup` to customise the cloning process.

By default, `pg_basebackup` performs a checkpoint before beginning the backup process. However, a normal checkpoint may take some time to complete; a fast checkpoint can be forced with `repmgr standby clone`'s `-c/--fast-checkpoint` option. Note that this may impact performance of the server being cloned from (typically the primary) so should be used with care.

If Barman is set up for the cluster, it's possible to clone the standby directly from Barman, without any impact on the server the standby is being cloned from. For more details see cloning-from-barman.

Other options can be passed to `pg_basebackup` by including them in the `repmgr.conf` setting `pg_basebackup_options`.

Not that by default, &repmgr; executes `pg_basebackup` with `-X/--wal-method` (PostgreSQL 9.6 and earlier: `-X/--xlog-method`) set to `stream`. From PostgreSQL 9.6, if replication slots are in use, it will also create a replication slot before running the base backup, and execute `pg_basebackup` with the `-S/--slot` option set to the name of the previously created replication slot.

These parameters can set by the user in `pg_basebackup_options`, in which case they will override the &repmgr; default values. However normally there's no reason to do this.

If using a separate directory to store WAL files, provide the option `--waldir` (`--xlogdir` in PostgreSQL 9.6 and earlier) with the absolute path to the WAL directory. Any WALs generated during the cloning process will be copied here, and a symlink will automatically be created from the main data directory.

The `--waldir` (`--xlogdir`) option, if present in `pg_basebackup_options`, will be honoured by &repmgr; when cloning from Barman (&repmgr; 5.2 and later).

See the PostgreSQL pg_basebackup documentation for more details of available options.

### Managing passwords

If replication connections to a standby's upstream server are password-protected, the standby must be able to provide the password so it can begin streaming replication.

The recommended way to do this is to store the password in the `postgres` system user's `~/.pgpass` file. For more information on using the password file, see the documentation section configuration-password-file.

If using a `pgpass` file, an entry for the replication user (by default the user who connects to the `repmgr` database) must be provided, with database name set to `replication`, e.g.:

              node1:5432:replication:repmgr:12345

If, for whatever reason, you wish to include the password in the replication configuration file, set `use_primary_conninfo_password` to `true` in `repmgr.conf`. This will read a password set in `PGPASSWORD` (but not `~/.pgpass`) and place it into the `primary_conninfo` string in the replication configuration. Note that `PGPASSWORD` will need to be set during any action which causes the replication configuration file to be rewritten, e.g. repmgr-standby-follow.

### Separate replication user

In some circumstances it might be desirable to create a dedicated replication-only user (in addition to the user who manages the &repmgr; metadata). In this case, the replication user should be set in `repmgr.conf` via the parameter `replication_user`; &repmgr; will use this value when making replication connections and generating the replication configuration. This value will also be stored in the parameter `repmgr.nodes` table for each node; it no longer needs to be explicitly specified when cloning a node or executing repmgr-standby-follow.

### Tablespace mapping

&repmgr; provides a `tablespace_mapping` configuration file option, which will makes it possible to map the tablespace on the source node to a different location on the local node.

To use this, add `tablespace_mapping` to `repmgr.conf` like this:

      tablespace_mapping='/var/lib/pgsql/tblspc1=/data/pgsql/tblspc1'

where the left-hand value represents the tablespace on the source node, and the right-hand value represents the tablespace on the standby to be cloned.

This parameter can be provided multiple times.