Description: To allow your LXD server to run a large number of instances, configure the following settings to avoid hitting server limits. The Value column contains the suggested value for each parameter./etc/s...

Hide navigation sidebar
Hide table of contents sidebar
Skip to content
Back to top
View this page
Toggle table of contents sidebar

# Server settings for a LXD production setup

To allow your LXD server to run a large number of instances, configure the following settings to avoid hitting server limits.

The `Value` column contains the suggested value for each parameter.

## `/etc/security/limits.conf`

Note

For users of the snap, those limits are automatically raised.

Domain | Type | Item | Value | Default | Description
--- | --- | --- | --- | --- | ---
`*` | soft | `nofile` | `1048576` | unset | Maximum number of open files
`*` | hard | `nofile` | `1048576` | unset | Maximum number of open files
`root` | soft | `nofile` | `1048576` | unset | Maximum number of open files
`root` | hard | `nofile` | `1048576` | unset | Maximum number of open files
`*` | soft | `memlock` | `unlimited` | unset | Maximum locked-in-memory address space (KB)
`*` | hard | `memlock` | `unlimited` | unset | Maximum locked-in-memory address space (KB)
`root` | soft | `memlock` | `unlimited` | unset | Maximum locked-in-memory address space (KB), only need with `bpf` syscall supervision
`root` | hard | `memlock` | `unlimited` | unset | Maximum locked-in-memory address space (KB), only need with `bpf` syscall supervision

## `/etc/sysctl.conf`

Note

Reboot the server after changing any of these parameters.

`fs.aio-max-nr`

Maximum number of concurrent asynchronous I/O operations

Key: | `fs.aio-max-nr`
Type: | integer
Default: | `65536`

Suggested value: `524288`

You might need to increase this limit further if you have a lot of workloads that use the AIO subsystem (for example, MySQL).

`fs.inotify.max_queued_events`

Upper limit on the number of events that can be queued

Key: | `fs.inotify.max_queued_events`
Type: | integer
Default: | `16384`

Suggested value: `1048576`

This option specifies the maximum number of events that can be queued to the corresponding `inotify` instance (see `inotify` for more information).

`fs.inotify.max_user_instances`

Upper limit on the number of `inotify` instances

Key: | `fs.inotify.max_user_instances`
Type: | integer
Default: | `128`

Suggested value: `1048576`

This option specifies the maximum number of `inotify` instances that can be created per real user ID (see `inotify` for more information).

`fs.inotify.max_user_watches`

Upper limit on the number of watches

Key: | `fs.inotify.max_user_watches`
Type: | integer
Default: | `8192`

Suggested value: `1048576`

This option specifies the maximum number of watches that can be created per real user ID (see `inotify` for more information).

`kernel.dmesg_restrict`

Whether to deny access to the messages in the kernel ring buffer

Key: | `kernel.dmesg_restrict`
Type: | integer
Default: | `0`

Suggested value: `1`

Set this option to `1` to deny container access to the messages in the kernel ring buffer. Note that setting this value to `1` will also deny access to non-root users on the host system.

`kernel.keys.maxbytes`

Maximum size of the key ring that non-root users can use

Key: | `kernel.keys.maxbytes`
Type: | integer
Default: | `20000`

Suggested value: `2000000`

`kernel.keys.maxkeys`

Maximum number of keys that a non-root user can use

Key: | `kernel.keys.maxkeys`
Type: | integer
Default: | `200`

Suggested value: `2000`

Set this option to a value that is higher than the number of instances.

`net.core.bpf_jit_limit`

Limit on the size of eBPF JIT allocations

Key: | `net.core.bpf_jit_limit`
Type: | integer
Default: | varies

Suggested value: `1000000000`

On kernels < 5.15 that are compiled with `CONFIG_BPF_JIT_ALWAYS_ON=y`, this value might limit the amount of instances that can be created.

`net.ipv4.neigh.default.gc_thresh3`

Maximum number of entries in the IPv4 ARP table

Key: | `net.ipv4.neigh.default.gc_thresh3`
Type: | integer
Default: | `1024`

Suggested value: `8192`

Increase this value if you plan to create over 1024 instances. Otherwise, you will get the error `neighbour: ndisc_cache: neighbor table overflow!` when the ARP table gets full and the instances cannot get a network configuration. See `ip-sysctl` for more information.

`net.ipv6.neigh.default.gc_thresh3`

Maximum number of entries in IPv6 ARP table

Key: | `net.ipv6.neigh.default.gc_thresh3`
Type: | integer
Default: | `1024`

Suggested value: `8192`

Increase this value if you plan to create over 1024 instances. Otherwise, you will get the error `neighbour: ndisc_cache: neighbor table overflow!` when the ARP table gets full and the instances cannot get a network configuration. See `ip-sysctl` for more information.

`vm.max_map_count`

Maximum number of memory map areas a process may have

Key: | `vm.max_map_count`
Type: | integer
Default: | `65530`

Suggested value: `262144`

Memory map areas are used as a side-effect of calling `malloc`, directly by `mmap` and `mprotect`, and also when loading shared libraries.

## Related topics

How-to guides:

- How to benchmark performance

- How to increase the network bandwidth

- How to monitor metrics

Explanation:

- Performance tuning