lrzip-next(1)

NAME
lrzip-next - a large-file compression program

SYNOPSIS

lrzip-next | lrzn [OPTIONS] <file>
lrzip-next | lrzn -d [OPTIONS] <file>
lrznunzip [OPTIONS] <file>
lrzncat [OPTIONS] <file>
lrzntar [lrzip-next options] <directory>
lrzntar -d [lrzip-next options] <directory>
lrznuntar [lrzip-next options] <directory>
LRZIP=NOCONFIG [lrzip-next|lrunzip] [OPTIONS] <file>

DESCRIPTION

LRZIP-NEXT is a file compression program designed to do particularly well on very large files containing long distance redundancy. lrztar is a wrapper for LRZIP-NEXT to simplify compression and decompression of directories.

OPTIONS SUMMARY
Here is a summary of the options to lrzip-next.
Compression Options:
 --lzma                  lzma compression (default)
 -b, --bzip2             bzip2 compression
 -B, --bzip3             bzip3 compression
 -g, --gzip              gzip compression using zlib
 -l, --lzo               lzo compression (ultra fast)
 -n, --no-compress       no backend compression - prepare for other compressor
 -z, --zpaq              zpaq compression (best, extreme compression, extremely slow)
 -Z, --zstd              zstd compression
 -L, --level level       Set lzma/bzip2/gzip compression level (1-9, default 7)
 --dictsize = ds         Set lzma Dictionary Size for LZMA ds=0 to 40 expressed as 2<<11, 3 * 2<<10, 2<<12, 3 * 2<<11...2<<31-1
 --nobemt                Inhibit backend compressor using multiple threads
 --zpaqbs = bs           Set ZPAQ Block Size overriding defaults. 1-11, 2^zpaqbs * 1MB
 --bzip3bs = bs          Set BZIP3 Block Size. 0-8, 32MB - 511MB
 --zstd-level = level    Set zstd level (1-22). zstd strategy will be selected as per simple API spec.
Filtering Options (for all compression modes):
 --x86                   Use x86 filter
 --arm                   Use ARM filter
 --armt                  Use ARMT filter
 --arm64                 Use ARM64 filter
 --ppc                   Use PPC filter
 --sparc                 Use SPARC filter
 --ia64                  Use IA64 filter
 --riscv                 Use RISC-V filter
 --delta [=offset]       Use DELTA filter (offset 1 (default) - 15, then multiples of 16 to 256)
Additional Compression Options:
 -C, --comment [comment] Add a comment up to 64 chars
 -e, --encrypt[=password] Password protected SHAKE128/AES128 SHAKE256/AES256 encryption on compression
 -E, --emethod method    Select Encryption method, 1=AES 128, 2=AES 256 bit
 -D, --delete            delete existing files
 -f, --force             force overwrite of any existing files
 -k, --keep-broken       keep broken or damaged output files
 -o, --outfile filename  specify the output file name and/or path
 -O, --outdir directory  specify the output directory when -o is not used
 -S, --suffix suffix     specify compressed suffix (default '.lrz')
Low level Compression Options:
 --costfactor value      Force SCRYPT costfactor to 2^N where N is between 10 and 40 (1KB to 1TB)
 -m, --maxram size       Set maximum available ram in hundreds of MB
                         overrides detected amount of available ram
 -N, --nice-level value  Set nice value to value (default 19)
 -R, --rzip-level level  Set independent RZIP Compression Level (1-9) for pre-processing (default=compression level)
 -T, --threshold [limit] Disable LZ4 compressibility testing OR set limit to determine compressibiity (1-99)
 -U, --unlimited         Use unlimited window size beyond ramsize (potentially much slower)
 -w, --window size       maximum compression window in hundreds of MB
                         default chosen by heuristic dependent on ram and chosen compression
Decompression Options:
 -d, --decompress        decompress
 -e, -f, -o -O           Same as above. See Compression Options
 -t, --test              test compressed file integrity
 -c, --check             check integrity of file written on decompression
General options:
 -h, -?, --help          show help
 -H, --hash[=hash code]  Set hash to compute (default md5) and display hash integrity information
 -i, --info              show compressed file information
 -p, --threads value     Set processor count to override number of threads
 -q, --quiet             don't show compression progress
 -Q, --very-quiet        don't show any output
 -v[v], --verbose        Increase verbosity
 -V, --version           show version
LRZIP=NOCONFIG environment variable setting can be used to bypass lrzip.conf.
TMP environment variable will be used for storage of temporary files when needed.
TMPDIR may also be stored in lrzip.conf file.
If no filenames or "-" is specified, stdin/out will be used.

OPTIONS

Compression Options:

--lzma (default)
    Lzma compression. Default.

-b | --bzip
    Bzip2 compression. Uses bzip2 compression for the 2nd stage, much like the original rzip does.

-B | --bzip3
    BZIP3 compression. Uses BZIP3, a statistical compressor and spiritual successor to BZip2. Features higher compression ratios and good performance. Usually as fast as BZip2 and provides ratio comparable or better than LZMA.

-g | --gzip
    Gzip compression. Uses gzip compression for the 2nd stage. Uses libz compress and uncompress functions.

-l | --lzo
    LZO compression. If this option is set then lrzip-next will use the ultra fast lzo compression algorithm for the 2nd stage. This mode of compression gives bzip2 like compression at the speed it would normally take to simply copy the file, giving excellent compression/time value.

-n | --rzip
    No 2nd stage compression. If this option is set then lrzip-next will only perform the long distance redundancy 1st stage compression. While this does not compress any faster than LZO compression, it produces a smaller file that then responds better to further compression (by eg another application), also reducing the compression time substantially.

-z | --zpaq
    ZPAQ compression. Uses ZPAQ compression which is from the PAQ family of compressors known for having some of the highest compression ratios possible but at the cost of being extremely slow on both compress and decompress (4x slower than lzma which is the default).

-Z | --zstd
    ZSTD compression. Compression levels and strategy are set according to the lrzip-next compression level selected using the simple zstd API. (see --zstd-level option which bypasses this).
    LRZIP-NEXT Level  ZSTD Level    ZSTD Strategy
    ================  ==========    =============
             1             2           ZSTD_fast
             2             4          ZSTD_dfast
             3             5         ZSTD_greedy
             4             7           ZSTD_lazy
             5            12          ZSTD_lazy2
             6            15        ZSTD_btlazy2
             7            17          ZSTD_btopt
             8            18        ZSTD_btultra
             9            22       ZSTD_btultra2

-L 1..9
    Set the compression level from 1 to 9. The default is to use level 7, which gives good all round compression. The compression level is also strongly related to how much memory lrzip-next uses. See the -w option for details.

--dictsize=0..40 (LZMA only)
    Set Dictionary Size for LZMA from 2^12 (4KB) to 2^32-1 (4GB-1). Normally this option is not useful since lrzip-next will set and sometimes change the dictionary size depending on the compression level selected and usable ram available.
    If set, Dictionary sizes will be set as follows and will be fixed. If sufficient ram is not available, lrzip-next will fail. Values are per thread.
    0:  2 * 2^11, 1:  3 * 2^10; 4,096, 6,144
    2:  2 * 2^12, 3:  3 * 2^11; 8,192, 12,288
     ...
    38: 2 * 2^30, 39: 3 * 2^29; 2,147,483,648, 3,221,225,472
    40: 2 * 2^31 - 1; 4,294,967,295

--nobemt
    No Backend Multi Threading. This option will inhibit the backend from using more threads than requested by the `-p` option or default maximum threads.

--zpaqbs=1..11 (ZPAQ only)
    Set ZPAQ Block Size from 1 to 11, 2^zpaqbs * 1MB (2MB to 2GB). This parameter will be passed to the zpaq backend. Normally, lrzip-next will set and sometimes reduce the zpaq block size depending on compression level and usable ram available.
    If set, the zpaq block size will be set as follows and will be fixed. If sufficient ram is not available, lrzip-next will fail. Values are per thread.
    1  = 2^1*1MB  =    2MB
    2  = 2^2*1MB  =    4MB
    3  = 2^3*1MB  =    8MB
    4  = 2^4*1MB  =   16MB
    5  = 2^5*1MB  =   32MB
    6  = 2^6*1MB  =   64MB
    7  = 2^7*1MB  =  128MB
    8  = 2^8*1MB  =  256MB
    9  = 2^9*1MB  =  512MB
    10 = 2^10*1MB = 1024MB
    11 = 2^11*1MB = 2048MB

--bzip3bs=0..8 (BZIP3 only)
    Set BZIP3 Block Size from 0 to 8, 2^bzip3bs * 1MB (32MB to 511MB). This parameter will be passed to the bzip3 backend. Normally, lrzip-next will set and sometimes reduce the bzip3 block size depending on compression level and usable ram available.
    If set, the bzip3 block size will be set as follows and will be fixed. If sufficient ram is not available, lrzip-next will fail. Values are per thread.
    0 = 2^25   =  32MB
    1 = 3*2^24 =  48MB
    2 = 2^26   =  64MB
    3 = 3*2^25 =  96MB
    4 = 2^27   = 128MB
    5 = 3*2^26 = 192MB
    6 = 2^28   = 256MB

--zstd-levels=1..22 (ZSTD only)
    Manually set zstd level which will also set zstd strategy for simple API compression. This bypasses auto setting of zstd levels based on lrzip-next compression levels. zstd strategies will be set according to level as defined in zstd clevels.h.
    ZSTD Level    ZSTD Strategy
    ==========    =============
       1-2           ZSTD_fast
       3-4          ZSTD_dfast
         5         ZSTD_greedy
       6-7           ZSTD_lazy
      8-12          ZSTD_lazy2
     13-15        ZSTD_btlazy2
     16-17          ZSTD_btopt
        18        ZSTD_btultra
     19-22       ZSTD_btultra2

Filtering Options (for all compression modes)

--x86
    Unconditionally use x86 filter prior to compression. Works for all compression modes.

--arm
    Unconditionally use ARM filter prior to compression. Works for all compression modes.

--armt
    Unconditionally use ARMT filter prior to compression. Works for all compression modes.

--arm64
    Unconditionally use ARM64 filter prior to compression. Works for all compression modes.

--ppc
    Unconditionally use PPC filter prior to compression. Works for all compression modes.

--sparc
    Unconditionally use SPARC filter prior to compression. Works for all compression modes.

--ia64
    Unconditionally use IA64 filter prior to compression. Works for all compression modes.

--riscv
    Unconditionally use RISC-V filter prior to compression. Works for all compression modes.

--delta [1..31]
    Unconditionally use DELTA filter prior to compression. Works for all compression modes. Delta offset 1 default. Delta offset is set as 1-16, then 32..256 in multiples of 16. e.g. An offset of 17 would be 32, 18:48, 19:64...31:256.

Additional Compression Options:

-C | --comment [comment]
    Add a comment for the archive up to 64 characters.

-e | --encrypt [password]
    Encrypt. This option enables high grade password encryption using a combination of multiply sha512 hashed password, random salt and aes128 CBC encryption. Passwords up to 500 characters long are supported, and the encryption mechanism used virtually guarantees that the same file created with the same password will never be the same. Furthermore, the password hashing is increased according to the date the file is encrypted, increasing the number of CPU cycles required for each password attempt in accordance with Moore's law, thus making the difficulty of attempting brute force attacks proportional to the power of modern computers.

    The optional password may be inserted on the command line, however there are obvious security risks to doing so, such as the cleartext password will be visible on a screen or persistent in a screen buffer; the cleartext password will be visible in a process listing, so take care. If the short option -e is used, there must be no space between the option and the password. e.g. -epassphrase.

    These options may be used for Encryption, Decryption, or Test under the following conditions.
    If output is to STDOUT, then the option cannot be used during compression.
    If input is from STDIN, then the password must be on the command line.
    If lrzip-next is embedded in a tar -I lrzip-next [options] command, then
    this option cannot be used.

-E | --emethod [encryption method]
    Select encryption method to use:
    1=AES 128 bit encryption
    2=AES 256 bit encryption
    AES 128 uses SHAKE128 hashing for key generation
    AES 256 uses SHAKE256 hashing

-D | --delete
    If this option is specified then lrzip-next will delete the source file after successful compression or decompression. When this option is not specified then the source files are not deleted.

-f | --force
    If this option is not specified (Default) then lrzip-next will not overwrite any existing files. If you set this option then rzip will silently overwrite any files as needed.

-k | --keep-broken
    This option will keep broken or damaged files instead of deleting them. When compression or decompression is interrupted either by user or error, or a file decompressed fails an integrity check, it is normally deleted by LRZIP.

-o | --outfile filename
    Set the output file name. If this option is not set then the output file name is chosen based on the input name and the suffix. The -o option cannot be used if more than one file name is specified on the command line.

-O | --outdir output_directory
    Set the output directory for the default filename. This option cannot be combined with -o.

-S | --suffix  suffix (default .lrz)
    Set the compression suffix. The default is '.lrz'.

Low Level Compression Options:

--costfactor costfactor
    Specify the cost factor for SCRYPT key derivation. Accepted values are 10-40 expressed as a 2s exponent (1KB to 1TB). Default if option not used is the power of 2 / 1024 less than available RAM. Ex. If available RAM is 16GB (2^34), the power of 2 less than that is 8GB (2^33). 8GB / 1024 = 8MB (2^23). RAM requirements for costfactor is N * 128 * 8.

-m | --maxram maxram
    Specify the maximum system memory in 100MB blocks. Overrides detected ram. Ex. 40=4GB.

-N | --nice-val value
    The default nice value is 19. This option can be used to set the priority scheduling for the lrzip-next backup or decompression. Valid nice values are from -20 to 19. Note this does NOT speed up or slow down compression.

-R | --rzip-level level
    Specify the rzip pre-processing compression level. If not set, will default to compression level.

-T | --threshold
    Disables the LZ4 compressibility threshold testing when a slower compression back-end is used. LZ4 testing is normally performed for the slower back-end compression of LZMA and ZPAQ. The reasoning is that if it is completely incompressible by LZ4 then it will also be incompressible by them. Thus if a block fails to be compressed by the very fast LZ4, lrzip-next will not attempt to compress that block with the slower compressor, thereby saving time. If this option is enabled, it will bypass the LZ4 testing and attempt to compress each block regardless.

-T | --threshold limit
    If the value limit is used, it will test compressibility as a percentage of chunk size. Limiting chunck compressibility threshold can speed up compression. If, for example, LZ4_TEST determines a chunk is 99.99% of chunk size, it will pass the chunk to the compressor even though it is barely compressible. Setting limit to a lower value, 95% for example, the chunk will not be passed to the compressor unless it is at least 5% compression benefit as determined by LZ4_TEST. Any value from 1-99 is allowed.

-U | --unlimited
    Unlimited window size. If this option is set, and the file being compressed does not fit into the available ram, lrzip-next will use a moving second buffer as a "sliding mmap" which emulates having infinite ram. This will provide the most possible compression in the first rzip stage which can improve the compression of ultra large files when they're bigger than the available ram. However it runs progressively slower the larger the difference between ram and the file size, so is best reserved for when the smallest possible size is desired on a very large file, and the time taken is not important.

-w | --window size
    Set the maximum allowable compression window size to n in hundreds of megabytes. This is the amount of memory lrzip-next will search during its first stage of pre-compression and is the main thing that will determine how much benefit lrzip-next will provide over ordinary compression with the 2nd stage algorithm. If not set (recommended), the value chosen will be determined by an internal heuristic in lrzip-next which uses the most memory that is reasonable, without any hard upper limit. It is limited to 2GB on 32bit machines. lrzip-next will always reduce the window size to the biggest it can be without running out of memory.

Decompression Options:

-d | --decompress
    Decompress. If this option is not used then lrzip-next looks at the name used to launch the program. If it contains the string "lrunzip" then the -d option is automatically set. If it contains the string "lrzcat" then the -d -o - options are automatically set.

-e, -E, -f, -o, -O
    Same as above. See Compression Options.

-t | --test
    This tests the compressed file integrity. It does this by decompressing it to a temporary file and then deleting it.

-c | --check
    This option enables integrity checking of the file written to disk on decompression. All decompression is tested internally in lrzip-next with either crc32 or md5 hash checking depending on the version of the archive already. However the file written to disk may be corrupted for other reasons to do with other userspace problems such as faulty library versions, drivers, hardware failure and so on. Enabling this option will make lrzip-next perform an md5 hash check on the file that's written to disk. When the archive has the md5 value stored in it, it is compared to this. Otherwise it is compared to the value calculated during decompression. This offers an extra guarantee that the file written is the same as the original archived.

General Options:

-h | -? | --help
    Print an options summary page

-H | --hash [hash code]
    This shows the hash value calculated on compressing or decompressing an lrzip-next archive. By default all compression has the md5 value calculated and stored in all archives since version 0.560. On decompression, when a hash value has been found, it will be calculated and used for integrity checking. If the hash value is not stored in the archive, it will not be calculated unless explicitly specified with this option, or check integrity (see below) has been requested.
    Hash values can be 1-13 and are defined as follows:
     0 CRC         - 32 bit CRC
     1 MD5         - MD5 Checksum
     2 RIPEMD      - RACE Integrity Primitives Evaluation Message Digest
     3 SHA256      - Secure Hash Algorithm 2, 256 bit
     4 SHA384      - Secure Hash Algorithm 2, 384 bit
     5 SHA512      - Secure Hash Algorithm 2, 512 bit
     6 SHA3_256    - Secure Hash Algorithm 3, 256 bit
     7 SHA3_512    - Secure Hash Algorithm 3, 512 bit
     8 SHAKE128_16 - Extendable Output Function (XOF) 128 bit, 16 byte output
     9 SHAKE128_32 - Extendable Output Function (XOF) 128 bit, 32 byte output
    10 SHAKE128_64 - Extendable Output Function (XOF) 128 bit, 64 byte output
    11 SHAKE256_16 - Extendable Output Function (XOF) 256 bit, 16 byte output
    12 SHAKE256_32 - Extendable Output Function (XOF) 256 bit, 32 byte output
    13 SHAKE256_64 - Extendable Output Function (XOF) 256 bit, 64 byte output

-i | --info
    This shows information about a compressed file. It shows the compressed size, the decompressed size, the compression ratio, what compression was used and what hash checking will be used for internal integrity checking. Note that the compression mode is detected from the first block only and it will show no compression used if the first block was incompressible, even if later blocks were compressible. If verbose options -v or -vv are added, a breakdown of all the internal blocks and progressively more information pertaining to them will also be shown.

-p | --threads value
    Set the number of processor count to determine the number of threads to run. Normally lrzip-next will scale according to the number of CPUs it detects. Using this will override the value in case you wish to use less CPUs to either decrease the load on your machine, or to improve compression. Setting it to 1 will maximise compression but will not attempt to use more than one CPU.

-q | --quiet
    If this option is specified then lrzip-next will not show the percentage progress while compressing. Note that compression happens in bursts with lzma compression which is the default compression. This means that it will progress very rapidly for short periods and then stop for long periods.

-Q | --very-quiet
    Suppress ALL output, even summary.

-v[v] | --verbose
    Increases verbosity. -vv will print more messages than -v.

-V | --version
    Print the lrzip-next version number.

INSTALLATION
"make install" or just install lrzip-next somewhere in your search path.

COMPRESSION ALGORITHM
LRZIP operates in two stages. The first stage finds and encodes large chunks of duplicated data over potentially very long distances in the input file. The second stage is to use a compression algorithm to compress the output of the first stage. The compression algorithm can be chosen to be optimised for extreme size (zpaq), size (lzma - default or bzip3), speed (lzo), legacy (bzip2 or gzip) or can be omitted entirely doing only the first stage. A one stage only compressed file can almost always improve both the compression size and speed done by a subsequent compression program.

The key difference between lrzip-next and other well known compression algorithms is its ability to take advantage of very long distance redundancy. The well known deflate algorithm used in gzip uses a maximum history buffer of 32k. The block sorting algorithm used in bzip2 is limited to 900k of history. The history buffer in lrzip-next can be any size long, not even limited by available ram.

It is quite common these days to need to compress files that contain long distance redundancies. For example, when compressing a set of home directories several users might have copies of the same file, or of quite similar files. It is also common to have a single file that contains large duplicated chunks over long distances, such as pdf files containing repeated copies of the same image. Most compression programs won't be able to take advantage of this redundancy, and thus might achieve a much lower compression ratio than lrzip-next can achieve.

FILES

LRZIP recognises a configuration file that contains default settings. This configuration is searched for in the current directory, /etc/lrzip, and $HOME/.lrzip. The configuration filename must be lrzip.conf.

ENVIRONMENT
By default, lrzip-next will search for and use a configuration file, lrzip.conf. If the user wishes to bypass the file, a startup ENV variable may be set.
LRZIP = NOCONFIG [lrzip-next|lrunzip] [OPTIONS] <file>
which will force lrzip-next to ignore the configuration file.

HISTORY - Notes on rzip by Andrew Tridgell

The ideas behind rzip were first implemented in 1998 while I was working on rsync. That version was too slow to be practical, and was replaced by this version in 2003. LRZIP was created by the desire to have better compression and/or speed by Con Kolivas on blending the lzma and lzo compression algorithms with the rzip first stage, and extending the compression windows to scale with increasing ram sizes.

BUGS

Please report! If you wish to report a problem, or make a suggestion, then please open an Issue at https://github.com/pete4abw/lrzip-next/issues or email the maintainer pete@peterhyman.com.

SEE ALSO
lrzip.conf(5), lrznunzip(1), lrzncat(1), lrzntar(1), lrznuntar(1), bzip2(1), gzip(1), lzop(1), rzip(1), bzip3(1), zip(1), zstd(1)

AUTHOR and CREDITS

lrzip was conceived and written by Con Kolivas.
lrzip-next is maintained by Peter Hyman.
rzip was written by Andrew Tridgell.
lzma was written by Igor Pavlov.
lzo was written by Markus Oberhumer.
zpaq was written by Matt Mahoney.
bzip3 was written by Kamila Szewczyk.
zstd was written by Yann Collet.
lrzip-next is released under the GNU General Public License version 2. Please see the file COPYING for license details. This manpage was written by Peter Hyman (pete@peterhyman.com)