29.2. System Requirements

Processor Architectures

One or more processors implementing the Intel 64 instruction set are required: that is, a processor of the AMD64 or Intel 64 architecture.

RAM

Each VDO volume has two distinct memory requirements:
  • The VDO module requires 370 MB plus an additional 268 MB per each 1 TB of physical storage managed.
  • The Universal Deduplication Service (UDS) index requires a minimum of 250 MB of DRAM, which is also the default amount that deduplication uses. For details on the memory usage of UDS, see Section 29.2.1, “UDS Index Memory Requirements”.

Storage

A single VDO volume can be configured to use up to 256 TB of physical storage. See Section 29.2.2, “VDO Storage Requirements” for the calculations to determine the usable size of a VDO-managed volume from the physical size of the storage pool the VDO is given.

Additional System Software

VDO depends on the following software:
  • LVM
  • Python 2.7
The yum package manager will install all necessary software dependencies automatically.

Placement of VDO in the Storage Stack

As a general rule, you should place certain storage layers under VDO and others on top of VDO:
  • Under VDO: DM-Multipath, DM-Crypt, and software RAID (LVM or mdraid).
  • On top of VDO: LVM cache, LVM Logical Volumes, LVM snapshots, and LVM Thin Provisioning.
The following configurations are not supported:
  • VDO on top of VDO volumes: storage → VDO → LVM → VDO
  • VDO on top of LVM Snapshots
  • VDO on top of LVM Cache
  • VDO on top of the loopback device
  • VDO on top of LVM Thin Provisioning
  • Encrypted volumes on top of VDO: storage → VDO → DM-Crypt
  • Partitions on a VDO volume: fdisk, parted, and similar partitions
  • RAID (LVM, MD, or any other type) on top of a VDO volume

Important

VDO supports two write modes: sync and async. When VDO is in sync mode, writes to the VDO device are acknowledged when the underlying storage has written the data permanently. When VDO is in async mode, writes are acknowledged before being written to persistent storage.
It is critical to set the VDO write policy to match the behavior of the underlying storage. By default, VDO write policy is set to the auto option, which selects the appropriate policy automatically.

29.2.1. UDS Index Memory Requirements

The UDS index consists of two parts:
  • A compact representation is used in memory that contains at most one entry per unique block.
  • An on-disk component which records the associated block names presented to the index as they occur, in order.
UDS uses an average of 4 bytes per entry in memory (including cache).
The on-disk component maintains a bounded history of data passed to UDS. UDS provides deduplication advice for data that falls within this deduplication window, containing the names of the most recently seen blocks. The deduplication window allows UDS to index data as efficiently as possible while limiting the amount of memory required to index large data repositories. Despite the bounded nature of the deduplication window, most datasets which have high levels of deduplication also exhibit a high degree of temporal locality — in other words, most deduplication occurs among sets of blocks that were written at about the same time. Furthermore, in general, data being written is more likely to duplicate data that was recently written than data that was written a long time ago. Therefore, for a given workload over a given time interval, deduplication rates will often be the same whether UDS indexes only the most recent data or all the data.
Because duplicate data tends to exhibit temporal locality, it is rarely necessary to index every block in the storage system. Were this not so, the cost of index memory would outstrip the savings of reduced storage costs from deduplication. Index size requirements are more closely related to the rate of data ingestion. For example, consider a storage system with 100 TB of total capacity but with an ingestion rate of 1 TB per week. With a deduplication window of 4 TB, UDS can detect most redundancy among the data written within the last month.
UDS's Sparse Indexing feature (the recommended mode for VDO) further exploits temporal locality by attempting to retain only the most relevant index entries in memory. UDS can maintain a deduplication window that is ten times larger while using the same amount of memory. While the sparse index provides the greatest coverage, the dense index provides more advice. For most workloads, given the same amount of memory, the difference in deduplication rates between dense and sparse indexes is negligible.
The memory required for the index is determined by the desired size of the deduplication window:
  • For a dense index, UDS will provide a deduplication window of 1 TB per 1 GB of RAM. A 1 GB index is generally sufficient for storage systems of up to 4 TB.
  • For a sparse index, UDS will provide a deduplication window of 10 TB per 1 GB of RAM. A 1 GB sparse index is generally sufficient for up to 40 TB of physical storage.
For concrete examples of UDS Index memory requirements, see Section 29.2.3, “Examples of VDO System Requirements by Physical Volume Size”

29.2.2. VDO Storage Requirements

VDO requires storage both for VDO metadata and for the actual UDS deduplication index:
  • VDO writes two types of metadata to its underlying physical storage:
    • The first type scales with the physical size of the VDO volume and uses approximately 1 MB for each 4 GB of physical storage plus an additional 1 MB per slab.
    • The second type scales with the logical size of the VDO volume and consumes approximately 1.25 MB for each 1 GB of logical storage, rounded up to the nearest slab.
    See Section 29.1.3, “VDO Volume” for a description of slabs.
  • The UDS index is stored within the VDO volume group and is managed by the associated VDO instance. The amount of storage required depends on the type of index and the amount of RAM allocated to the index. For each 1 GB of RAM, a dense UDS index will use 17 GB of storage, and a sparse UDS index will use 170 GB of storage.
For concrete examples of VDO storage requirements, see Section 29.2.3, “Examples of VDO System Requirements by Physical Volume Size”

29.2.3. Examples of VDO System Requirements by Physical Volume Size

The following tables provide approximate system requirements of VDO based on the size of the underlying physical volume. Each table lists requirements appropriate to the intended deployment, such as primary storage or backup storage.
The exact numbers depend on your configuration of the VDO volume.

Primary Storage Deployment

In the primary storage case, the UDS index is between 0.01% to 25% the size of the physical volume.

Table 29.2. VDO Storage and Memory Requirements for Primary Storage

Physical Volume Size10 GB – 1–TB2–10 TB11–50 TB51–100 TB101–256 TB
RAM Usage250 MB
Dense: 1 GB
Sparse: 250 MB
2 GB3 GB12 GB
Disk Usage2.5 GB
Dense: 10 GB
Sparse: 22 GB
170 GB255 GB1020 GB
Index TypeDenseDense or SparseSparseSparseSparse

Backup Storage Deployment

In the backup storage case, the UDS index covers the size of the backup set but is not bigger than the physical volume. If you expect the backup set or the physical size to grow in the future, factor this into the index size.

Table 29.3. VDO Storage and Memory Requirements for Backup Storage

Physical Volume Size10 GB – 1 TB2–10 TB11–50 TB51–100 TB101–256 TB
RAM Usage250 MB2 GB10 GB20 GB26 GB
Disk Usage2.5 GB170 GB850 GB1700 GB3400 GB
Index TypeDenseSparseSparseSparseSparse