cpio maximum file size limit

Latest response

I'm currently writing a backup utility based on cpio (main reason for using cpio is the flexibility of selecting files using find and then piping the list into cpio.) The system is RH6 (64-bit) and the cpio is 2.10-9 (I appreciate that this is downlevel and if cpio has been changed recently to fix this problem, I'll be happy to try a newer version.)

Several of the individual files in the tree being backed up are >4G and growing steadily. For this reason, I started to get concerned about the fact that cpio can supposedly only back up files which are <=8G in size. So I decided to run a few experiments, and the results so far are interesting.

I decided to start on the big side, so I created a test file which was 40G in size. I then tried to back it up using cpio (using the ustar format). It completed successfully, and the resulting cpio archive was the sort of size one would expect. I then tried to restore the file from the cpio archive. Again, it completed successfully (or at least it gave an exit code of 0) - but the restored file was 0 bytes.

I then tried backing up and restoring the same file using tar - this time, both backup and restore completed successfully.

I'll try some smaller files with cpio and see if I can establish empirically what is the maximum size which can be archived and restored successfully.

I would have expected that cpio would have complained that it couldn't archive the 40G file in the first place. In my opinion - given that this utility is normally used for unattended backups - the fact that it didn't complain is a disaster waiting to happen.

Responses

I personally would be using tar for this. The issues with file size limits are well documented for cpio (from the man page):

Use archive format format. The valid formats are listed below with file size limits for individual files in parentheses; the same names are also recognized in all-caps. The default in copy-in mode is to automatically detect the archive format, and in copy-out mode is ‘bin’.

‘bin’
    The obsolete binary format. (2147483647 bytes)
‘odc’
    The old (POSIX.1) portable format. (8589934591 bytes)
‘newc’
    The new (SVR4) portable format, which supports file systems having more than 65536 i-nodes. (4294967295 bytes)
‘crc’
    The new (SVR4) portable format with a checksum added.
‘tar’
    The old tar format. (8589934591 bytes)
‘ustar’
    The POSIX.1 tar format. Also recognizes GNU tar archives, which are similar but not identical. (8589934591 bytes)
‘hpbin’
    The obsolete binary format used by HPUX's cpio (which stores device files differently).
‘hpodc’
    The portable format used by HPUX's cpio (which stores device files differently).