11/27/95, 4FAX 3753 About Sparse Files SPECIAL NOTICES Information in this document is correct to the best of our knowledge at the time of this writing. Please send feedback by fax to "AIXServ Information" at (512) 823-4009. Please use this information with care. IBM will not be responsible for damages of any kind resulting from its use. The use of this information is the sole responsibility of the customer and depends on the customer's ability to eval- uate and integrate this information into the customer's operational environment. ABOUT THIS DOCUMENT This document describes sparse files, exposure due to sparse files, and the effects of certain commands on sparse files. This document applies to AIX 3.2. OVERVIEW Many applications, particularly databases, maintain data in sparse files. A sparse file is a file with empty space, or gaps, left open for future addition of data. If the empty spaces are filled with the ASCII null character and the spaces are large enough, the file will be sparse, and disk blocks will not be allocated to it. This creates an exposure: a large file will be created, but the disk blocks will not be allocated. Then, as data is added to the file, the disk blocks will be allocated but there may not be enough free disk blocks in the file system. Then the file system will be full and writes to any file in the file system will fail. You can prevent these problems by either assuring that you have no sparse files on your system or by planning to have enough free space in the file system for the future allo- cation of the blocks. You also need to be aware of how you manipulate sparse or potentially sparse files because you can easily change them from sparse to not sparse or vice-versa. CREATING A SPARSE FILE An example sparse file can be created fairly easily. To do this, open the file, seek to a large address, and write some data. This can be demonstrated with the dd command, as follows: 1. First, create a regular file: date > notsparse ls -l About Sparse Files 1 11/27/95, 4FAX 3753 The output of the ls command will be similar to: total 8 -rw-r--r-- 1 root sys 29 Dec 21 08:12 notsparse 2. Use the fileplace command to see how many allocated and unallocated blocks are included in the file "notsparse". (Note: BOS Extensions 1/Extended Commands | [bosext1.extcmds.obj] must be installed to enable the | fileplace commands or perfagent.tools for 4.1.) fileplace notsparse | The output for 3.2 will look similar to: File: notsparse Size: 29 bytes Vol: /dev/lv03 (4096 byte blks) Logical blocks -------------- 00016 1 blk, 4 KB, 100.0% (Note: BOS Extensions 1 [bosext1.extcmds.obj] must be installed to enable the fileplace command.) | The output for 4.1 will look similiar to: | File: notsparse Size: 29 bytes Vol: /dev/lv03 | Blk Size: 4096 Frag size: 4096 Nfrags: 1 Compress: no | Logical Fragment | ---------------- | 00716 1 frags 4096 bytes, 100.0% | (Note: Performance Analysis and Control Commands | [perfagent.tools] must be installed to enable the | fileplace command for 4.1.) 3. The du command will also reflect how many 512-byte blocks a file occupies. du -rs * Example output: 8 notsparse 4. Now create a sparse file using the regular file "notsparse" as input: touch sparse.1 dd if=notsparse of=sparse.1 seek=100 Example output: dd: 0+1 records in. dd: 0+1 records out. About Sparse Files 2 11/27/95, 4FAX 3753 The dd command takes the data from the regular file and places it 100 512-byte blocks into the "sparse.1" file. Note that nothing is written to the initial 99 512-byte blocks. The following steps show the characteristics of the resulting file. 5. The ls command reports the distance from block zero to the last block in the file: ls -l Example output: total 16 -rw-r--r-- 1 root sys 29 Dec 21 08:12 notsparse -rw-r--r-- 1 root sys 51229 Dec 21 08:13 sparse.1 6. The fileplace command tells the story accurately -- there are 12 unallocated 4K blocks and one allocated 4K block in the file: fileplace sparse.1 | Example output for 3.2: File: sparse.1 Size: 51229 bytes Vol: /dev/lv03 (4096 byte blks) Logical blocks -------------- unallocated 12 blks, 48 KB, 92.3% 00018 1 blk, 4 KB, 7.7% | Example output for 4.1 | File: sparse.1 Size: 51229 bytes Vol: /dev/lv03 | Blk Size: 4096 Frag Size: 4096 Nfrags: 1 Compress: no | Logical Fragment | ---------------- | unallocated 12 frags 49152 Bytes, 0.0% | 0000769 1 frags 4096 Bytes, 100.0% 7. The du command reports the number of allocated blocks the file takes: du -rs * Example output: 8 notsparse 8 sparse.1 THE EFFECT OF CERTAIN COMMANDS ON SPARSE FILES About Sparse Files 3 11/27/95, 4FAX 3753 backup/restore (by name and inode) The restore command aggressively preserves sparseness. In fact, the restore command will unallocate any blocks filled with zeroes, thus making a file sparse. Currently there is a defect in the restbynode command where inode backups do not preserve sparseness on the restore. The fix is ix50200, while the workaround calls for the use of a version of restbyinode with the sccs ID for tape.c at 1.25.1.1. | Note: Users with 4.1.3 or later will not need this fix, | however, users with earlier versions of 4.1 will need | ix50815. To order the fix via voice, call 1-512-823-9468 or 1-817-961-6077. To request a fix via fax send a fax message to 1-800-225-5249 referencing the ix number above. Finally, Internet users can obtained electronic download instructions from IBM 4FAX (1-800-426-4329). Request Index number 1228 1228 "FixDist - Electronic Fixes for AIX on the Internet". cp The cp command does not preserve the sparseness of a file. cpio If you create a backup using the cpio command on sparse files, you will need to use the pax command to restore that data. Using the cpio command to restore the data will not preserve sparseness. NOTE: The pax command can only read cpio backups if the cpio "c" flag was used when the backup was created. If the "c" flag was not used, the pax command will not be able to restore the data. | Note: This pax limitation is no longer valid for 4.1. WORKAROUND IF "C" FLAG NOT USED 1. Restore the file(s) to disk. 2. Back up the file(s) with one of the following: o backup (by name or inode) o cpio with the "c" flag o tar 3. Verify that you have a good backup by listing a table of contents from your backup: o If tar or cpio was used: pax -vf /dev/rmt# o If backup by name was used: About Sparse Files 4 11/27/95, 4FAX 3753 restore -Tvf /dev/rmt0 o If backup by inode was used: restore -tvf /dev/rmt0 4. Remove the file(s) you restored. 5. Now, restore the file(s) from the newer backup (that you just made). NOTE: If you used cpio or tar, restore with pax. dd Using the dd command on the file itself does not preserve sparseness. However, using dd on the file system device does preserve the state of the individual files. Example: Backing up a logical volume: dd if=/dev/datalv of=/dev/rmt0 ibs=4096 obs=1024 conv=sync mksysb The mksysb command in AIX 3.2 uses tar to back up the mounted JFS file systems in the rootvg. See the section on tar. Mksysb uses the pax command to restore the system. | See the section on pax. The mksysb command in AIX 4.1 uses | backup/restore. See the section on backup/restore. pax NOTE: The pax command can read tar archives and can read cpio archives if the "c" flag was used. The pax command aggressively preserves sparseness. In fact, the pax command will unallocate any blocks filled with zeroes, thus making a file sparse. sysback Sysback will use either backup by name or inode to backup the data on the system. See the section on backup. tar If you create a backup using the tar command on sparse files, you will have to use the pax command to restore that data. Using the tar command to restore the data will not preserve sparseness. About Sparse Files 5 11/27/95, 4FAX 3753 READER'S COMMENTS Please fax this form to (512) 823-4009, attention "AIXServ Informa- tion". You may also e-mail comments to: elizabet@austin.ibm.com. These comments should include the same customer information requested below. Use this form to tell us what you think about this document. If you have found errors in it, or if you want to express your opinion about it (such as organization, subject matter, appearance) or make sug- gestions for improvement, this is the form to use. If you need technical assistance, contact your local branch office, point of sale, or 1-800-CALL-AIX (for information about support offer- ings). These services may be billable. Faxes on a variety of sub- jects may be ordered free of charge from 1-800-IBM-4FAX. Outside the U.S. call 415-855-4329 using a fax machine phone. When you send comments to IBM, you grant IBM a nonexclusive right to use or distribute your comments in any way it believes appropriate without incurring any obligation to you. NOTE: If you have a problem report or item number, supplying that number may help us determine why a procedure did or did not work in your specific situation. Problem Report or Item #: Branch Office or Customer #: Be sure to print your name and fax number below if you would like a reply: Name: Fax Number: ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ END OF DOCUMENT (sparse.files.32.bak, 4FAX# 3753) About Sparse Files 6