Why Numbers from "du -sk" and "df" Disagree

About This Document
About du -sk and df
The Problem
Why the Numbers Do Not Add Up
ls Command

About This Document

This document describes why numbers from du -s and df disagree and is applicable to AIX Version 3.2 and Version 4.x.

About "du -sk" and "df"

Notice du and df report on only the blocks allocated for data actually written, The ls command reports slightly different results depending on the type of file. See section on ls Command.

On AIX versions prior to 4.1, df reports its statistics in 1024 byte units and du reports in 512 byte units. On AIX 4.1 and later, both df and du default to 512 byte units. The following discussion uses 4.x df and du; thus all units are in 512-byte blocks.

The Problem

When the user runs du -s /filesystem_path and subtracts this value from the total block count as reported by df to get a free block value, the calculation yields a value that is greater than the free block value reported by df. For example:

 
   % du -s /tmp 
   12920   /tmp 
   % df /tmp 
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on 
/dev/hd3           57344     42208   26%      391     4% /tmp 
<total from df> - <used from du> = <false free block count> 
57344 - 12920 = 44424

44424 is greater than 42208. The reason for this discrepancy has to do with the implementation of du and df.

du -sk

du -sk traverses the file tree, adding up the number of blocks allocated to each directory, symlink, and file as reported by the stat() system call. This is how du arrives at its total value.

df

df looks at the file system disk block allocation maps to arrive at its total and free values.

Why the Numbers Do Not Add Up

The file system allocates some of the disk blocks in the file system to record its data. This data is referred to as meta data. Meta data is not visible to most user level programs. Examples of meta data are inodes, disk maps, indirect blocks, and super blocks.

du is an example of a user-level program that is not aware of file system meta data, while df looks at the file system disk allocation maps and is aware of file system meta data. df obtains the true file system statistics, whereas du sees only a partial picture. For example, an empty 4 MB JFS file system created with frag=4096 and nbpi=4096 has the following meta data allocated:
```
 
   1 4k block for the LVM 
   2 4k super blocks 
   2 4k blocks for disk maps 
   2 4k blocks for inode maps 
   2 4k blocks for .indirect 
   32 4k blocks for inodes 
   ------------------------- 
   41 4k blocks for meta data on an empty 4M file system 
```
For AIX Version 3.2:
```
 
   # du /foo 
   8       /foo 
```
The eight 512-byte blocks reported for du on this empty file system are the blocks used by the root directory.

To get the output to match df we must add in the meta data. First, convert 41 4K blocks to 512 byte units:
```
 
   41 * 8 = 328 
   328(meta data) + 8(from du) = 336 
```
so there are 336 512-byte blocks allocated on this empty file system, thus:
```
 
   8192(total blocks) - 336(used from du + meta data) = 7856 
```
This does match the output from the free column reported by df.
```
 
  df /foo 
  Filesystem   512-blocks   Free  %Used   Iused  %Iused  Mounted on 
  /dev/lv01          8192   7856     5%      16      2%  /foo 
```
For AIX Version 4.x:
```
 
    du /foo 
   8       /foo/lost+found 
   16      /foo 
```
The 16 512-byte blocks reported for du on this empty file system are the blocks used by the root directory.

To get the output to match df we must add in the meta data. First, convert 41 4K blocks to 512-byte units:
```
 
    41 * 8 = 328 
    328(meta data) + 16(from du) = 344 
```
so there are 344 512-byte blocks allocated on this empty file system. For example:
```
 
    8192(total blocks) - 344(used from du + meta data) = 7848 
```
This does match the output from the free column reported by df.
```
 
 df /foo 
  Filesystem  512-blocks  Free   %Used   Iused   %Iused   Mounted on 
  /dev/lv01         8192  7848      5%      16       2%   /foo 
```
This calculation was easy to perform on an empty file system. However, on a non-empty file system, the meta data for file indirect blocks comes into play and such calculations are tedious and impractical.

In conclusion, du -s produces a value that reflects the number of disk blocks that are allocated to files and directories. df reports on the actual allocation state of the file system. The true allocation state includes both user data (files and directories) plus meta data.
Another example that contributes to a difference between du and df is the following:
If someone is running an application with a file open in a directory and the open file is removed, the du output reflects a reduced size for this directory. However, df does not show a reduced size because all blocks in the file system remain allocated until the application that has the file open closes the file. After the file closure, df shows reduced usage for the filesystem.

ls Command

The following illustrates ls output vs du and df for sparse files.

ls gives data on individual files based on the difference between the end-of-file (the largest offset where data is written) and the beginning of file, whether or not blocks were actually allocated to the file. A 32MB file (as reported by ls) may not have 32MB of data written to it if not written sequentially.
du shows the blocks actually allocated to an individual file.
df shows the blocks allocated in the entire filesystem, including inodes and other meta data.

An example sparse file can be created fairly easily. To do this, open the file, seek to a large address, and write some data. This can be demonstrated with the dd command, as follows:

Create a regular file.

 
    date > notsparse 
    ls -l

The output of the ls command will be similar to the following:

 
    total 8 
    -rw-r--r--   1 root     sys         29 Dec 21 08:12 notsparse

Use the fileplace command to see how many allocated and unallocated blocks are included in the file notsparse.

NOTE: BOS Extensions 1/Extended Commands (bosext1.extcmds.obj) must be installed to enable the fileplace commands or perfagent.tools for AIX Version 4.x.
```
 
    fileplace notsparse 
```
The output for 3.2 will look similar to the following:
```
 
    File: notsparse  Size: 29 bytes  Vol: /dev/lv03 (4096 byte blks) 
    Logical blocks 
    -------------- 
    00016                         1 blk,      4 KB,  100.0% 
```
NOTE: BOS Extensions 1 (bosext1.extcmds.ob) must be installed to enable the fileplace command.

The output for AIX Version 4 will look similar to the following:
```
 
    File: notsparse  Size: 29 bytes  Vol: /dev/lv03 
    Bkl Size: 4096  Frag size: 4096 Nfrags: 1 Compress: no 
    Logical Fragment 
    ---------------- 
    00716                   1 frags        4096 bytes,  100.0% 
```
NOTE: Performance Analysis and Control Commands (perfagent.tools) must be installed to enable the fileplace command for AIX Version 4.x.
The du command will also reflect how many 512-byte blocks a file occupies.
```
 
    du -rs * 
```
Example output will look similar to the following:
```
 
    8 notsparse 
```
Now create a sparse file using the regular file notsparse as input, as shown in the following:
```
 
    touch sparse.1 
    dd if=notsparse of=sparse.1 seek=100 
```
Example output will look similar to the following:
```
 
    dd: 0+1 records in. 
    dd: 0+1 records out. 
```
The dd command takes the data from the regular file and places it, in 100 512-byte blocks, into the sparse.1 file. Nothing is written to the initial 99 512-byte blocks. The following steps show the characteristics of the resulting file.

The ls command reports the distance from block zero to the last block in the file:

 
    ls -l

Example output will look similar to the following:

 
    total 16 
    -rw-r--r--  1 root   sys       29 Dec 21 08:12 notsparse 
    -rw-r--r--  1 root   sys    51229 Dec 21 08:13 sparse.1

The fileplace command will accurately report what blocks are unallocated and allocated, for example:
```
 
    fileplace sparse.1 
```
Example output for AIX Version 3.2 will look similar to the following:
```
 
    File: sparse.1  Size: 51229 bytes  Vol: /dev/lv03 (4096 byte blks) 
    Logical blocks 
    -------------- 
    unallocated               12 blks,       48 KB,   92.3% 
    00018                      1 blk,         4 KB,    7.7% 
```
Example output for AIX Version 4.1 will look similar to the following:
```
 
    File: sparse.1  Size: 51229 bytes Vol: /dev/lv03 
    Blk Size: 4096  Frag Size: 4096  Nfrags: 1  Compress: no 
    Logical Fragment 
    ---------------- 
    unallocated               12 frags  49152 Bytes,  0.0% 
    0000769                    1 frags   4096 Bytes, 100.0% 
```
The du command reports the number of allocated blocks the file takes, for example:
```
 
    du -rs * 
```
The example output will look similar to the following:
```
 
    8 notsparse 
    8 sparse.1 
```
Each command is correct reporting data specific to its intended purpose. ls shows the range of offsets where data can be read from or written to a file. Reading from an offset where no data was written will appear to be zero-filled. du and df report only blocks allocated for data actually written.

Why Numbers from "du -sk" and "df" Disagree: diff.between.du.df.all.cmd ITEM: FAX
Dated: 99/03/10~00:00 Category: cmd
This HTML file was generated 99/06/24~12:41:57
Comments or suggestions? Contact us