[Ocfs2-tools-devel] [PATCH 0/4] defrag.ocfs2
Tristan Ye
tristan.ye at oracle.com
Tue Nov 23 17:39:21 PST 2010
Goldwyn Rodrigues wrote:
> Hi Tristan,
>
>
> On Tue, Nov 23, 2010 at 1:15 AM, Tristan Ye <tristan.ye at oracle.com> wrote:
>> Goldwyn Rodrigues wrote:
>>> These patches add the defrag.ocfs2 utility to ocfs2-tools. The defrag
>>> utility defrags the files of the filessystem, by consolidating the
>>> extents of the files and consolidating the dirents of the directory. It
>>> also clears up unused allocation groups.
>>>
>>> This utility requires the patch to fix alloc_clusters to allocate the
>>> best clusters. I have sent this patch independently, since it is
>>> not part of the defragmentation process.
>>>
>>> Changes:
>>> - incorporated Joel's comments
>>>
>> Hi Goldwyn,
>>
>> Thanks for your persistent efforts on making the defrag.ocfs2 tools
>> reality;)
>>
>> I'm playing it around with my current o2info patches, without quite diving
>> into
>> your codes deeply now, per my understanding, your patches seems aiming at
>> mainly
>> at defraging files, to reduce the extent records as your main goal in other
>> words.
>> which seems be changed a lot since your first version from the very
>> beginning, right?
>> after pass1/2, we reduce the number of extents for files and directories,
>> the pass3
>> can hopefully make some extent blocks be released, right?
>
>
> Thanks for reviewing. There is no pass 3 in the defrag utility. Pass1
> which defrags files and directories and pass 2 which frees unused
> allocation groups.
>
> After the first post, when you pointed out similar numbers, I added
> the pass. In the post of the utility in July, I had posted pass 3, to
> consolidate the filesystem as a whole, but Joel did not like the idea
> of copying extents across filesystem, so I dropped it in the post of
> September onwards. I guess, you and Joel can discuss what you really
> want in the utility ;)
Yep,
As Joel told, we may start with file's defragmentation first at the
very beginning, then take whole-volume things in later series;)
>
>> Following are some interesting numbers from 'o2info' and 'debugfs.ocfs2'
>>
>> [1]. Free chunks number/size being reported against global bitmap:
>>
>> -. Before defrag:
>> --------------------------------------------------------
>> # o2info --freefrag 32 /dev/sda9
>> Blocksize: 4096 bytes
>> Clustersize: 32768 bytes
>> Total clusters: 1250307
>> Free clusters: 406165 (32.5%)
>>
>> Min. free extent: 32 KB
>> Max. free extent: 184288 KB
>> Avg. free extent: 320 KB
>>
>> Chunksize: 32768 bytes (1 clusters)
>> Total chunks: 1250308
>> Free chunks: 406165 (32.5%)
>>
>> HISTOGRAM OF FREE EXTENT SIZES:
>> Extent Size Range : Free extents Free Clusters Percent
>> 32K... 64K- : 5237 5237 1.29%
>> 64K... 128K- : 7438 17878 4.40%
>> 128K... 256K- : 14372 79073 19.47%
>> 256K... 512K- : 6455 74499 18.34%
>> 512K... 1024K- : 2060 43276 10.65%
>> 1M... 2M- : 1134 47662 11.73%
>> 2M... 4M- : 888 64649 15.92%
>> 4M... 8M- : 24 3463 0.85%
>> 8M... 16M- : 4 1324 0.33%
>> 16M... 32M- : 1 764 0.19%
>> 128M... 256M- : 12 68340 16.83%
>> --------------------------------------------------------
>>
>> -. After defrag:
>> --------------------------------------------------------
>> # o2info --freefrag 32 /dev/sda9
>> Blocksize: 4096 bytes
>> Clustersize: 32768 bytes
>> Total clusters: 1250307
>> Free clusters: 408550 (32.7%)
>>
>> Min. free extent: 32 KB
>> Max. free extent: 184288 KB
>> Avg. free extent: 320 KB
>>
>> Chunksize: 32768 bytes (1 clusters)
>> Total chunks: 1250308
>> Free chunks: 407869 (32.6%)
>>
>> HISTOGRAM OF FREE EXTENT SIZES:
>> Extent Size Range : Free extents Free Clusters Percent
>> 32K... 64K- : 5872 5872 1.44%
>> 64K... 128K- : 7864 18959 4.64%
>> 128K... 256K- : 15009 82058 20.09%
>> 256K... 512K- : 6321 73165 17.91%
>> 512K... 1024K- : 1980 41703 10.21%
>> 1M... 2M- : 1123 47472 11.62%
>> 2M... 4M- : 886 64717 15.84%
>> 4M... 8M- : 27 4080 1.00%
>> 8M... 16M- : 6 2055 0.50%
>> 16M... 32M- : 3 2128 0.52%
>> 32M... 64M- : 2 2951 0.72%
>> 128M... 256M- : 11 62709 15.35%
>>
>> --------------------------------------------------------
>>
>>
>> [2]. Extent allocator layout:
>> -. Before defrag:
>> --------------------------------------------------------------------
>> debugfs: stat extent_alloc:0000
>> Inode: 28 Mode: 0644 Generation: 2820423521 (0xa81c3f61)
>> FS Generation: 2820423521 (0xa81c3f61)
>> CRC32: edba1e7d ECC: 0270
>> Type: Regular Attr: 0x0 Flags: Valid System Allocbitmap Chain
>> Dynamic Features: (0x0)
>> User: 0 (root) Group: 0 (root) Size: 12582912
>> Links: 1 Clusters: 384
>> ctime: 0x4ceb3934 -- Tue Nov 23 11:47:00 2010
>> atime: 0x4ceb3934 -- Tue Nov 23 11:47:00 2010
>> mtime: 0x4ceb3934 -- Tue Nov 23 11:47:00 2010
>> dtime: 0x0 -- Thu Jan 1 08:00:00 1970
>> ctime_nsec: 0x00000000 -- 0
>> atime_nsec: 0x00000000 -- 0
>> mtime_nsec: 0x00000000 -- 0
>> Refcount Block: 0
>> Last Extblk: 0 Orphan Slot: 0
>> Sub Alloc Slot: Global Sub Alloc Bit: 12
>> Bitmap Total: 3072 Used: 100 Free: 2972
>> Clusters per Group: 128 Bits per Cluster: 8
>> Count: 243 Next Free Rec: 3
>> ## Total Used Free Block#
>> 0 1024 32 992 250624
>> 1 1024 34 990 251648
>> 2 1024 34 990 252672
>>
>> Group Chain: 0 Parent Inode: 28 Generation: 2820423521
>> CRC32: a492f1a3 ECC: 00f8
>> ## Block# Total Used Free Contig Size 0
>> 250624 1024 32 992 983 4032
>> Group Chain: 1 Parent Inode: 28 Generation: 2820423521
>> CRC32: 3dedeb22 ECC: 01a0
>> ## Block# Total Used Free Contig Size 0
>> 251648 1024 34 990 983 4032
>> Group Chain: 2 Parent Inode: 28 Generation: 2820423521
>> CRC32: 059ae49a ECC: 0187
>> ## Block# Total Used Free Contig Size 0
>> 252672 1024 34 990 984 4032
>> --------------------------------------------------------------------
>> -. After defrag:
>> --------------------------------------------------------------------
>> debugfs: stat extent_alloc:0000
>> Inode: 28 Mode: 0644 Generation: 2820423521 (0xa81c3f61)
>> FS Generation: 2820423521 (0xa81c3f61)
>> CRC32: dec80e05 ECC: 00b7
>> Type: Regular Attr: 0x0 Flags: Valid System Allocbitmap Chain
>> Dynamic Features: (0x0)
>> User: 0 (root) Group: 0 (root) Size: 4194304
>> Links: 1 Clusters: 128
>> ctime: 0x4ceb3934 -- Tue Nov 23 11:47:00 2010
>> atime: 0x4ceb3934 -- Tue Nov 23 11:47:00 2010
>> mtime: 0x4ceb3934 -- Tue Nov 23 11:47:00 2010
>> dtime: 0x0 -- Thu Jan 1 08:00:00 1970
>> ctime_nsec: 0x00000000 -- 0
>> atime_nsec: 0x00000000 -- 0
>> mtime_nsec: 0x00000000 -- 0
>> Refcount Block: 0
>> Last Extblk: 0 Orphan Slot: 0
>> Sub Alloc Slot: Global Sub Alloc Bit: 12
>> Bitmap Total: 1024 Used: 98 Free: 926
>> Clusters per Group: 128 Bits per Cluster: 8
>> Count: 243 Next Free Rec: 1
>> ## Total Used Free Block#
>> 0 1024 98 926 250624
>>
>> Group Chain: 0 Parent Inode: 28 Generation: 2820423521
>> CRC32: 1aec9e0b ECC: 00a0
>> ## Block# Total Used Free Contig Size 0
>> 250624 1024 98 926 923 4032
>> --------------------------------------------------------------------
>>
>>
>> From above statistics, the awesome thing is that we do reduce the number of
>> extents
>> for files, and gained 2 extent block being freed, while looking at the
>> global_bitmap,
>> we even found more smaller free chunks at the cost of eating bigger free
>> chunks up, it
>> somehow makes the volume look more fragmented in terms of 'how many free
>> contiguous chunks
>> and their size, in global_bitmap for our allocation'.
>>
>>
>
> Yes, I understand. Having pass 3 of defragging the filesystem as a
> whole would have shown better numbers with your utility.
More information about the Ocfs2-tools-devel
mailing list