[Ocfs2-users] Slow backups, slow rsync

[Sunil Mushran]

A cold cache stat() in a local file system implies reading an inode
from disk. On a clustered fs, it implies that plus mastering the
lock resource and taking a clustered read lock on that resource.
(The latter process involves some network messaging).

A hot cache stat() implies a cached inode(). On a clustered fs,
it typically also implies the resource has been mastered and
that at the most it has to take a clustered read lock.

In short, while a cold cache stat takes a perf hit on a local fs,
it takes a double perf hit on a cfs.

rsync will stat each and every inode. To get good performance, one
needs to ensure that the inodes are cached.

With 2 million inodes and 8G RAM, I suspect your system is not being
able to keep the inodes in cache and is being forced to read all the
inodes from disk. (Note that a cfs require more memory than a local
fs too as it has to also track the lock resources.)

One suggestion is to increase the RAM. You will have to do it on
all nodes. But it may not help if say you have a user process that
consumes a lot of memory forcing the kernel to free up the inodes.

Secondly, as far as I know, rsync reads the entire file. It was written
for environments where the network is by far the slowest link. If in
your environment, you can use mtime to detect file changes, then maybe
a cruder "find -mtime ... -exec scp ..." may be quicker.

The last point would be concerning block sizes. As you have small files,
you could have gotten more mileage by using a 2K sized blocks. Disk
io for inode reads would have reduced by half. But it's not something
you can change midstream.

In 2.6.24 we added inline data that pushed really small files into the
inode itself saving a block read. The tools support is being worked on.
If you have a large number of < 4K files, you could benefit from it.

Sunil

Michael Moody wrote:
> gsc at pba2 ~ $ uname -a
> Linux pba2.bodybuilding.com 2.6.24-gentoo-r8 #1 SMP Tue May 20 15:04:37 PDT 2008 x86_64 Intel(R) Xeon(R) CPU 5148 @ 2.33GHz GenuineIntel GNU/Linux
>
> pba2 ~ # debugfs.ocfs2 -R "stats" /dev/sdc1
>         Revision: 0.90
>         Mount Count: 0   Max Mount Count: 20
>         State: 0   Errors: 0
>         Check Interval: 0   Last Check: Sun Dec 30 00:15:02 2007
>         Creator OS: 0
>         Feature Compat: 1 BackupSuper
>         Feature Incompat: 16 Sparse
>         Tunefs Incomplete: 0 None
>         Feature RO compat: 0 None
>         Root Blknum: 5   System Dir Blknum: 6
>         First Cluster Group Blknum: 3
>         Block Size Bits: 12   Cluster Size Bits: 12
>         Max Node Slots: 10
>         Label: pba
>         UUID: BD59CD9153324E1BAC360B1F1D4B6918
>         Inode: 2   Mode: 00   Generation: 3898535000 (0xe85eec58)
>         FS Generation: 3898535000 (0xe85eec58)
>         Type: Unknown   Attr: 0x0   Flags: Valid System Superblock
>         User: 0 (root)   Group: 0 (root)   Size: 0
>         Links: 0   Clusters: 139458249
>         ctime: 0x47775386 -- Sun Dec 30 00:15:02 2007
>         atime: 0x0 -- Wed Dec 31 16:00:00 1969
>         mtime: 0x47775386 -- Sun Dec 30 00:15:02 2007
>         dtime: 0x0 -- Wed Dec 31 16:00:00 1969
>         ctime_nsec: 0x00000000 -- 0
>         atime_nsec: 0x00000000 -- 0
>         mtime_nsec: 0x00000000 -- 0
>         Last Extblk: 0
>         Sub Alloc Slot: Global   Sub Alloc Bit: 65535
>
> pba2 ~ # cat /proc/meminfo
> MemTotal:      8195240 kB
> MemFree:        126828 kB
> Buffers:        968396 kB
> Cached:        2518248 kB
> SwapCached:          0 kB
> Active:        2598712 kB
> Inactive:      3121128 kB
> SwapTotal:     2008116 kB
> SwapFree:      2007932 kB
> Dirty:            9568 kB
> Writeback:           0 kB
> AnonPages:     2233240 kB
> Mapped:          42972 kB
> Slab:          2017924 kB
> SReclaimable:   798440 kB
> SUnreclaim:    1219484 kB
> PageTables:     212700 kB
> NFS_Unstable:        0 kB
> Bounce:              0 kB
> CommitLimit:   6105736 kB
> Committed_AS:  7952860 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed:    293184 kB
> VmallocChunk: 34359444779 kB
> HugePages_Total:     0
> HugePages_Free:      0
> HugePages_Rsvd:      0
> HugePages_Surp:      0
> Hugepagesize:     2048 kB
>
> 5 nodes in the cluster.
>
> Michael
>
> -----Original Message-----
> From: Sunil Mushran [mailto:Sunil.Mushran at oracle.com]
> Sent: Tuesday, July 01, 2008 4:31 PM
> To: Michael Moody
> Cc: ocfs2-users at oss.oracle.com
> Subject: Re: [Ocfs2-users] Slow backups, slow rsync
>
> Which kernel?
> uname -a?
>
> block/cluster sizes?
> debugfs.ocfs2 -R "stats" /dev/sdX
>
> How many nodes in your cluster?
>
> memory?
> cat /proc/meminfo
>