[Ocfs2-users] Concurrent write performance issues with OCFS2

[Sunil Mushran]

In 1.4, the local allocator window is small. 8MB. Meaning the node
has to hit the global bitmap after every 8MB. In later releases, the
window is much larger.

Second, a single node is not a good baseline. A better baseline is
multiple nodes writing concurrently to the block device. Not fs.
Use dd. Set different write offsets. This should help figure out how
the shared device works with multiple nodes.

On 2/28/2012 9:24 AM, Erik Schwartz wrote:
> I have a two-node RHEL5 cluster that runs the following Linux kernel and
> accompanying OCFS2 module packages:
>
>    * kernel-2.6.18-274.17.1.el5
>    * ocfs2-2.6.18-274.17.1.el5-1.4.7-1.el5
>
> A 2.5TB LUN is presented to both nodes via DM-Multipath. I have carved
> out a single partition (using the entire LUN), and formatted it with OCFS2:
>
>    # mkfs.ocfs2 -N 2 -L 'foofs' -T datafiles /dev/mapper/bams01p1
>
> Finally, the filesystem is mounted to both nodes with the following options:
>
>    # mount | grep bams01
> /dev/mapper/bams01p1 on /foofs type ocfs2
> (rw,_netdev,noatime,data=writeback,heartbeat=local)
>
> ----------
>
> When a single node is writing arbitrary data (i.e. dd(1) with /dev/zero
> as input) to a large (say, 10 GB) file in /foofs, I see the expected
> performance of ~850 MB/sec.
>
> If both nodes are concurrently writing large files full of zeros to
> /foofs, performance drops way down to ~45 MB/s. I experimented with each
> node writing to /foofs/test01/ and /foofs/test02/ subdirectories,
> respectively, and found that performance increased slightly to a - still
> poor - 65 MB/s.
>
> ----------
>
> I understand from searching past mailing list threads that the culprit
> is likely related to the negotiation of file locks, and waiting for data
> to be flushed to journal / disk.
>
> My two questions are:
>
> 1. Does this dramatic write performance slowdown sound reasonable and
> expected?
>
> 2. Are there any OCFS2-level steps I can take to improve this situation?
>
>
> Thanks -
>