[Ocfs2-users] read/write performance across cluster

[Srinivas Eeda]

Florin Andrei wrote:
> Red Hat 5 kernel 64 bit, OCFS2 1.4.4.
>
> Two servers with lots of RAM, filesystem is on a SAN via Fiber Channel, 
> the Ethernet between servers is pretty fast.
>
> There's a volume on the SAN used for logs. A log collector runs on one 
> server, appending lines to a variety of log files in real time. The same 
> filesystem is mounted on the other server via OCFS2.
>
> Now here's the thing: if I grep the log files on the same system that 
> appends data to the log files, grep is fast. If I grep the logs on the 
> other system, it's slower. Depending on the situation, it can be MUCH 
> slower, many orders of magnitude slower.
>
> Is this some kind of file locking issue?
>   
Yes, there is locking involved. Extending a file needs an exclusive 
lock. Grepping a file needs read lock. If the same node(lets call it 
writer node) does extending and grepping, then grep already has a 
compatible lock and the data is also cached on this node. When another 
node does the grep(lets call it reader node), then it needs to acquire 
read lock. This requires the writer node to down grade the lock which 
requires the node to flush data to the disk. Then the reader node has to 
read the data to the cache. So when grep is done on the reader node, 
there are additional messages and additional I/O's to the disk are done.
> I expected the performance to be somewhat different. After all, if I run 
> grep on the system that is also writing to the logs, the files are fresh 
> in RAM, so it must be fast. But I was surprised by how much slower grep 
> is on the other system. This doesn't feel right. Both systems are on the 
> same fast LAN, the SAN is fast, etc.
>   
Just curious, do you have any numbers to show the time difference?
> Any suggestions?
>
>