[Ocfs-devel] Re: [Ocfs-users] Lock contention issue with ocfs
Wim Coekaerts
wim.coekaerts at oracle.com
Thu Mar 11 17:36:07 CST 2004
sunil, kurt mark and myself do ocfs work so I think we read this ;)
On Thu, Mar 11, 2004 at 08:27:21PM -0500, Jeremy Schneider wrote:
> Hey list...
>
> Sorry for all the emails lately about this bug I'm working on. I
> finally got tired of waiting for support to track it down so today I
> found it myself and fixed it. Of course I'm going through normal
> support channels to get an official [supported] fix; but if anyone's
> interested here's a patch that'll do it. And you never know... it
> might speed up the process a bit by posting this explanation. I tested
> the patch (it only changes a few lines) and it successfully fixed the
> problem without any side-effects. I'll try to explain it as clearly as
> possible but programmers aren't always the most brilliant communicators.
> ;)
>
> VERSIONS AFFECTED:
> all that I'm aware of (including 1.0.10-1)
>
> BUG:
> the EXCLUSIVE_LOCK on the DirNode is not being released after a file
> is created if the directory has multiple DirNodes and the last DirNode
> (not the one used for locking) has ENABLE_CACHE_LOCK set.
> ENABLE_CACHE_LOCK can be set on the last DirNode if the DirNode has this
> lock when the 255th file is added because ocfs copies the entire
> structure byte-for-byte (including locks) to the new DirNode it is
> creating and does not clear out the locks.
>
> CAUSE:
> ocfs_insert_file() in Common/ocfsgendirnode.c (line 1594 in the code
> available through subversion at oss.oracle.com right now) looks at the
> incorrect variable (DirNode, the last one instead of LockNode the
> locking one) to check for ENABLE_CACHE_LOCK and does not release the
> EXCLUSIVE_LOCK because it thinks that the node is holding an
> ENABLE_CACHE_LOCK instead of an EXCLUSIVE_LOCK. I've attached a patch
> that fixes it if anyone cares to have a look...
>
> STEPS TO REPRODUCE BUG:
> create a new ocfs directory. when you initially create it it will
> have ENABLE_CACHE_LOCK set... *BEFORE* accessing the directory from
> *ANY* other node (this will release the ENABLE_CACHE_LOCK) create at
> least 255 files in the directory. optionally you can check with
> debugocfs -D to see if the last (not-locking) DirNode has
> ENABLE_CACHE_LOCK set. *NOW* access it from the other node (to change
> the locking DirNode from ENABLE_CACHE_LOCK to NO_LOCK) and then create a
> file from either node -- the EXCLUSIVE_LOCK will not be released.
> attempting any operation from the opposite node that requires an
> EXCLUSIVE_LOCK will cause that node to hang until you do an operation
> that successfully releases the EXCLUSIVE_LOCK (e.g. deleting a file).
>
> FIX:
> Of course the best solution is to fix ocfs_insert_file() to look at
> the correct DirNode for the lock. Another thought would be to update
> the code that creates a new DirNode and have it clear out locks after
> copying the DirNode, although this just seems extraneous to me.
>
> PATCH:
> I have tested this patch (it is a minimal change) and it successfully
> fixes the bug. It releases the lock from the correct pointer (either
> the variable "DirNode" or "LockNode" depending on whether they come from
> the same disk DirNode) but it now also checks the correct variable for
> ENABLE_CACHE_LOCK.
>
> REQUESTED COMPENSATION:
> hehe... email me for the address where you can send pizza. I wonder
> if any oracle developers really do read this list...
>
>
>
> Index: ocfsgendirnode.c
> ===================================================================
> --- ocfsgendirnode.c (revision 5)
> +++ ocfsgendirnode.c (working copy)
> @@ -1591,17 +1591,26 @@
> indexOffset = -1;
> }
>
> - if (DISK_LOCK_FILE_LOCK (DirNode) !=
> OCFS_DLM_ENABLE_CACHE_LOCK) {
> - /* This is an optimization... */
> - ocfs_acquire_lockres (LockResource);
> - LockResource->lock_type = OCFS_DLM_NO_LOCK;
> - ocfs_release_lockres (LockResource);
> + if (LockNode->node_disk_off == DirNode->node_disk_off) {
> + if (DISK_LOCK_FILE_LOCK (DirNode) !=
> OCFS_DLM_ENABLE_CACHE_LOCK) {
> + /* This is an optimization... */
> + ocfs_acquire_lockres (LockResource);
> + LockResource->lock_type = OCFS_DLM_NO_LOCK;
> + ocfs_release_lockres (LockResource);
>
> - if (LockNode->node_disk_off == DirNode->node_disk_off)
> + /* Reset the lock on the disk */
> + DISK_LOCK_FILE_LOCK (DirNode) =
> OCFS_DLM_NO_LOCK;
> + }
> + } else {
> + if (DISK_LOCK_FILE_LOCK (LockNode) !=
> OCFS_DLM_ENABLE_CACHE_LOCK) {
> + /* This is an optimization... */
> + ocfs_acquire_lockres (LockResource);
> + LockResource->lock_type = OCFS_DLM_NO_LOCK;
> + ocfs_release_lockres (LockResource);
> +
> /* Reset the lock on the disk */
> - DISK_LOCK_FILE_LOCK (DirNode) =
> OCFS_DLM_NO_LOCK;
> - else
> DISK_LOCK_FILE_LOCK (LockNode) =
> OCFS_DLM_NO_LOCK;
> + }
> }
>
> status = ocfs_write_dir_node (osb, DirNode, indexOffset);
>
>
>
> >>> Sunil Mushran <Sunil.Mushran at oracle.com> 03/10/2004 5:49:58 PM >>>
> I hope, that when you were reading the dirnode, etc. using debugocfs,
> you were accessing the volume via the raw device. If you weren't, do
> so.
> This is important because that's the only way to ensure directio.
> Else,
> you will be reading potentially stale data from the buffer cache.
>
> Coming to the issue at hand. ls does not take an EXCLUSIVE_LOCK.
> And all EXCLUSIVE_LOCKS are released when the operation is over.
> So am not sure what is happening. Using debugocfs correctly should
> help
> us understand the problem.
>
> Also, whenever you do your file operations (cat etc.) ensure those ops
> are
> o_direct. Now I am not sure why this would cause a problem, but do not
> do buffered operations. ocfs does not support shared mmap.
>
> If you download the 1.0.10 tools, you will not need to manually map
> the
> raw device. The tools do that automatically.
>
> So, upgrade to 1.0.10 module and tools. See if you can reproduce the
> problem.
>
>
> <<<<...>>>>
> _______________________________________________
> Ocfs-devel mailing list
> Ocfs-devel at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs-devel
More information about the Ocfs-devel
mailing list