<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are running a Ruby/Passenger app called Canvas (by Instructure). It runs from 3 application servers that use shared storage (iSCSI, EqualLogic Volume, LVM2, OCFS2) for user uploaded files. We have been experiencing strange behavior
where a ruby and jbd2 process will go into a ‘D’ (I/O wait) state randomly with some I/O operations on the shared storage (a file upload most commonly). I turned on some kernel debugging options for one of the application servers and captured the following
‘possible circular locking dependency’ error during one of these times. Would anyone be able to provide any insight into possible ways to troubleshoot this further?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Our distro is Gentoo Linux using hardened kernel sources v3.11.2 on this particular app server. We do not have GrSecurity or PaX enabled. The other app servers are running kernel 3.2.30. Was testing newer kernel to see if it helped –
which it doesn’t appear to have.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for any help.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[ 858.570825] ======================================================<o:p></o:p></p>
<p class="MsoNormal">[ 858.570878] [ INFO: possible circular locking dependency detected ]<o:p></o:p></p>
<p class="MsoNormal">[ 858.570932] 3.11.2-hardenedcanvas-2013-12-16 #4 Not tainted<o:p></o:p></p>
<p class="MsoNormal">[ 858.570980] -------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">[ 858.571033] ruby/3710 is trying to acquire lock:<o:p></o:p></p>
<p class="MsoNormal">[ 858.571073] (&sc->sc_send_lock){+.+...}, at: [<ffffffff81355567>] o2net_send_message_vec+0x543/0x8dd<o:p></o:p></p>
<p class="MsoNormal">[ 858.571165]<o:p></o:p></p>
<p class="MsoNormal">but task is already holding lock:<o:p></o:p></p>
<p class="MsoNormal">[ 858.571213] (jbd2_handle){+.+...}, at: [<ffffffff8123986e>] start_this_handle+0x4e8/0x57d<o:p></o:p></p>
<p class="MsoNormal">[ 858.571294]<o:p></o:p></p>
<p class="MsoNormal">which lock already depends on the new lock.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[ 858.571361]<o:p></o:p></p>
<p class="MsoNormal">the existing dependency chain (in reverse order) is:<o:p></o:p></p>
<p class="MsoNormal">[ 858.571423]<o:p></o:p></p>
<p class="MsoNormal">-> #4 (jbd2_handle){+.+...}:<o:p></o:p></p>
<p class="MsoNormal">[ 858.571464] [<ffffffff810c761c>] lock_acquire+0xbf/0xfe<o:p></o:p></p>
<p class="MsoNormal">[ 858.571539] [<ffffffff812398e0>] start_this_handle+0x55a/0x57d<o:p></o:p></p>
<p class="MsoNormal">[ 858.571615] [<ffffffff81239b0c>] jbd2__journal_start+0x9f/0x15c<o:p></o:p></p>
<p class="MsoNormal">[ 858.571691] [<ffffffff8121feab>] __ext4_journal_start_sb+0x89/0xa6<o:p></o:p></p>
<p class="MsoNormal">[ 858.571771] [<ffffffff812005c1>] ext4_dirty_inode+0x25/0x54<o:p></o:p></p>
<p class="MsoNormal">[ 858.571844] [<ffffffff8117a338>] __mark_inode_dirty+0x54/0x210<o:p></o:p></p>
<p class="MsoNormal">[ 858.571920] [<ffffffff8116d0c7>] update_time+0x9c/0xa7<o:p></o:p></p>
<p class="MsoNormal">[ 858.571989] [<ffffffff8116d16e>] file_update_time+0x9c/0xbd<o:p></o:p></p>
<p class="MsoNormal">[ 858.572060] [<ffffffff8120077f>] ext4_page_mkwrite+0x50/0x2a1<o:p></o:p></p>
<p class="MsoNormal">[ 858.572133] [<ffffffff81130997>] __do_fault+0x125/0x2f5<o:p></o:p></p>
<p class="MsoNormal">[ 858.572203] [<ffffffff81132d9a>] handle_pte_fault+0x1d4/0x4f2<o:p></o:p></p>
<p class="MsoNormal">[ 858.572277] [<ffffffff81133459>] handle_mm_fault+0x1af/0x1c9<o:p></o:p></p>
<p class="MsoNormal">[ 858.572349] [<ffffffff81029a3c>] __do_page_fault+0x317/0x35e<o:p></o:p></p>
<p class="MsoNormal">[ 858.572423] [<ffffffff81029aab>] do_page_fault+0x9/0xb<o:p></o:p></p>
<p class="MsoNormal">[ 858.572491] [<ffffffff816e4e45>] page_fault+0x25/0x30<o:p></o:p></p>
<p class="MsoNormal">[ 858.572560]<o:p></o:p></p>
<p class="MsoNormal">-> #3 (sb_pagefaults){.+.+..}:<o:p></o:p></p>
<p class="MsoNormal">[ 858.572643] [<ffffffff810c761c>] lock_acquire+0xbf/0xfe<o:p></o:p></p>
<p class="MsoNormal">[ 858.572712] [<ffffffff81159900>] __sb_start_write+0x111/0x161<o:p></o:p></p>
<p class="MsoNormal">[ 858.572786] [<ffffffff81200773>] ext4_page_mkwrite+0x44/0x2a1<o:p></o:p></p>
<p class="MsoNormal">[ 858.572860] [<ffffffff81130997>] __do_fault+0x125/0x2f5<o:p></o:p></p>
<p class="MsoNormal">[ 858.572928] [<ffffffff81132d9a>] handle_pte_fault+0x1d4/0x4f2<o:p></o:p></p>
<p class="MsoNormal">[ 858.573002] [<ffffffff81133459>] handle_mm_fault+0x1af/0x1c9<o:p></o:p></p>
<p class="MsoNormal">[ 858.573075] [<ffffffff81029a3c>] __do_page_fault+0x317/0x35e<o:p></o:p></p>
<p class="MsoNormal">[ 858.573148] [<ffffffff81029aab>] do_page_fault+0x9/0xb<o:p></o:p></p>
<p class="MsoNormal">[ 858.573216] [<ffffffff816e4e45>] page_fault+0x25/0x30<o:p></o:p></p>
<p class="MsoNormal">[ 858.573284]<o:p></o:p></p>
<p class="MsoNormal">-> #2 (&mm->mmap_sem){++++++}:<o:p></o:p></p>
<p class="MsoNormal">[ 858.573361] [<ffffffff810c761c>] lock_acquire+0xbf/0xfe<o:p></o:p></p>
<p class="MsoNormal">[ 858.573430] [<ffffffff81130bc9>] might_fault+0x62/0x85<o:p></o:p></p>
<p class="MsoNormal">[ 858.573498] [<ffffffff8163e41b>] skb_do_copy_data_nocache+0x155/0x168<o:p></o:p></p>
<p class="MsoNormal">[ 858.573578] [<ffffffff8163f72a>] tcp_sendmsg+0x398/0x7e1<o:p></o:p></p>
<p class="MsoNormal">[ 858.573648] [<ffffffff8165ee8b>] inet_sendmsg+0x5e/0x67<o:p></o:p></p>
<p class="MsoNormal">[ 858.573718] [<ffffffff815f76d3>] sock_aio_write+0x101/0x111<o:p></o:p></p>
<p class="MsoNormal">[ 858.573790] [<ffffffff81157544>] do_sync_write+0x5d/0x89<o:p></o:p></p>
<p class="MsoNormal">[ 858.573860] [<ffffffff81158035>] vfs_write+0x105/0x14c<o:p></o:p></p>
<p class="MsoNormal">[ 858.573929] [<ffffffff8115832d>] SyS_write+0x57/0x83<o:p></o:p></p>
<p class="MsoNormal">[ 858.573996] [<ffffffff816e5424>] system_call_fastpath+0x1c/0x21<o:p></o:p></p>
<p class="MsoNormal">[ 858.574072]<o:p></o:p></p>
<p class="MsoNormal">-> #1 (sk_lock-AF_INET){+.+.+.}:<o:p></o:p></p>
<p class="MsoNormal">[ 858.574150] [<ffffffff810c761c>] lock_acquire+0xbf/0xfe<o:p></o:p></p>
<p class="MsoNormal">[ 858.574219] [<ffffffff815fc22e>] lock_sock_nested+0x69/0x79<o:p></o:p></p>
<p class="MsoNormal">[ 858.574291] [<ffffffff8163d8bc>] lock_sock+0xb/0xd<o:p></o:p></p>
<p class="MsoNormal">[ 858.574356] [<ffffffff8163efe4>] tcp_sendpage+0x55/0x3da<o:p></o:p></p>
<p class="MsoNormal">[ 858.574426] [<ffffffff8165ef7a>] inet_sendpage+0x82/0x97<o:p></o:p></p>
<p class="MsoNormal">[ 858.574496] [<ffffffff81354b02>] o2net_sendpage+0xae/0x21b<o:p></o:p></p>
<p class="MsoNormal">[ 858.574568] [<ffffffff81356d71>] o2net_accept_many+0x383/0x40a<o:p></o:p></p>
<p class="MsoNormal">[ 858.574642] [<ffffffff81099dc6>] process_one_work+0x1e9/0x358<o:p></o:p></p>
<p class="MsoNormal">[ 858.574716] [<ffffffff8109a31f>] worker_thread+0x159/0x1ee<o:p></o:p></p>
<p class="MsoNormal">[ 858.574787] [<ffffffff810a02b1>] kthread+0xac/0xb4<o:p></o:p></p>
<p class="MsoNormal">[ 858.574853] [<ffffffff816e5372>] ret_from_fork+0x72/0xa0<o:p></o:p></p>
<p class="MsoNormal">[ 858.574924]<o:p></o:p></p>
<p class="MsoNormal">-> #0 (&sc->sc_send_lock){+.+...}:<o:p></o:p></p>
<p class="MsoNormal">[ 858.575003] [<ffffffff810c6e3c>] __lock_acquire+0xb65/0xe65<o:p></o:p></p>
<p class="MsoNormal">[ 858.575077] [<ffffffff810c761c>] lock_acquire+0xbf/0xfe<o:p></o:p></p>
<p class="MsoNormal">[ 858.575146] [<ffffffff816e1732>] mutex_lock_nested+0x3f/0x353<o:p></o:p></p>
<p class="MsoNormal">[ 858.575219] [<ffffffff81355567>] o2net_send_message_vec+0x543/0x8dd<o:p></o:p></p>
<p class="MsoNormal">[ 858.575296] [<ffffffff81355925>] o2net_send_message+0x24/0x26<o:p></o:p></p>
<p class="MsoNormal">[ 858.575369] [<ffffffff81369390>] dlm_do_master_request+0xca/0x630<o:p></o:p></p>
<p class="MsoNormal">[ 858.575446] [<ffffffff8136c727>] dlm_get_lock_resource+0xa58/0xe1d<o:p></o:p></p>
<p class="MsoNormal">[ 858.575523] [<ffffffff813747d7>] dlmlock+0x865/0x11d3<o:p></o:p></p>
<p class="MsoNormal">[ 858.575593] [<ffffffff81349ad8>] o2cb_dlm_lock+0x59/0x69<o:p></o:p></p>
<p class="MsoNormal">[ 858.575665] [<ffffffff81349029>] ocfs2_dlm_lock+0x28/0x2a<o:p></o:p></p>
<p class="MsoNormal">[ 858.575736] [<ffffffff812ecef9>] ocfs2_lock_create+0xfa/0x270<o:p></o:p></p>
<p class="MsoNormal">[ 858.575811] [<ffffffff812ed68d>] ocfs2_create_new_lock+0x7a/0x89<o:p></o:p></p>
<p class="MsoNormal">[ 858.575887] [<ffffffff812ed81d>] ocfs2_create_new_inode_locks+0x181/0x2ae<o:p></o:p></p>
<p class="MsoNormal">[ 858.575988] [<ffffffff813068fc>] __ocfs2_mknod_locked.isra.21+0x3b1/0x4d4<o:p></o:p></p>
<p class="MsoNormal">[ 858.576088] [<ffffffff81306b39>] ocfs2_mknod_locked.isra.22+0x11a/0x12d<o:p></o:p></p>
<p class="MsoNormal">[ 858.576186] [<ffffffff813072d2>] ocfs2_mknod+0x786/0xe09<o:p></o:p></p>
<p class="MsoNormal">[ 858.576257] [<ffffffff813079d2>] ocfs2_mkdir+0x7d/0x11f<o:p></o:p></p>
<p class="MsoNormal">[ 858.576326] [<ffffffff81162ba7>] vfs_mkdir+0x5d/0x9e<o:p></o:p></p>
<p class="MsoNormal">[ 858.576393] [<ffffffff811650f9>] SyS_mkdirat+0x8f/0xe2<o:p></o:p></p>
<p class="MsoNormal">[ 858.576462] [<ffffffff81165160>] SyS_mkdir+0x14/0x16<o:p></o:p></p>
<p class="MsoNormal">[ 858.576528] [<ffffffff816e5424>] system_call_fastpath+0x1c/0x21<o:p></o:p></p>
<p class="MsoNormal">[ 858.576604]<o:p></o:p></p>
<p class="MsoNormal">other info that might help us debug this:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[ 858.576721] Chain exists of:<o:p></o:p></p>
<p class="MsoNormal"> &sc->sc_send_lock --> sb_pagefaults --> jbd2_handle<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[ 858.576847] Possible unsafe locking scenario:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[ 858.576931] CPU0 CPU1<o:p></o:p></p>
<p class="MsoNormal">[ 858.576986] ---- ----<o:p></o:p></p>
<p class="MsoNormal">[ 858.577042] lock(jbd2_handle);<o:p></o:p></p>
<p class="MsoNormal">[ 858.577090] lock(sb_pagefaults);<o:p></o:p></p>
<p class="MsoNormal">[ 858.577161] lock(jbd2_handle);<o:p></o:p></p>
<p class="MsoNormal">[ 858.577229] lock(&sc->sc_send_lock);<o:p></o:p></p>
<p class="MsoNormal">[ 858.577282]<o:p></o:p></p>
<p class="MsoNormal">*** DEADLOCK ***<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[ 858.577383] 11 locks held by ruby/3710:<o:p></o:p></p>
<p class="MsoNormal">[ 858.577432] #0: (sb_writers#11){.+.+.+}, at: [<ffffffff811714f9>] mnt_want_write+0x1f/0x46<o:p></o:p></p>
<p class="MsoNormal">[ 858.577551] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff8116279e>] kern_path_create+0x7f/0x14a<o:p></o:p></p>
<p class="MsoNormal">[ 858.577681] #2: (Meta){++++++}, at: [<ffffffff81306c68>] ocfs2_mknod+0x11c/0xe09<o:p></o:p></p>
<p class="MsoNormal">[ 858.577789] #3: (&ocfs2_sysfile_lock_key[args->fi_sysfile_type]#3){+.+.+.}, at: [<ffffffff8131e853>] ocfs2_reserve_suballoc_bits+0xc3/0x12ce<o:p></o:p></p>
<p class="MsoNormal">[ 858.577942] #4: (inode_alloc:%04d){+.+.+.}, at: [<ffffffff8131e869>] ocfs2_reserve_suballoc_bits+0xd9/0x12ce<o:p></o:p></p>
<p class="MsoNormal">[ 858.578071] #5: (&ocfs2_sysfile_lock_key[args->fi_sysfile_type]#4){+.+.+.}, at: [<ffffffff8131e853>] ocfs2_reserve_suballoc_bits+0xc3/0x12ce<o:p></o:p></p>
<p class="MsoNormal">[ 858.578225] #6: (extent_alloc:%04d){+.+.+.}, at: [<ffffffff8131e869>] ocfs2_reserve_suballoc_bits+0xd9/0x12ce<o:p></o:p></p>
<p class="MsoNormal">[ 858.578354] #7: (&ocfs2_sysfile_lock_key[args->fi_sysfile_type]#5){+.+...}, at: [<ffffffff8130162c>] ocfs2_reserve_local_alloc_bits+0xad/0xc93<o:p></o:p></p>
<p class="MsoNormal">[ 858.578509] #8: (sb_internal#2){.+.+..}, at: [<ffffffff812fcb30>] ocfs2_start_trans+0xa7/0x1be<o:p></o:p></p>
<p class="MsoNormal">[ 858.578632] #9: (&journal->j_trans_barrier){.+.+..}, at: [<ffffffff812fcb40>] ocfs2_start_trans+0xb7/0x1be<o:p></o:p></p>
<p class="MsoNormal">[ 858.578760] #10: (jbd2_handle){+.+...}, at: [<ffffffff8123986e>] start_this_handle+0x4e8/0x57d<o:p></o:p></p>
<p class="MsoNormal">[ 858.578880]<o:p></o:p></p>
<p class="MsoNormal">stack backtrace:<o:p></o:p></p>
<p class="MsoNormal">[ 858.578952] CPU: 3 PID: 3710 Comm: ruby Not tainted 3.11.2-hardenedcanvas-2013-12-16 #4<o:p></o:p></p>
<p class="MsoNormal">[ 858.579054] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/09/2012<o:p></o:p></p>
<p class="MsoNormal">[ 858.579178] ffffffff85305fa0 ffff880516e27348 ffffffff816dd690 0000000000000006<o:p></o:p></p>
<p class="MsoNormal">[ 858.579283] ffffffff8531ace0 ffff880516e27398 ffffffff816d9328 0000000000000002<o:p></o:p></p>
<p class="MsoNormal">[ 858.579387] ffffffff856d9850 ffff8804ffa8a190 ffff8804ffa89600 ffff8804ffa8a190<o:p></o:p></p>
<p class="MsoNormal">[ 858.579492] Call Trace:<o:p></o:p></p>
<p class="MsoNormal">[ 858.580924] [<ffffffff816dd690>] dump_stack+0x46/0x58<o:p></o:p></p>
<p class="MsoNormal">[ 858.580994] [<ffffffff816d9328>] print_circular_bug+0x1f8/0x209<o:p></o:p></p>
<p class="MsoNormal">[ 858.581071] [<ffffffff810c6e3c>] __lock_acquire+0xb65/0xe65<o:p></o:p></p>
<p class="MsoNormal">[ 858.581145] [<ffffffff810c60e4>] ? mark_lock+0x2d/0x220<o:p></o:p></p>
<p class="MsoNormal">[ 858.581215] [<ffffffff81355567>] ? o2net_send_message_vec+0x543/0x8dd<o:p></o:p></p>
<p class="MsoNormal">[ 858.581296] [<ffffffff810c761c>] lock_acquire+0xbf/0xfe<o:p></o:p></p>
<p class="MsoNormal">[ 858.581366] [<ffffffff81355567>] ? o2net_send_message_vec+0x543/0x8dd<o:p></o:p></p>
<p class="MsoNormal">[ 858.581447] [<ffffffff810c7c21>] ? debug_check_no_locks_freed+0x132/0x147<o:p></o:p></p>
<p class="MsoNormal">[ 858.581531] [<ffffffff81355567>] ? o2net_send_message_vec+0x543/0x8dd<o:p></o:p></p>
<p class="MsoNormal">[ 858.581612] [<ffffffff816e1732>] mutex_lock_nested+0x3f/0x353<o:p></o:p></p>
<p class="MsoNormal">[ 858.581687] [<ffffffff81355567>] ? o2net_send_message_vec+0x543/0x8dd<o:p></o:p></p>
<p class="MsoNormal">[ 858.581770] [<ffffffff810c45bd>] ? lockdep_init_map+0xc2/0x147<o:p></o:p></p>
<p class="MsoNormal">[ 858.581848] [<ffffffff81009715>] ? paravirt_read_tsc+0x9/0xd<o:p></o:p></p>
<p class="MsoNormal">[ 858.581922] [<ffffffff810097bc>] ? read_tsc+0x9/0x19<o:p></o:p></p>
<p class="MsoNormal">[ 858.581991] [<ffffffff810bac88>] ? timekeeping_get_ns.constprop.8+0x13/0x3a<o:p></o:p></p>
<p class="MsoNormal">[ 858.582077] [<ffffffff81355567>] o2net_send_message_vec+0x543/0x8dd<o:p></o:p></p>
<p class="MsoNormal">[ 858.582163] [<ffffffff810c60e4>] ? mark_lock+0x2d/0x220<o:p></o:p></p>
<p class="MsoNormal">[ 858.582233] [<ffffffff810c792d>] ? mark_held_locks+0x6d/0x95<o:p></o:p></p>
<p class="MsoNormal">[ 858.582309] [<ffffffff810f2470>] ? rcu_irq_exit+0x7c/0x90<o:p></o:p></p>
<p class="MsoNormal">[ 858.582381] [<ffffffff81355925>] o2net_send_message+0x24/0x26<o:p></o:p></p>
<p class="MsoNormal">[ 858.582457] [<ffffffff81369390>] dlm_do_master_request+0xca/0x630<o:p></o:p></p>
<p class="MsoNormal">[ 858.582536] [<ffffffff81009a93>] ? sched_clock+0x9/0xd<o:p></o:p></p>
<p class="MsoNormal">[ 858.582606] [<ffffffff8136c727>] dlm_get_lock_resource+0xa58/0xe1d<o:p></o:p></p>
<p class="MsoNormal">[ 858.582684] [<ffffffff810c7539>] ? lock_release+0x1ad/0x1d1<o:p></o:p></p>
<p class="MsoNormal">[ 858.582758] [<ffffffff813d0718>] ? do_raw_spin_unlock+0x8f/0x98<o:p></o:p></p>
<p class="MsoNormal">[ 858.582835] [<ffffffff81373ee0>] ? dlm_new_lock+0xbc/0x14e<o:p></o:p></p>
<p class="MsoNormal">[ 858.582908] [<ffffffff813495fc>] ? ocfs2_cluster_connect_agnostic+0x3b/0x3b<o:p></o:p></p>
<p class="MsoNormal">[ 858.582993] [<ffffffff813747d7>] dlmlock+0x865/0x11d3<o:p></o:p></p>
<p class="MsoNormal">[ 858.583062] [<ffffffff8134960d>] ? o2dlm_lock_ast_wrapper+0x11/0x11<o:p></o:p></p>
<p class="MsoNormal">[ 858.583142] [<ffffffff810c60e4>] ? mark_lock+0x2d/0x220<o:p></o:p></p>
<p class="MsoNormal">[ 858.583211] [<ffffffff81349ad8>] o2cb_dlm_lock+0x59/0x69<o:p></o:p></p>
<p class="MsoNormal">[ 858.583282] [<ffffffff813495fc>] ? ocfs2_cluster_connect_agnostic+0x3b/0x3b<o:p></o:p></p>
<p class="MsoNormal">[ 858.583369] [<ffffffff8134960d>] ? o2dlm_lock_ast_wrapper+0x11/0x11<o:p></o:p></p>
<p class="MsoNormal">[ 858.583448] [<ffffffff810c7aa9>] ? trace_hardirqs_on_caller+0x154/0x18b<o:p></o:p></p>
<p class="MsoNormal">[ 858.583531] [<ffffffff81349029>] ocfs2_dlm_lock+0x28/0x2a<o:p></o:p></p>
<p class="MsoNormal">[ 858.583603] [<ffffffff812ecef9>] ocfs2_lock_create+0xfa/0x270<o:p></o:p></p>
<p class="MsoNormal">[ 858.583679] [<ffffffff816e47be>] ? _raw_spin_unlock_irqrestore+0x40/0x4d<o:p></o:p></p>
<p class="MsoNormal">[ 858.583763] [<ffffffff810c7aa9>] ? trace_hardirqs_on_caller+0x154/0x18b<o:p></o:p></p>
<p class="MsoNormal">[ 858.583846] [<ffffffff812ed68d>] ocfs2_create_new_lock+0x7a/0x89<o:p></o:p></p>
<p class="MsoNormal">[ 858.583924] [<ffffffff812ed81d>] ocfs2_create_new_inode_locks+0x181/0x2ae<o:p></o:p></p>
<p class="MsoNormal">[ 858.584010] [<ffffffff813068fc>] __ocfs2_mknod_locked.isra.21+0x3b1/0x4d4<o:p></o:p></p>
<p class="MsoNormal">[ 858.584095] [<ffffffff81306b39>] ocfs2_mknod_locked.isra.22+0x11a/0x12d<o:p></o:p></p>
<p class="MsoNormal">[ 858.584178] [<ffffffff813072d2>] ocfs2_mknod+0x786/0xe09<o:p></o:p></p>
<p class="MsoNormal">[ 858.584249] [<ffffffff813079d2>] ocfs2_mkdir+0x7d/0x11f<o:p></o:p></p>
<p class="MsoNormal">[ 858.584319] [<ffffffff81162ba7>] vfs_mkdir+0x5d/0x9e<o:p></o:p></p>
<p class="MsoNormal">[ 858.584386] [<ffffffff811650f9>] SyS_mkdirat+0x8f/0xe2<o:p></o:p></p>
<p class="MsoNormal">[ 858.584456] [<ffffffff81165160>] SyS_mkdir+0x14/0x16<o:p></o:p></p>
<p class="MsoNormal">[ 858.584524] [<ffffffff816e5424>] system_call_fastpath+0x1c/0x21<o:p></o:p></p>
</div>
</body>
</html>