<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:D="DAV:" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>We have a 2 node OCFS2 cluster running Oracle 10g, both
nodes crashed.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Node 1 because it panic running IOSTAT, the second node crashed
with this error message you can see below. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I was hoping to see a newer version of OCFS2 so I could
proceed with the upgrade if necessary.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Have you seen this problem, anybody has resolved this?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Node 1 reboots<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>reboot system boot
2.6.18-92.el5 Tue Jun 16
09:26 (02:14)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Node 2 reboots<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>reboot system boot
2.6.18-92.el5 Tue Jun 16
09:29 (02:10)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Running Kernel<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Linux uscosprdvrtxdb02 2.6.18-92.el5 #1 SMP Tue Apr 29
13:16:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>OCFS2 Version installed<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ocfs2console-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ocfs2-tools-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ocfs2-2.6.18-92.el5-1.4.1-1.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><b><u><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial;font-weight:bold'>Crash analysis:<o:p></o:p></span></font></u></b></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>KERNEL:
/usr/lib/debug/lib/modules/2.6.18-92.el5/vmlinux<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> DUMPFILE: vmcore
[PARTIAL DUMP]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>
CPUS: 8<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>
DATE: Tue Jun 16 09:15:26 2009<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> UPTIME: 2
days, 02:17:01<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>LOAD AVERAGE: 0.22, 0.31, 0.21<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>
TASKS: 570<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> NODENAME: uscosprdvrtxdb02<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> RELEASE:
2.6.18-92.el5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> VERSION: #1 SMP
Tue Apr 29 13:16:15 EDT 2008<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> MACHINE:
x86_64 (2666 Mhz)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> MEMORY:
11.8 GB<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> </span></font><b><font
size=2 color=red face=Arial><span style='font-size:10.0pt;font-family:Arial;
color:red;font-weight:bold'>PANIC: ""<o:p></o:p></span></font></b></p>
<p class=MsoNormal><b><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red;font-weight:bold'>
PID: 28123<o:p></o:p></span></font></b></p>
<p class=MsoNormal><b><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red;font-weight:bold'>
COMMAND: "oracle"<o:p></o:p></span></font></b></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>
TASK: ffff8102e25e97e0 [THREAD_INFO: ffff8102cf0ba000]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>
CPU: 3<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> </span></font><b><font
size=2 color=red face=Arial><span style='font-size:10.0pt;font-family:Arial;
color:red;font-weight:bold'>STATE: TASK_RUNNING (PANIC)<o:p></o:p></span></font></b></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><b><u><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial;font-weight:bold'><o:p><span style='text-decoration:none'> </span></o:p></span></font></u></b></p>
<p class=MsoNormal><b><u><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial;font-weight:bold'>Kernel messages:<o:p></o:p></span></font></u></b></p>
<p class=MsoNormal><b><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red;font-weight:bold'>o2net: connection to node
uscosprdvrtxdb01 (num 0) at 192.168.5.1:7000 has been idle for 60.0 seconds,
shutting it down.<o:p></o:p></span></font></b></p>
<p class=MsoNormal><b><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red;font-weight:bold'>(0,0):o2net_idle_timer:1476
here are some times that might help debug the situation: (tmr 1245143657.942607
now 1245143717.944198 dr 1245143657.942600 adv 12<o:p></o:p></span></font></b></p>
<p class=MsoNormal><b><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red;font-weight:bold'>45143657.942608:1245143657.942609
func (5010bc9a:505) 1245128670.144972:1245128670.144981)<o:p></o:p></span></font></b></p>
<p class=MsoNormal><b><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red;font-weight:bold'>o2net: no longer connected
to node uscosprdvrtxdb01 (num 0) at 192.168.5.1:7000<o:p></o:p></span></font></b></p>
<p class=MsoNormal><b><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red;font-weight:bold'>(28123,3):dlm_do_master_request:1330
ERROR: unhandled error!----------- [cut here ] --------- [please bite here ]
---------<o:p></o:p></span></font></b></p>
<p class=MsoNormal><b><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red;font-weight:bold'>Kernel BUG at
...mushran/BUILD/ocfs2-1.4.1/fs/ocfs2/dlm/dlmmaster.c:1331<o:p></o:p></span></font></b></p>
<p class=MsoNormal><b><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red;font-weight:bold'>invalid opcode: 0000 [1]
SMP</span></font></b><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>last sysfs file:
/devices/pci0000:00/0000:00:05.0/0000:10:00.0/0000:11:01.0/0000:14:00.0/0000:15:00.0/irq<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>CPU 3<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Modules linked in: nfs lockd fscache
nfs_acl mptctl mptbase ipmi_si(U) ipmi_devintf(U) ipmi_msghandler(U) autofs4
hidp l2cap bluetooth ocfs2(U) ocfs2_dlmfs(U<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>) ocfs2_dlm(U) ocfs2_nodemanager(U)
configfs sunrpc hp_ilo(U) bonding ipv6 xfrm_nalgo crypto_api emcpdm(PU)
emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_mirror dm_mul<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>tipath dm_mod video sbs backlight i2c_ec
i2c_core button battery asus_acpi acpi_memhotplug ac parport_pc lp parport
i5000_edac edac_mc bnx2 sg serio_raw shpc<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>hp pcspkr usb_storage lpfc
scsi_transport_fc cciss(U) sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd<o:p></o:p></span></font></p>
<p class=MsoNormal><b><font size=2 color=red face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:red;font-weight:bold'>Pid: 28123, comm: oracle
Tainted: P 2.6.18-92.el5 #1<o:p></o:p></span></font></b></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>RIP: 0010:[<ffffffff88652f8a>]
[<ffffffff88652f8a>] :ocfs2_dlm:dlm_do_master_request+0x2f1/0x61c<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>RSP: 0018:ffff8102cf0bba38 EFLAGS:
00010286<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>RAX: 000000000000003f RBX:
00000000fffffe00 RCX: ffffffff802ec9a8<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>RDX: ffffffff802ec9a8 RSI:
0000000000000000 RDI: ffffffff802ec9a0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>RBP: ffff8101b98d3e40 R08:
ffffffff802ec9a8 R09: 0000000000000046<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>R10: 0000000000000000 R11:
0000000000000080 R12: 0000000000000000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>R13: ffff810316df5c00 R14:
ffff810316df5c00 R15: ffff8101bc0625c0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>FS: 00002b806dfccc40(0000)
GS:ffff81032ff24640(0000) knlGS:0000000000000000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>CS: 0010 DS: 0000 ES: 0000 CR0:
000000008005003b<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>CR2: 00000000086229b8 CR3:
00000002cf119000 CR4: 00000000000006e0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Process oracle (pid: 28123, threadinfo
ffff8102cf0ba000, task ffff8102e25e97e0)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Stack: 0000000000001f01 3030303030303057
3030303030303030 3435303061323030<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> 0061316437626364 0000000000000000
0000000000000000 0000000000000000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> 0000000000000000 000000008865344a
0000000116df5c00 0000000000000000<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Call Trace:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff88658669>] :ocfs2_dlm:dlm_get_lock_resource+0xa5e/0x1913<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff8005be70>]
cache_alloc_refill+0x106/0x186<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff8865dde5>]
:ocfs2_dlm:dlm_wait_for_recovery+0xa1/0x116<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff88650c46>]
:ocfs2_dlm:dlmlock+0x731/0x11f9<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff886a5ad0>]
:ocfs2:ocfs2_cluster_unlock+0x240/0x2ad<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff80009523>]
__d_lookup+0xb0/0xff<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff886a17d8>]
:ocfs2:ocfs2_dentry_revalidate+0x111/0x259<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff886a69c1>]
:ocfs2:ocfs2_init_mask_waiter+0x24/0x3d<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff8000cb46>]
do_lookup+0x65/0x1d4<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff886a7e00>]
:ocfs2:ocfs2_cluster_lock+0x354/0x7eb<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff886a9a5c>]
:ocfs2:ocfs2_locking_ast+0x0/0x486<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff886acfd2>]
:ocfs2:ocfs2_blocking_ast+0x0/0x2c1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff801458b9>]
snprintf+0x44/0x4c<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff886ac242>]
:ocfs2:ocfs2_rw_lock+0x10f/0x1d6<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff886b0159>]
:ocfs2:ocfs2_file_aio_read+0x128/0x394<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff886a75eb>]
:ocfs2:ocfs2_add_lockres_tracking+0x73/0x81<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff8000caa4>]
do_sync_read+0xc7/0x104<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff886aedcc>]
:ocfs2:ocfs2_init_file_private+0x4d/0x5a<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff8001e35e>] __dentry_open+0x101/0x1dc<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff8009dde2>]
autoremove_wake_function+0x0/0x2e<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff80027338>]
do_filp_open+0x2a/0x38<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff8000b337>]
vfs_read+0xcb/0x171<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff800130a3>]
sys_pread64+0x50/0x70<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff8005d229>]
tracesys+0x71/0xe0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> [<ffffffff8005d28d>]
tracesys+0xd5/0xe0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Code: 0f 0b 68 de 85 66 88 c2 33 05 48 b8
00 09 00 00 01 00 00 00<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>RIP [<ffffffff88652f8a>]
:ocfs2_dlm:dlm_do_master_request+0x2f1/0x61c<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> RSP <ffff8102cf0bba38><o:p></o:p></span></font></p>
<p class=MsoNormal><b><font size=2 color=black face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:black;font-weight:bold'><o:p> </o:p></span></font></b></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color="#121212" face="Times New Roman"><span
style='font-size:12.0pt;color:#121212'><img width=48 height=42 id="_x0000_i1025"
src="cid:image001.jpg@01C9EE3D.2216DFF0"><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color="#121212" face="Times New Roman"><span
style='font-size:12.0pt;color:#121212'><o:p> </o:p></span></font></p>
<p class=MsoNormal><em><b><i><font size=3 color=green face=Verdana><span
style='font-size:12.0pt;font-family:Verdana;color:green;font-weight:bold;
font-style:normal'>Saul J. Gabay</span></font></i></b></em><b><font color=navy
face=Verdana><span style='font-family:Verdana;color:navy;font-weight:bold'><o:p></o:p></span></font></b></p>
<p class=MsoNormal><em><i><font size=2 color=green face=Verdana><span
style='font-size:11.0pt;font-family:Verdana;color:green;font-style:normal'>Sr.
Linux Engineer</span></font></i></em><em><i><font size=2 color=green
face="Times New Roman"><span style='font-size:11.0pt;color:green;font-style:
normal'><o:p></o:p></span></font></i></em></p>
<p class=MsoNormal><em><i><font size=2 color=green face=Verdana><span
style='font-size:10.0pt;font-family:Verdana;color:green;font-style:normal'>IT
Infrastructure & Operations</span></font></i></em><i><font size=2
color=navy><span style='font-size:10.0pt;color:navy;font-style:italic'><o:p></o:p></span></font></i></p>
<p class=MsoNormal><em><i><font size=1 color=green face=Verdana><span
style='font-size:9.0pt;font-family:Verdana;color:green;font-style:normal'>Herbalife
International Inc.</span></font></i></em><i><font size=1 color=navy
face=Verdana><span style='font-size:9.0pt;font-family:Verdana;color:navy;
font-style:italic'><o:p></o:p></span></font></i></p>
<p class=MsoNormal><em><i><font size=1 color=green face=Verdana><span
style='font-size:8.0pt;font-family:Verdana;color:green;font-style:normal'>310-410-9600
x24341</span></font></i><font color=green><span style='color:green'><o:p></o:p></span></font></em></p>
<p class=MsoNormal><em><i><font size=1 color=green face=Verdana><span
style='font-size:8.0pt;font-family:Verdana;color:green;font-style:normal'>saulg@herbalife.com</span></font></i></em><font
color=navy><span style='color:navy'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
</body>
</html>