<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:278340782;
mso-list-type:hybrid;
mso-list-template-ids:247243030 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:293557981;
mso-list-type:hybrid;
mso-list-template-ids:-48832030 1519126140 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0D8;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:Calibri;
color:#1F497D;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='color:#1F497D'>If memory serves me correctly, shared VMDK for clustering only works if both cluster nodes sit on the same ESX server. This pretty much defeats the purpose of clustering because your ESX server becomes a single point of failure. ESX servers crash and fail sometimes… Not frequently, but they do. <o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>When setting up RDM make sure you configure it for physical mode (pass thru). If you use virtual mode, your nodes will once again be limited to a single ESX server and your max RDM size will be limited to ~2TB.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Best, <o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Michael <o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Nathan Patwardhan [mailto:npatwardhan@llbean.com] <br><b>Sent:</b> Friday, April 27, 2012 11:32 AM<br><b>To:</b> Kushnir, Michael (NIH/NLM/LHC) [C]; ocfs2-users@oss.oracle.com<br><b>Subject:</b> RE: RHEL 5.8, ocfs2 v1.4.4, stability issues<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Kushnir, Michael (NIH/NLM/LHC) [C] <a href="mailto:[mailto:michael.kushnir@nih.gov]">[mailto:michael.kushnir@nih.gov]</a> <br><b>Sent:</b> Wednesday, April 25, 2012 12:52 PM<br><b>To:</b> Nathan Patwardhan; <a href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</a><br><b>Subject:</b> RE: RHEL 5.8, ocfs2 v1.4.4, stability issues</span><o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings;color:#1F497D'><span style='mso-list:Ignore'>Ø<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Hi Nathan,</span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings;color:#1F497D'><span style='mso-list:Ignore'>Ø<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>I’ve no particular insight on splunk…</span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings;color:#1F497D'><span style='mso-list:Ignore'>Ø<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>We use OCFS 1.4 with RHEL 5.8 VMs on ESX 4.1 and ESXi 5 as storage for a Hadoop cluster as well as an index and document store for application servers running a search engine application (java, tomcat, apache). We also export the same volumes from a few cluster nodes over NFS for developer access. We’ve had no stability issues or corruption. </span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'>I think I’ve identified the issue.</span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'><a href="http://communities.vmware.com/thread/164967">http://communities.vmware.com/thread/164967</a></span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'>In short, we had been allocated a shared vmdk through shared bus rather than as an RDM. As a result, and after some digging into the logs, I found a bunch of SCSI reservation conflicts. I’ll get setup with RDM within the next week or so and will update with results as I have them.</span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='color:#1F497D'> </span><o:p></o:p></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Nathan Patwardhan <a href="mailto:[mailto:npatwardhan@llbean.com]">[mailto:npatwardhan@llbean.com]</a> <br><b>Sent:</b> Tuesday, April 24, 2012 9:44 AM<br><b>To:</b> <a href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</a><br><b>Subject:</b> [Ocfs2-users] RHEL 5.8, ocfs2 v1.4.4, stability issues</span><o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Hi everyone,<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>We’re running a two-node cluster w/ocfs2 v.1.4.4 under RHEL 5.8 and we’re having major stability issues:<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>[root@splwww02 ~]# lsb_release -a ; uname -a ; rpm -qa |grep -i ocfs<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>LSB Version: :core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Distributor ID: RedHatEnterpriseServer<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Description: Red Hat Enterprise Linux Server release 5.8 (Tikanga)<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Release: 5.8<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Codename: Tikanga<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Linux splwww02.llbean.com 2.6.18-308.1.1.el5 #1 SMP Fri Feb 17 16:51:01 EST 2012 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>ocfs2-2.6.18-308.1.1.el5-1.4.7-1.el5<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>ocfs2-tools-1.4.4-1.el5<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>ocfs2console-1.4.4-1.el5<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>These rpms were downloaded and installed from oracle.com.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>The node are ESX VMs and the storage is shared to both VMs through vSphere.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>We’re using ocfs2 because we needed shared back end storage for a splunk implementation (splunk web search heads can used pooled storage and we didn’t want to just use NFS for the obvious reasons – where shared storage seemed to be a better solution all the way around).<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>/etc/ocfs2/cluster.conf contains the following on each node:<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>cluster:<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> node_count = 2<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> name = splwww<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>node:<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> ip_port = 7777<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> ip_address = 10.130.245.40<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> number = 0<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> name = splwww01<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> cluster = splwww<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>node:<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> ip_port = 7777<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> ip_address = 10.130.245.41<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> number = 1<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> name = splwww02<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> cluster = splwww<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Note that each node uses eth0 for clustering. We do NOT currently have a private network setup in ESX for this purpose but CAN if this is recommended by people here.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>o2cb status on both nodes:<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Driver for "configfs": Loaded<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Filesystem "configfs": Mounted<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Driver for "ocfs2_dlmfs": Loaded<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Filesystem "ocfs2_dlmfs": Mounted<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Checking O2CB cluster splwww: Online<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Heartbeat dead threshold = 3<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> Network idle timeout: 90000<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> Network keepalive delay: 5000<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> Network reconnect delay: 5000<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Checking O2CB heartbeat: Active<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>/etc/sysconfig/o2cb:<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>O2CB_ENABLED=true<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>O2CB_STACK=o2cb<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>O2CB_BOOTCLUSTER=splwww<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>O2CB_HEARTBEAT_THRESHOLD=3<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>O2CB_IDLE_TIMEOUT_MS=90000<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>O2CB_KEEPALIVE_DELAY_MS=5000<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>O2CB_RECONNECT_DELAY_MS=5000<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Each host mounts the (ocfs2) file system then turns around and exports the file system where it mounts it on localhost as shown below (yes I know this is awful but splunk does not support ocfs2 and as such is unable to lock files properly no matter what options I give to mount an ocfs2 file system):<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'># >From /etc/fstab<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>/dev/mapper/splwwwshared-splwwwshared--lv /opt/splunk/pooled ocfs2 _netdev,datavolume,errors=remount-ro,rw,noatime 0 0<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'># From /etc/exports<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>/opt/splunk/pooled 127.0.0.1(rw,no_root_squash)<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'># From /bin/mount<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>/dev/mapper/splwwwshared-splwwwshared--lv on /opt/splunk/pooled type ocfs2 (rw,_netdev,noatime,datavolume,errors=remount-ro,heartbeat=local)<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>localhost:/opt/splunk/pooled on /opt/splunk/mnt type nfs (rw,nordirplus,addr=127.0.0.1)<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>The above should supposedly allow us to enjoy the benefits of an ocfs2 shared/clustered file system while functionally supporting what splunk needs. Unfortunately we have experienced a number of problems with stability, summarized here:<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo4'><![if !supportLists]><span style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Cluster timeouts, node(s) leaving cluster.<o:p></o:p></p><p class=MsoListParagraph> <o:p></o:p></p><p class=MsoListParagraph>Apr 21 21:03:50 splwww01 kernel: (o2hb-DA9199FC9F,3090,0):o2hb_do_disk_heartbeat:768 ERROR: status = -52<o:p></o:p></p><p class=MsoListParagraph>Apr 21 21:03:52 splwww01 kernel: (o2hb-DA9199FC9F,3090,0):o2hb_do_disk_heartbeat:777 ERROR: Device "dm-5": another node is heartbeating in our slot!<o:p></o:p></p><p class=MsoListParagraph> <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo4'><![if !supportLists]><span style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Ocfs2 crashes, nfs mount becoming unavailable, /opt/splunk/pooled becoming inaccessible, splunk crashing.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Apr 24 06:53:43 splwww01 kernel: (o2hb-DA9199FC9F,3089,0):ocfs2_dlm_eviction_cb:98 device (253,5): dlm has evicted node 1<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo4'><![if !supportLists]><span style='mso-list:Ignore'>3.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>inode irregularities on either/both node(s).<o:p></o:p></p><p class=MsoListParagraph> <o:p></o:p></p><p class=MsoListParagraph>Apr 24 06:40:47 splwww01 kernel: (nfsd,3330,1):ocfs2_read_locked_inode:472 ERROR: status = -5<o:p></o:p></p><p class=MsoListParagraph> <o:p></o:p></p><p class=MsoListParagraph>Apr 21 10:09:31 splwww01 kernel: (ocfs2rec,7676,0):ocfs2_clear_journal_error:690 ERROR: File system error -5 recorded in journal 0.<o:p></o:p></p><p class=MsoListParagraph>Apr 21 10:09:31 splwww01 kernel: (ocfs2rec,7676,0):ocfs2_clear_journal_error:692 ERROR: File system on device dm-5 needs checking.<o:p></o:p></p><p class=MsoListParagraph> <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo4'><![if !supportLists]><span style='mso-list:Ignore'>4.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Full system crashes on both node(s).<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Ideally, I’d love to continue using ocfs2 because it solves an age-old problem for us, but since we can’t keep either system stable for more than an hour or so, I’m looking for some more insight as to what we’re seeing and how we might resolve some of these issues.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Note also that I’m using monit to check a .state file I created under /opt/splunk/pool and re-mount all the fs, restart splunk, etc, but this only takes us so far once the system(s) crash and reboot.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>--<o:p></o:p></p><p class=MsoNormal>Nathan Patwardhan, Sr. System Engineer<o:p></o:p></p><p class=MsoNormal><a href="mailto:npatwardhan@llbean.com">npatwardhan@llbean.com</a><o:p></o:p></p><p class=MsoNormal>x26662<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p></div></div></body></html>