<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v =
"urn:schemas-microsoft-com:vml" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:m =
"http://schemas.microsoft.com/office/2004/12/omml"><HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2800.1593" name=GENERATOR><!--[if !mso]>
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<![endif]-->
<STYLE>@font-face {
font-family: Cambria Math;
}
@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@page Section1 {size: 612.0pt 792.0pt; margin: 70.85pt 70.85pt 70.85pt 70.85pt; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0cm 0cm 0pt; FONT-FAMILY: "Times New Roman","serif"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0cm 0cm 0pt; FONT-FAMILY: "Times New Roman","serif"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0cm 0cm 0pt; FONT-FAMILY: "Times New Roman","serif"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
P {
FONT-SIZE: 12pt; MARGIN-LEFT: 0cm; MARGIN-RIGHT: 0cm; FONT-FAMILY: "Times New Roman","serif"; mso-style-priority: 99; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto
}
SPAN.EmailStyle18 {
COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.Section1 {
page: Section1
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></HEAD>
<BODY lang=SV vLink=purple link=blue>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>Well,
I'm not 100% sure I solved the problem in a definitve way, but here's
the complete story:</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>1 -
install, if you can, the latest release of ocfs2 + tools. The fact that a node
reboots instead of panicking (and resting in peace until manual intervention) is
a real life saver if you do not have immediate access to the server farm. Plus
timeouts are configurable.</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>2 -
when a cluster node is rebooted by the ocfs daemon, a telltale message is
present on the console of the node. Messages from the ocfs daemon will also be
present in /var/log/messages on the other nodes, but looking at those it is hard
to understand if the dying node was shutdown by ocfs or by other
causes.</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>You
can either sit in front of the screen or start the netdump service on the
rebooting node and the netdump-server service on a spare machine (another node
on the cluster is fine. For best results use a different nic interconnect from
the one used by ocfs.) If you are using red-hat the man pages for both services
are quite straightforward</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>3 - in
our case, the log we netdumped said:</FONT></SPAN></DIV><SPAN
class=312070809-11052007><PRE>(6,0):o2hb_write_timeout:269 ERROR: Heartbeat write timeout to device emcpowere2 after 12000 milliseconds</PRE><PRE>Heartbeat thread (6) printing last 24 blocking operations (cur = 7):</PRE><PRE>Heartbeat thread stuck at waiting for read completion, stuffing current time into that blocker (index 7)</PRE><PRE>Index 8: took 0 ms to do submit_bio for read</PRE><PRE>[ ... ]</PRE><PRE>Index 7: took 9998 ms to do waiting for read completion</PRE><PRE>*** ocfs2 is very sorry to be fencing this system by restarting ***</PRE>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>4
- thus we determined ocfs2 was indeed at fault. Operations on other
files where ok, but using rman to create a single 1,3 GB file on the ocfs disk
was somehow triggering an heartbeat timeout.</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>5 - we
modified the configuration of our rman scripts to try to keep the size of the
files created smaller. We tested again, and there was no reboot. I am not sure
you can achieve the same result for failovers though - the general idea is to
keep io in smaller chunks (or slow it down somehow?)</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>6- As
Sunil recommended (sorry, I think this was off list), we also raised the ocfs
timeout value for O2CB_HEARTBEAT_THRESHOLD. Precise instructions for that can be
found here: <A
href="http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_faq.html#TIMEOUT">http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_faq.html#TIMEOUT</A>.
We decided to go with a value of 31. We did not raise timeouts for the network
keepalives (yet), since we are not using bonded nics for the ocfs2 interconnect.
We might do that in the future if we find out that traffic on that network is
extremely high / the network unstable, though...</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>Hope
it helps</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2>Gaetano</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV></SPAN>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B> Mattias Segerdahl
[mailto:mattias.segerdahl@mandator.com]<BR><B>Sent:</B> Friday, May 11, 2007
10:00 AM<BR><B>To:</B> Gaetano Giunta<BR><B>Subject:</B> RE: [Ocfs2-users] PBL
with RMAN and ocfs2<BR><BR></FONT></DIV>
<DIV class=Section1>
<P class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">Hi,<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">We’re
having the exact same problem, if we do a failover between two filers/san’s,
the server reboots.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">So
far I haven’t found a solution to the problem, would you mind trying to
explain how you solved the problem, step by step?<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">Best
Regards,<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">Mattias
Segerdahl<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></P>
<DIV
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0cm; BORDER-TOP: medium none; PADDING-LEFT: 4pt; PADDING-BOTTOM: 0cm; BORDER-LEFT: blue 1.5pt solid; PADDING-TOP: 0cm; BORDER-BOTTOM: medium none">
<DIV>
<DIV
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0cm; BORDER-TOP: #b5c4df 1pt solid; PADDING-LEFT: 0cm; PADDING-BOTTOM: 0cm; BORDER-LEFT: medium none; PADDING-TOP: 3pt; BORDER-BOTTOM: medium none">
<P class=MsoNormal><B><SPAN lang=EN-US
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Tahoma','sans-serif'">From:</SPAN></B><SPAN
lang=EN-US style="FONT-SIZE: 10pt; FONT-FAMILY: 'Tahoma','sans-serif'">
ocfs2-users-bounces@oss.oracle.com [mailto:ocfs2-users-bounces@oss.oracle.com]
<B>On Behalf Of </B>Gaetano Giunta<BR><B>Sent:</B> den 11 maj 2007
09:47<BR><B>To:</B> Ocfs2-users@oss.oracle.com<BR><B>Subject:</B> RE:
[Ocfs2-users] PBL with RMAN and ocfs2<o:p></o:p></SPAN></P></DIV></DIV>
<P class=MsoNormal><o:p> </o:p></P>
<DIV>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">Thanks,
but I had alreday checked out all logs I could find (oracle and crs alerts,
/var/log stuff) and there was no clear indication in
there.</SPAN><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal> <o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">The
trick is the ocfs was sending the alert message to the console only (I wonder
why it does not also leva traces into syslog, my best guess is it tries to
shutdown as fast as it can, and sending a message to console is faster than
sending it to syslog - but I'm in no way a linux
guru...).</SPAN><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal> <o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">By using the
netdump tool suggested by Sunil I managed to see the console messages of the
dying node (without having to phisycally be in the server farm, which is 40 km
away from my ususal workplace), and diagnosed the ocfs2 heartbeat as "the
killer".</SPAN><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal> <o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">Bye</SPAN><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">Gaetano</SPAN><o:p></o:p></P></DIV>
<BLOCKQUOTE
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0cm; BORDER-TOP: medium none; MARGIN-TOP: 5pt; PADDING-LEFT: 4pt; MARGIN-BOTTOM: 5pt; PADDING-BOTTOM: 0cm; MARGIN-LEFT: 3.75pt; BORDER-LEFT: blue 1.5pt solid; PADDING-TOP: 0cm; BORDER-BOTTOM: medium none">
<P class=MsoNormal style="MARGIN-BOTTOM: 12pt"><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Tahoma','sans-serif'">-----Original
Message-----<BR><B>From:</B> Luis Freitas
[mailto:lfreitas34@yahoo.com]<BR><B>Sent:</B> Thursday, May 10, 2007 11:17
PM<BR><B>To:</B> Gaetano Giunta<BR><B>Cc:</B>
Ocfs2-users@oss.oracle.com<BR><B>Subject:</B> Re: [Ocfs2-users] PBL with
RMAN and ocfs2</SPAN><o:p></o:p></P>
<DIV>
<P class=MsoNormal>Gaetano,<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal> <o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal> If o2cb or CRS is killing the
machine, it usually shows on /var/log/messages with lines explaining
what happened. Take a look on the /var/log/messages just before the last
"syslogd x.x.x: restart".<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal> <o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal>Regards,<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal>Luis<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><BR><BR><BR>Gaetano Giunta wrote:<BR>>
Hello.<BR>><BR>> On a 2 node RAC 10.2.0.3 setup, on RH ES 4.4 x86_64,
with ocfs 1.2.5-1, we are experiencing some troubles with RMAN: when the
archive log destination is on an ASM partition, and the backup detsination
is on ocfs2, running<BR>><BR>> backup archivelog all format
'/home/SANstorage/oracle/backup/rman/dump_log/FULL_20070509_154916/arc_%d_%u'
delete input;<BR>><BR>> consistently causes a reboot.<BR>><BR>>
The rman catalog is clean, and has been crosschecked in every
way.<BR>><BR>> We tried on both nodes, and the node executing the
backup always reboots.<BR>> I am thus inclined to think that it is not
the ocfs2 dlm that triggers the reboot, because in that case the victim
would always be the second node.<BR>><BR>> I also tested the same
command using as backup destination /tmp, and all was fine. The backup file
of the archived logs is 1249843712 in size.<BR>><BR>> Our local oracle
guy went through metalink and said there is no open bug/patch for that at
this time.<BR>><BR>> Any suggestions ???<BR>><BR>>
Thanks<BR>> Gaetano Giunta<BR>><BR>> <BR>>
------------------------------------------------------------------------<BR>><BR>>
_______________________________________________<BR>> Ocfs2-users mailing
list<BR>> Ocfs2-users@oss.oracle.com<BR>>
http://oss.oracle.com/mailman/listinfo/ocfs2-users<BR><BR><BR>_______________________________________________<BR>Ocfs2-users
mailing
list<BR>Ocfs2-users@oss.oracle.com<BR>http://oss.oracle.com/mailman/listinfo/ocfs2-users<o:p></o:p></P></DIV>
<P class=MsoNormal><o:p> </o:p></P>
<DIV class=MsoNormal style="TEXT-ALIGN: center" align=center>
<HR align=center width="100%" SIZE=1>
</DIV>
<P class=MsoNormal>Ahhh...imagining that irresistible "new car"
smell?<BR>Check out <A
href="http://us.rd.yahoo.com/evt=48245/*http:/autos.yahoo.com/new_cars.html;_ylc=X3oDMTE1YW1jcXJ2BF9TAzk3MTA3MDc2BHNlYwNtYWlsdGFncwRzbGsDbmV3LWNhcnM-">new
cars at Yahoo! Autos.</A>
<o:p></o:p></P></BLOCKQUOTE></DIV></DIV></BLOCKQUOTE></BODY></HTML>