<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns:o = "urn:schemas-microsoft-com:office:office"><HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META content="MSHTML 6.00.2800.1593" name=GENERATOR></HEAD>
<BODY>
<DIV><SPAN class=427055710-14052007><FONT face=Arial color=#0000ff size=2>I did
not change the scheduler, as the online docs mention that has to be done for RH
versions up to U3, and we're deploying un update 4.</FONT></SPAN></DIV>
<DIV><SPAN class=427055710-14052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=427055710-14052007><FONT face=Arial color=#0000ff size=2>Dunno
if it might help - I guess we could do some benchmarking of db performance to
see if it has any impact...</FONT></SPAN></DIV>
<DIV><SPAN class=427055710-14052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=427055710-14052007><FONT face=Arial color=#0000ff
size=2>Yhanks</FONT></SPAN></DIV>
<DIV><SPAN class=427055710-14052007><FONT face=Arial color=#0000ff
size=2>gaetano</FONT></SPAN></DIV>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid">
<DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B>
ocfs2-users-bounces@oss.oracle.com
[mailto:ocfs2-users-bounces@oss.oracle.com]<B>On Behalf Of </B>Luis
Freitas<BR><B>Sent:</B> Friday, May 11, 2007 9:29 PM<BR><B>To:</B>
Ocfs2-users@oss.oracle.com<BR><B>Subject:</B> RE: [Ocfs2-users] PBL with RMAN
and ocfs2<BR><BR></FONT></DIV>
<DIV>Gaetano,</DIV>
<DIV> </DIV>
<DIV> I am using RMAN with the default
configuration here in RH 4.0, but I had to change the I/O scheduler to the
"deadline" I/O scheduler to prevent these reboots, and increased the o2cb
timeouts too. We had some just after implementing but it seems very stable
now. We increased the timeout here to 130, to account for SAN switch
failures, powerpath and such.</DIV>
<DIV> </DIV>
<DIV> I am still on 1.2.1 on the production nodes
and it panics the machine, which do is annoying even when the servers are on
the same building, but there are always messages on /var/log/messages of the
killed node showing what happened. Funny that 1.2.5 no longer shows
these.</DIV>
<DIV> </DIV>
<DIV>Regards,</DIV>
<DIV>Luis</DIV>
<DIV><BR><B><I>Gaetano Giunta
<giunta.gaetano@sea-aeroportimilano.it></I></B> wrote:</DIV>
<BLOCKQUOTE class=replbq
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid">
<META content="MSHTML 6.00.2800.1593" name=GENERATOR>
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<STYLE>@font-face {
font-family: Cambria Math;
}
@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@page Section1 {size: 612.0pt 792.0pt; margin: 70.85pt 70.85pt 70.85pt 70.85pt; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0cm 0cm 0pt; FONT-FAMILY: "Times New Roman","serif"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0cm 0cm 0pt; FONT-FAMILY: "Times New Roman","serif"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0cm 0cm 0pt; FONT-FAMILY: "Times New Roman","serif"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
P {
FONT-SIZE: 12pt; MARGIN-LEFT: 0cm; MARGIN-RIGHT: 0cm; FONT-FAMILY: "Times New Roman","serif"; mso-style-priority: 99; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto
}
SPAN.EmailStyle18 {
COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.Section1 {
page: Section1
}
</STYLE>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2>Well, I'm not 100% sure I solved the problem in a definitve way, but
here's the complete story:</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>1
- install, if you can, the latest release of ocfs2 + tools. The fact that a
node reboots instead of panicking (and resting in peace until manual
intervention) is a real life saver if you do not have immediate access to
the server farm. Plus timeouts are configurable.</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>2
- when a cluster node is rebooted by the ocfs daemon, a telltale message is
present on the console of the node. Messages from the ocfs daemon will also
be present in /var/log/messages on the other nodes, but looking at those it
is hard to understand if the dying node was shutdown by ocfs or by other
causes.</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2>You can either sit in front of the screen or start the netdump
service on the rebooting node and the netdump-server service on a spare
machine (another node on the cluster is fine. For best results use a
different nic interconnect from the one used by ocfs.) If you are using
red-hat the man pages for both services are quite
straightforward</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>3
- in our case, the log we netdumped said:</FONT></SPAN></DIV><SPAN
class=312070809-11052007><PRE>(6,0):o2hb_write_timeout:269 ERROR: Heartbeat write timeout to device emcpowere2 after 12000 milliseconds</PRE><PRE>Heartbeat thread (6) printing last 24 blocking operations (cur = 7):</PRE><PRE>Heartbeat thread stuck at waiting for read completion, stuffing current time into that blocker (index 7)</PRE><PRE>Index 8: took 0 ms to do submit_bio for read</PRE><PRE>[ ... ]</PRE><PRE>Index 7:
took 9998 ms to do waiting for read completion</PRE><PRE>*** ocfs2 is very sorry to be fencing this system by restarting ***</PRE>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>4
- thus we determined ocfs2 was indeed at fault. Operations on
other files where ok, but using rman to create a single 1,3 GB file on the
ocfs disk was somehow triggering an heartbeat timeout.</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>5
- we modified the configuration of our rman scripts to try to keep the size
of the files created smaller. We tested again, and there was no reboot. I am
not sure you can achieve the same result for failovers though - the general
idea is to keep io in smaller chunks (or slow it down
somehow?)</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff size=2>6-
As Sunil recommended (sorry, I think this was off list), we also raised the
ocfs timeout value for O2CB_HEARTBEAT_THRESHOLD. Precise instructions for
that can be found here: <A
href="http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_faq.html#TIMEOUT">http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_faq.html#TIMEOUT</A>.
We decided to go with a value of 31. We did not raise timeouts for the
network keepalives (yet), since we are not using bonded nics for the ocfs2
interconnect. We might do that in the future if we find out that traffic on
that network is extremely high / the network unstable,
though...</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2>Hope it helps</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2>Gaetano</FONT></SPAN></DIV>
<DIV><SPAN class=312070809-11052007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV></SPAN>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B> Mattias Segerdahl
[mailto:mattias.segerdahl@mandator.com]<BR><B>Sent:</B> Friday, May 11,
2007 10:00 AM<BR><B>To:</B> Gaetano Giunta<BR><B>Subject:</B> RE:
[Ocfs2-users] PBL with RMAN and ocfs2<BR><BR></FONT></DIV>
<DIV class=Section1>
<DIV class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">Hi,<o:p></o:p></SPAN></DIV>
<DIV class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></DIV>
<DIV class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">We're
having the exact same problem, if we do a failover between two
filers/san's, the server reboots.<o:p></o:p></SPAN></DIV>
<DIV class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></DIV>
<DIV class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">So
far I haven't found a solution to the problem, would you mind trying to
explain how you solved the problem, step by step?<o:p></o:p></SPAN></DIV>
<DIV class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></DIV>
<DIV class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">Best
Regards,<o:p></o:p></SPAN></DIV>
<DIV class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></DIV>
<DIV class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'">Mattias
Segerdahl<o:p></o:p></SPAN></DIV>
<DIV class=MsoNormal><SPAN lang=EN-US
style="FONT-SIZE: 11pt; COLOR: #1f497d; FONT-FAMILY: 'Calibri','sans-serif'"><o:p> </o:p></SPAN></DIV>
<DIV
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0cm; BORDER-TOP: medium none; PADDING-LEFT: 4pt; PADDING-BOTTOM: 0cm; BORDER-LEFT: blue 1.5pt solid; PADDING-TOP: 0cm; BORDER-BOTTOM: medium none">
<DIV>
<DIV
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0cm; BORDER-TOP: #b5c4df 1pt solid; PADDING-LEFT: 0cm; PADDING-BOTTOM: 0cm; BORDER-LEFT: medium none; PADDING-TOP: 3pt; BORDER-BOTTOM: medium none">
<DIV class=MsoNormal><B><SPAN lang=EN-US
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Tahoma','sans-serif'">From:</SPAN></B><SPAN
lang=EN-US style="FONT-SIZE: 10pt; FONT-FAMILY: 'Tahoma','sans-serif'">
ocfs2-users-bounces@oss.oracle.com
[mailto:ocfs2-users-bounces@oss.oracle.com] <B>On Behalf Of </B>Gaetano
Giunta<BR><B>Sent:</B> den 11 maj 2007 09:47<BR><B>To:</B>
Ocfs2-users@oss.oracle.com<BR><B>Subject:</B> RE: [Ocfs2-users] PBL with
RMAN and ocfs2<o:p></o:p></SPAN></DIV></DIV></DIV>
<DIV class=MsoNormal><o:p> </o:p></DIV>
<DIV>
<DIV class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">Thanks,
but I had alreday checked out all logs I could find (oracle and crs
alerts, /var/log stuff) and there was no clear indication in
there.</SPAN><o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal> <o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">The
trick is the ocfs was sending the alert message to the console only (I
wonder why it does not also leva traces into syslog, my best guess is it
tries to shutdown as fast as it can, and sending a message to console is
faster than sending it to syslog - but I'm in no way a linux
guru...).</SPAN><o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal> <o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">By using the
netdump tool suggested by Sunil I managed to see the console messages of
the dying node (without having to phisycally be in the server farm, which
is 40 km away from my ususal workplace), and diagnosed the ocfs2 heartbeat
as "the killer".</SPAN><o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal> <o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">Bye</SPAN><o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">Gaetano</SPAN><o:p></o:p></DIV></DIV>
<BLOCKQUOTE
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0cm; BORDER-TOP: medium none; MARGIN-TOP: 5pt; PADDING-LEFT: 4pt; MARGIN-BOTTOM: 5pt; PADDING-BOTTOM: 0cm; MARGIN-LEFT: 3.75pt; BORDER-LEFT: blue 1.5pt solid; PADDING-TOP: 0cm; BORDER-BOTTOM: medium none">
<DIV class=MsoNormal style="MARGIN-BOTTOM: 12pt"><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Tahoma','sans-serif'">-----Original
Message-----<BR><B>From:</B> Luis Freitas
[mailto:lfreitas34@yahoo.com]<BR><B>Sent:</B> Thursday, May 10, 2007
11:17 PM<BR><B>To:</B> Gaetano Giunta<BR><B>Cc:</B>
Ocfs2-users@oss.oracle.com<BR><B>Subject:</B> Re: [Ocfs2-users] PBL with
RMAN and ocfs2</SPAN><o:p></o:p></DIV>
<DIV>
<DIV class=MsoNormal>Gaetano,<o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal> <o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal> If o2cb or CRS is killing
the machine, it usually shows on /var/log/messages with lines
explaining what happened. Take a look on the /var/log/messages just
before the last "syslogd x.x.x: restart".<o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal> <o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal>Regards,<o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal>Luis<o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal><BR><BR><BR>Gaetano Giunta wrote:<BR>>
Hello.<BR>><BR>> On a 2 node RAC 10.2.0.3 setup, on RH ES 4.4
x86_64, with ocfs 1.2.5-1, we are experiencing some troubles with RMAN:
when the archive log destination is on an ASM partition, and the backup
detsination is on ocfs2, running<BR>><BR>> backup archivelog all
format
'/home/SANstorage/oracle/backup/rman/dump_log/FULL_20070509_154916/arc_%d_%u'
delete input;<BR>><BR>> consistently causes a
reboot.<BR>><BR>> The rman catalog is clean, and has been
crosschecked in every way.<BR>><BR>> We tried on both nodes, and
the node executing the backup always reboots.<BR>> I am thus inclined
to think that it is not the ocfs2 dlm that triggers the reboot, because
in that case the victim would always be the second node.<BR>><BR>>
I also tested the same command using as backup destination /tmp, and all
was fine. The backup file of the archived logs is 1249843712 in
size.<BR>><BR>> Our local oracle guy went through metalink and
said there is no open bug/patch for that at this time.<BR>><BR>>
Any suggestions ???<BR>><BR>> Thanks<BR>> Gaetano
Giunta<BR>><BR>> <BR>>
------------------------------------------------------------------------<BR>><BR>>
_______________________________________________<BR>> Ocfs2-users
mailing list<BR>> Ocfs2-users@oss.oracle.com<BR>>
http://oss.oracle.com/mailman/listinfo/ocfs2-users<BR><BR><BR>_______________________________________________<BR>Ocfs2-users
mailing
list<BR>Ocfs2-users@oss.oracle.com<BR>http://oss.oracle.com/mailman/listinfo/ocfs2-users<o:p></o:p></DIV></DIV>
<DIV class=MsoNormal><o:p> </o:p></DIV>
<DIV class=MsoNormal style="TEXT-ALIGN: center" align=center>
<HR align=center width="100%" SIZE=1>
</DIV>
<DIV class=MsoNormal>Ahhh...imagining that irresistible "new car"
smell?<BR>Check out <A
href="http://us.rd.yahoo.com/evt=48245/*http:/autos.yahoo.com/new_cars.html;_ylc=X3oDMTE1YW1jcXJ2BF9TAzk3MTA3MDc2BHNlYwNtYWlsdGFncwRzbGsDbmV3LWNhcnM-">new
cars at Yahoo! Autos.</A>
<o:p></o:p></DIV></BLOCKQUOTE></DIV></DIV></BLOCKQUOTE>_______________________________________________<BR>Ocfs2-users
mailing
list<BR>Ocfs2-users@oss.oracle.com<BR>http://oss.oracle.com/mailman/listinfo/ocfs2-users</BLOCKQUOTE><BR>
<P>
<HR SIZE=1>
<A
href="http://us.rd.yahoo.com/evt=48250/*http://searchmarketing.yahoo.com/arp/sponsoredsearch_v9.php?o=US2226&cmp=Yahoo&ctv=AprNI&s=Y&s2=EM&b=50">Pinpoint
customers </A>who are looking for what you sell. </BLOCKQUOTE></BODY></HTML>