<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=NO-BOK link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Hi,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>We’re running an Oracle cluster with Oracle cluster dataguard. For a testing reason and to change hostnames, we wanted to reinstall the three dataguard nodes. The old installation was/is:<o:p></o:p></span></p><p class=MsoNormal style='text-indent:35.4pt'>`uname –a`<o:p></o:p></p><p class=MsoNormal style='text-indent:35.4pt'>Linux dbnode1 2.6.18-36.el5 #1 SMP Fri Jul 20 14:26:46 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></p><p class=MsoNormal style='text-indent:35.4pt'><o:p> </o:p></p><p class=MsoNormal style='text-indent:35.4pt'><span lang=EN-US>Installed with ocfs2:<o:p></o:p></span></p><p class=MsoNormal style='text-indent:35.4pt'><span lang=EN-US>rpm –qa|grep ocfs2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> ocfs2-tools-debuginfo-1.2.6-1.el5<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-US>ocfs2-tools-1.2.6-1.el5<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-US>ocfs2-2.6.18-8.1.8.el5-1.2.6-6.el5<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-US>ocfs2console-1.2.6-1.el5<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-US>ocfs2-tools-devel-1.2.6-1.el5<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The reinstalled nodes was installed with:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> `uname –a`<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> Linux dbnode2 2.6.18-238.1.1.el5 #1 SMP Tue Jan 4 13:32:19 EST 2011 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US> rpm –qa|grep ocfs2<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-US>ocfs2-2.6.18-238.1.1.el5-1.4.7-1.el5<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-US>ocfs2console-1.4.4-1.el5<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-US>ocfs2-tools-1.4.4-1.el5<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Oracle version is 10.2.0.3 on both.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The installation seems to be OK. I first reinstalled the two node who wasn’t running the dataguard instance. When these two was installed, I tried to start the dataguard instance on node2. To get node1 ready for reinstallation. But when I run “startup mount”, the database wouldn’t mount. It was “hanging” until I run “shutdown abort” in another session. After a logn while, while “hanging”, I finally got an ora-600 [2116] in my alertlog. As well as two tracefiles in the bdump catalog. One of the tracefiles was saying:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>*****snipp*****<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>*** 2011-02-07 09:33:55.096<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>*** SERVICE NAME:() 2011-02-07 09:33:55.096<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>*** SESSION ID:(2195.1) 2011-02-07 09:33:55.096<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Received ORADEBUG command 'dump errorstack 3' from process Unix process pid: 17868, image:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>*** 2011-02-07 09:33:55.096<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>ksedmp: internal or fatal error<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>****snipp****<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The other tracefile:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>*** 2011-02-07 09:23:52.445<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>*** SERVICE NAME:() 2011-02-07 09:23:52.444<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>*** SESSION ID:(2185.1) 2011-02-07 09:23:52.444<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Waited for detached process: CKPT for 300 seconds:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>*** 2011-02-07 09:23:52.445<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Dumping diagnostic information for CKPT:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>OS pid = 17835<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>loadavg : 3.01 3.04 2.91<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>memory info: free memory = 0.00M<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>swap info: free = 0.00M alloc = 0.00M total = 0.00M<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>0 S oracle 17835 1 0 75 0 - 1871417 io_get 09:18 ? 00:00:00 ora_ckpt_DBDG02<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[Thread debugging using libthread_db enabled]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffc2de9000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>0x00002b7f59d5f5b4 in ?? () from /usr/lib64/libaio.so.1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The alertlog sayings:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>ALTER DATABASE MOUNT<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Mon Feb 7 09:18:52 2011<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>This instance was first to mount<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Mon Feb 7 09:33:55 2011<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Errors in file /disk00/admin/DBDG/bdump/DBDG02_ckpt_17835.trc:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Mon Feb 7 09:33:55 2011<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Trace dumping is performing id=[cdmp_20110207093355]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Mon Feb 7 09:33:55 2011<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Errors in file /disk00/admin/ DBDG/udump/ DBDG02_ora_17868.trc:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>ORA-00600: internal error code, arguments: [2116], [900], [], [], [], [], [], []<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Mon Feb 7 09:33:56 2011<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Trace dumping is performing id=[cdmp_20110207093356]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Mon Feb 7 09:43:23 2011<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Shutting down instance (abort)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>License high water mark = 2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Termination issued to instance processes. Waiting for the processes to exit<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Mon Feb 7 09:43:33 2011<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Instance termination failed to kill one or more processes<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Instance terminated by USER, pid = 25297<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>After some investigation, I thought there must be something wrong in the OS or OCFS2. So we downgraded the kernel to:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> Linux dbnode2 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>And installed the ocfs2 kernelversion. And then everything worked OK. So my question is if there is something wrong with ocfs2 kernel version ocfs2-2.6.18-238.1.1.el5-1.4.7-1.el5 or is it the OS that fails?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:NO-BOK'>Mvh<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:NO-BOK'>Morten Kristiansen<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div></body></html>