<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7652.24">
<TITLE>Failures in rac-st-asmlib-mempressure due to ASM start failure</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">I am running through the OLT for the first time and have ran into an issue with some rac test</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Calibri">s, i</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Calibri">n particular</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Calibri"> the</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Calibri">rac-st-asmlib-mempressure</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"> and</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><BR>
<FONT SIZE=2 FACE="Calibri">rac-st-asmlib-memleak2</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">. Each of these testcases have three sub-tests and the failure may occur in any</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Calibri">one</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Calibri">o</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">f the three sub-tests</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">. The failures occur after the DBT2 completes</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"> and when ASM is restarted</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"> in the validation stage</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">. The ASM start returns an ORA-3113 to the OLT which in turns causes the</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"></FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Calibri">testcase</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"> to fail.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">I am looking for</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Calibri">advice on</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Calibri">whether this is a known issue (maybe seen in bug</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">zilla</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"> n</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">umber</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"></FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Calibri">3450) with OLT or if I need to log an SR</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"> with Oracle support.</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"> Searching Metalink, I have found a close match with bug</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Calibri">5659909</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">.</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">Here is the rac_tuning.tlg output from which you can see that the DBT2 test completes, and the ASM startup issue occurs in the validation stage.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 12:29:25 : f_dbt2execute(oltdbt2-executetest) Invoking the DBT2 rac test</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 12:57:05 : f_dbt2execute For automated distructive test now failure will happen</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 12:57:05 : f_dbt2execute For manual distructive tests manually do the failure part</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:57:35 : f_dbt2execute (oltdbt2-executetest) DBT2 rac test completed</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:57:35 : f_stopdb(oltdbt2-executetest) Going to stop Database instance on ipht03</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:03 : f_stopdb(oltdbt2-executetest) Going to stop Database instance on ipht04</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:29 : f_stop_asm (oltdbt2-asm) Stopping ASM Instance on ipht03</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:34 : f_stop_asm (oltdbt2-asm) Stopping ASM Instance on ipht04</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:39 : f_stoplsnr(oltdbt2-executetest) Going to stop Listener on ipht03</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:45 : f_stoplsnr(oltdbt2-executetest) Going to stop Listener on ipht04</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:45 : f_stopqa (oltdbt2-qamonitor) Stopping qamonitor on ipht04</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:47 : f_validate(oltdbt2-results) Validating the run</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:47 : f_crsctrlmain(oltdbt2-executetest) Executing CRS - start</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:50 : f_crsctrlmain(oltdbt2-executetest) Completed CRS - start</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:50 : f_start_asm (oltdbt2-asm) Starting ASM Instance on ipht03</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">ERROR: 16:58:55: OULT_ERR_50:Unable to start ASM instance:f_start_asm(oltdbt2-executetest) Unable to start ASM instance on node 3</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:55 : f_stopdb(oltdbt2-executetest) Going to stop Database instance on ipht03</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:55 : f_stopdb(oltdbt2-executetest) Going to stop Database instance on ipht04</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:56 : f_stop_asm (oltdbt2-asm) Stopping ASM Instance on ipht03</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:56 : f_stop_asm (oltdbt2-asm) Stopping ASM Instance on ipht04</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:56 : f_stop_asm (oltdbt2-asm) Stopping ASM Instance on ipht03</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:56 : f_stop_asm (oltdbt2-asm) Stopping ASM Instance on ipht04</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:56 : f_stoplsnr(oltdbt2-executetest) Going to stop Listener on ipht03</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:56 : f_stoplsnr(oltdbt2-executetest) Going to stop Listener on ipht04</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:57 : f_clean_devshm(oltdbt2-results) Cleaning up /dev/shm</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:57 : f_analyzetest(oltdbt2-results) Analyzing the test </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:58:57 : f_analyzelogs(oltdbt2-results) Analyzing logs </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">INFO: 16:59:00 : f_master_relieve (oltdbt2-nodesync) Relieving all the RAC nodes</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">Looking through the log files, I am finding that the LMON process terminates the ASM instance</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Calibri">during</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"> ASM startup of the first node in a 2 node cluster in</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"> <FONT SIZE=2 FACE="Calibri">the validation stage with the following:</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">-----------start cut------------</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">*** 2008-03-11 16:58:50.973</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">kjxggpoll: change poll time to 600 ms</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">GES Client Freeze unsuccessful- retrying...</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi">Process 27403(LMS) not frozen</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi"><lines skipped></FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi"> GRANTED_Q: </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi"> DEFER MSG QUEUE ON LMS0 IS EMPTY</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="r_ansi"> SEQUENCES: Terminating instance due to freeze timeout (lms0 not frozen)</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">-------------end cut--------------------</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">I looked through the SYSLOGs and did not see</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri"> any issues with regards to memory being exhausted (13 GB free) at the time of the ASM startup.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">Any advice would be appreciated.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">Thanks</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT SIZE=2 FACE="Calibri">Robert</FONT></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN></P>
<HR>CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.<BR>
</BODY>
</HTML>