<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 92.4pt 1.0in 92.4pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="2050" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoPlainText>Just as a FYI, we currently trying to track down a memory
leak too, but that is really with Oracle 10gR2/RAC related. We have a SGA size
configured of 6GB, yet the used memory over time goes further up. When we stop
the database itself (not OCFS2) the memory gets released. Neither Oracle nor
RedHat has been any help so far in trying to track down the component causing
this increased memory usage.<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Below is a graph of our memory usage on one of the nodes:<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText><img width=897 height=315 id="Picture_x0020_1"
src="cid:image001.png@01CADC77.2990B9D0"
alt="http://log01/atcgraph2/atc-grapher2.cgi?host=dbprd01.autc.com;type=memory;sub=;range=year"><o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>In September last year we increased memory to 16GB and increased
the SGA size (using Huge Pages also). Every time you see the memory drop, it is
because we either stopped Oracle or rebooted. This memory use caused us
problems before we switched to Huge Pages, in similar fashion as you reported.
High CPU/IO load due to swapping, as far we could track it down, some process/subtask
of Oracle had a large amount of memory allocated but not actively used.
Something triggered the process to wake up or terminate, which caused the
memory needed to be swapped back in (to free as far we could tell). That caused
such high CPU/IO load that RAC ejected the node from the cluster due to missing
communication.<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>> -----Original Message-----<o:p></o:p></p>
<p class=MsoPlainText>> From: ocfs2-users-bounces@oss.oracle.com
[mailto:ocfs2-users-<o:p></o:p></p>
<p class=MsoPlainText>> bounces@oss.oracle.com] On Behalf Of Kristiansen
Morten<o:p></o:p></p>
<p class=MsoPlainText>> Sent: Thursday, April 15, 2010 5:39 AM<o:p></o:p></p>
<p class=MsoPlainText>> To: Joel Becker<o:p></o:p></p>
<p class=MsoPlainText>> Cc: ocfs2-users@oss.oracle.com<o:p></o:p></p>
<p class=MsoPlainText>> Subject: Re: [Ocfs2-users] memory leak<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> OK, didn't know that this was normal. And yes, we
have had a system<o:p></o:p></p>
<p class=MsoPlainText>> crash where the servers where swapping a lot, and
cpu and load<o:p></o:p></p>
<p class=MsoPlainText>> therefore was high, causing the servers to reboote.
So I suspected a<o:p></o:p></p>
<p class=MsoPlainText>> memory leak, causing swapping, high load and reboot.
I can see that the<o:p></o:p></p>
<p class=MsoPlainText>> systems free memory is slowly continuously dropping,
maybe that's<o:p></o:p></p>
<p class=MsoPlainText>> normal referring to your answer. Yesterday we had
serious problems with<o:p></o:p></p>
<p class=MsoPlainText>> the system, where the servers rebooted in the end.
After reboot,<o:p></o:p></p>
<p class=MsoPlainText>> everything works well. It is several months between
each reboot caused<o:p></o:p></p>
<p class=MsoPlainText>> by swap. So maybe there is something else that
triggers something that<o:p></o:p></p>
<p class=MsoPlainText>> are causing the swap?<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> Our plan now is to upgrade ocfs2 to the newest
version and RedHat as<o:p></o:p></p>
<p class=MsoPlainText>> well. I guess ocfs2 version 1.2.6 can run together
with 1.4.4 during<o:p></o:p></p>
<p class=MsoPlainText>> the upgrade?<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> Morten K<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> -----Opprinnelig melding-----<o:p></o:p></p>
<p class=MsoPlainText>> Fra: ocfs2-users-bounces@oss.oracle.com
[mailto:ocfs2-users-<o:p></o:p></p>
<p class=MsoPlainText>> bounces@oss.oracle.com] På vegne av Joel Becker<o:p></o:p></p>
<p class=MsoPlainText>> Sendt: 15. april 2010 14:19<o:p></o:p></p>
<p class=MsoPlainText>> Til: Kristiansen Morten<o:p></o:p></p>
<p class=MsoPlainText>> Kopi: ocfs2-users@oss.oracle.com<o:p></o:p></p>
<p class=MsoPlainText>> Emne: Re: [Ocfs2-users] memory leak<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> On Thu, Apr 15, 2010 at 12:31:02PM +0200,
Kristiansen Morten wrote:<o:p></o:p></p>
<p class=MsoPlainText>> > I discovered our four node cluster running on
RedHat EL5, Ocfs2 1.2.6<o:p></o:p></p>
<p class=MsoPlainText>> and Oracle 10.2.0.3 have memory leak. I suspect
ocfs2, but I could be<o:p></o:p></p>
<p class=MsoPlainText>> wrong. I suspect ocfs2 because when we run RMAN
backup the free memory<o:p></o:p></p>
<p class=MsoPlainText>> goes from 8 GB down to 200 MB. When I umount the
ocfs2 backupdisk after<o:p></o:p></p>
<p class=MsoPlainText>> the backup is finished, the memory is released
again.<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> You don't have a memory leak. Your backup is
reading every file<o:p></o:p></p>
<p class=MsoPlainText>> into cache in order to process it. This is a normal
behavior of<o:p></o:p></p>
<p class=MsoPlainText>> filesystem cache. If other processes need memory,
the file data will<o:p></o:p></p>
<p class=MsoPlainText>> be<o:p></o:p></p>
<p class=MsoPlainText>> evicted from cache.<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> > I want to test it some more and found a script
to test writing to the<o:p></o:p></p>
<p class=MsoPlainText>> disk. This script contains a command saying
"echo 3 ><o:p></o:p></p>
<p class=MsoPlainText>> /proc/sys/vm/drop_caches". Is this a safe
command to run in production?<o:p></o:p></p>
<p class=MsoPlainText>> Meaning the cluster and oracle database is running.
Or should I run the<o:p></o:p></p>
<p class=MsoPlainText>> "sync" command pre to this command? Or
should I never run this command<o:p></o:p></p>
<p class=MsoPlainText>> in a production environment? I'm afraid that this
command will free up<o:p></o:p></p>
<p class=MsoPlainText>> memory not written to disk yet. And therefore I
would get into trouble<o:p></o:p></p>
<p class=MsoPlainText>> in my production environment. The script I want to
test looks like this<o:p></o:p></p>
<p class=MsoPlainText>> and is captured from this mailing list:<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> The command "echo 3 >
/proc/sys/vm/drop_caches" is safe to run<o:p></o:p></p>
<p class=MsoPlainText>> in a production environment. However, I don't think
you want to do so.<o:p></o:p></p>
<p class=MsoPlainText>> What it does is evicts all file data from cache.
Don't worry, you<o:p></o:p></p>
<p class=MsoPlainText>> won't<o:p></o:p></p>
<p class=MsoPlainText>> lose any data. But you will not only evict the data
from the backup<o:p></o:p></p>
<p class=MsoPlainText>> volume, you will also evict data from any other
file. This may slow<o:p></o:p></p>
<p class=MsoPlainText>> down some processes as they have to re-read their
data.<o:p></o:p></p>
<p class=MsoPlainText>> Why do you feel there is a problem? Is it just
that the free<o:p></o:p></p>
<p class=MsoPlainText>> memory number shrinks? That's not a problem, as
stated above. Is<o:p></o:p></p>
<p class=MsoPlainText>> there<o:p></o:p></p>
<p class=MsoPlainText>> some other affect on the system?<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> Joel<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> --<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> "Here's something to think about: How come you
never see a headline<o:p></o:p></p>
<p class=MsoPlainText>> like ``Psychic Wins Lottery''?"<o:p></o:p></p>
<p class=MsoPlainText>> - Jay Leno<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> Joel Becker<o:p></o:p></p>
<p class=MsoPlainText>> Principal Software Developer<o:p></o:p></p>
<p class=MsoPlainText>> Oracle<o:p></o:p></p>
<p class=MsoPlainText>> E-mail: joel.becker@oracle.com<o:p></o:p></p>
<p class=MsoPlainText>> Phone: (650) 506-8127<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> _______________________________________________<o:p></o:p></p>
<p class=MsoPlainText>> Ocfs2-users mailing list<o:p></o:p></p>
<p class=MsoPlainText>> Ocfs2-users@oss.oracle.com<o:p></o:p></p>
<p class=MsoPlainText>> http://oss.oracle.com/mailman/listinfo/ocfs2-users<o:p></o:p></p>
<p class=MsoPlainText>> <o:p></o:p></p>
<p class=MsoPlainText>> _______________________________________________<o:p></o:p></p>
<p class=MsoPlainText>> Ocfs2-users mailing list<o:p></o:p></p>
<p class=MsoPlainText>> Ocfs2-users@oss.oracle.com<o:p></o:p></p>
<p class=MsoPlainText>> http://oss.oracle.com/mailman/listinfo/ocfs2-users<o:p></o:p></p>
</div>
</body>
</html>