[Ocfs2-users] memory leak

Thu Apr 15 08:39:29 PDT 2010

Just as a FYI, we currently trying to track down a memory leak too, but that is really with Oracle 10gR2/RAC related. We have a SGA size configured of 6GB, yet the used memory over time goes further up. When we stop the database itself (not OCFS2) the memory gets released. Neither Oracle nor RedHat has been any help so far in trying to track down the component causing this increased memory usage.

Below is a graph of our memory usage on one of the nodes:

[cid:image001.png at 01CADC77.2990B9D0]

In September last year we increased memory to 16GB and increased the SGA size (using Huge Pages also). Every time you see the memory drop, it is because we either stopped Oracle or rebooted. This memory use caused us problems before we switched to Huge Pages, in similar fashion as you reported. High CPU/IO load due to swapping, as far we could track it down, some process/subtask of Oracle had a large amount of memory allocated but not actively used. Something triggered the process to wake up or terminate, which caused the memory needed to be swapped back in (to free as far we could tell). That caused such high CPU/IO load that RAC ejected the node from the cluster due to missing communication.

> -----Original Message-----

> From: ocfs2-users-bounces at oss.oracle.com [mailto:ocfs2-users-

> bounces at oss.oracle.com] On Behalf Of Kristiansen Morten

> Sent: Thursday, April 15, 2010 5:39 AM

> To: Joel Becker

> Cc: ocfs2-users at oss.oracle.com

> Subject: Re: [Ocfs2-users] memory leak

>

> OK, didn't know that this was normal. And yes, we have had a system

> crash where the servers where swapping a lot, and cpu and load

> therefore was high, causing the servers to reboote. So I suspected a

> memory leak, causing swapping, high load and reboot. I can see that the

> systems free memory is slowly continuously dropping, maybe that's

> normal referring to your answer. Yesterday we had serious problems with

> the system, where the servers rebooted in the end. After reboot,

> everything works well. It is several months between each reboot caused

> by swap. So maybe there is something else that triggers something that

> are causing the swap?

>

> Our plan now is to upgrade ocfs2 to the newest version and RedHat as

> well. I guess ocfs2 version 1.2.6 can run together with 1.4.4 during

> the upgrade?

>

> Morten K

>

> -----Opprinnelig melding-----

> Fra: ocfs2-users-bounces at oss.oracle.com [mailto:ocfs2-users-

> bounces at oss.oracle.com] På vegne av Joel Becker

> Sendt: 15. april 2010 14:19

> Til: Kristiansen Morten

> Kopi: ocfs2-users at oss.oracle.com

> Emne: Re: [Ocfs2-users] memory leak

>

> On Thu, Apr 15, 2010 at 12:31:02PM +0200, Kristiansen Morten wrote:

> > I discovered our four node cluster running on RedHat EL5, Ocfs2 1.2.6

> and Oracle 10.2.0.3 have memory leak. I suspect ocfs2, but I could be

> wrong. I suspect ocfs2 because when we run RMAN backup the free memory

> goes from 8 GB down to 200 MB. When I umount the ocfs2 backupdisk after

> the backup is finished, the memory is released again.

>

>     You don't have a memory leak.  Your backup is reading every file

> into cache in order to process it.  This is a normal behavior of

> filesystem cache.  If other processes need memory, the file data will

> be

> evicted from cache.

>

> > I want to test it some more and found a script to test writing to the

> disk. This script contains a command saying "echo 3 >

> /proc/sys/vm/drop_caches". Is this a safe command to run in production?

> Meaning the cluster and oracle database is running. Or should I run the

> "sync" command pre to this command? Or should I never run this command

> in a production environment? I'm afraid that this command will free up

> memory not written to disk yet. And therefore I would get into trouble

> in my production environment. The script I want to test looks like this

> and is captured from this mailing list:

>

>     The command "echo 3 > /proc/sys/vm/drop_caches" is safe to run

> in a production environment.  However, I don't think you want to do so.

> What it does is evicts all file data from cache.  Don't worry, you

> won't

> lose any data.  But you will not only evict the data from the backup

> volume, you will also evict data from any other file.  This may slow

> down some processes as they have to re-read their data.

>     Why do you feel there is a problem?  Is it just that the free

> memory number shrinks?  That's not a problem, as stated above.  Is

> there

> some other affect on the system?

>

> Joel

>

> --

>

> "Here's something to think about:  How come you never see a headline

>  like ``Psychic Wins Lottery''?"

>     - Jay Leno

>

> Joel Becker

> Principal Software Developer

> Oracle

> E-mail: joel.becker at oracle.com

> Phone: (650) 506-8127

>

> _______________________________________________

> Ocfs2-users mailing list

> Ocfs2-users at oss.oracle.com

> http://oss.oracle.com/mailman/listinfo/ocfs2-users

>

> _______________________________________________

> Ocfs2-users mailing list

> Ocfs2-users at oss.oracle.com

> http://oss.oracle.com/mailman/listinfo/ocfs2-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20100415/b12248e4/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 33425 bytes
Desc: image001.png
Url : http://oss.oracle.com/pipermail/ocfs2-users/attachments/20100415/b12248e4/attachment-0001.png 