[Ocfs2-users] memory leak

[Herbert van den Bergh]

1. use filesystemio_options=directio or =setall to bypass the Linux cache
2. drop_caches has a bug that causes hugepages to be dropped, so don't 
use it if you're using hugepages for the SGA, or get a fix for it. 
(oracle bug 9461825, RedHat bugzilla 578977)

The swapping most likely happens because the Linux cache fills up with 
dirty blocks, and the disk can't keep up with RMAN filling the cache.  
When processes then start allocating memory, the kernel is forced to 
free whatever memory it can, and since the cache pages are waiting to be 
written to disk, the kernel starts reclaiming (paging out and freeing) 
process pages.  Using O_DIRECT should avoid this situation.

Thanks,
Herbert.


On 04/15/2010 05:38 AM, Kristiansen Morten wrote:
> OK, didn't know that this was normal. And yes, we have had a system crash where the servers where swapping a lot, and cpu and load therefore was high, causing the servers to reboote. So I suspected a memory leak, causing swapping, high load and reboot. I can see that the systems free memory is slowly continuously dropping, maybe that's normal referring to your answer. Yesterday we had serious problems with the system, where the servers rebooted in the end. After reboot, everything works well. It is several months between each reboot caused by swap. So maybe there is something else that triggers something that are causing the swap?
>
> Our plan now is to upgrade ocfs2 to the newest version and RedHat as well. I guess ocfs2 version 1.2.6 can run together with 1.4.4 during the upgrade?
>
> Morten K
>
> -----Opprinnelig melding-----
> Fra: ocfs2-users-bounces at oss.oracle.com [mailto:ocfs2-users-bounces at oss.oracle.com] På vegne av Joel Becker
> Sendt: 15. april 2010 14:19
> Til: Kristiansen Morten
> Kopi: ocfs2-users at oss.oracle.com
> Emne: Re: [Ocfs2-users] memory leak
>
> On Thu, Apr 15, 2010 at 12:31:02PM +0200, Kristiansen Morten wrote:
>    
>> I discovered our four node cluster running on RedHat EL5, Ocfs2 1.2.6 and Oracle 10.2.0.3 have memory leak. I suspect ocfs2, but I could be wrong. I suspect ocfs2 because when we run RMAN backup the free memory goes from 8 GB down to 200 MB. When I umount the ocfs2 backupdisk after the backup is finished, the memory is released again.
>>      
> 	You don't have a memory leak.  Your backup is reading every file
> into cache in order to process it.  This is a normal behavior of
> filesystem cache.  If other processes need memory, the file data will be
> evicted from cache.
>
>    
>> I want to test it some more and found a script to test writing to the disk. This script contains a command saying "echo 3>  /proc/sys/vm/drop_caches". Is this a safe command to run in production? Meaning the cluster and oracle database is running. Or should I run the "sync" command pre to this command? Or should I never run this command in a production environment? I'm afraid that this command will free up memory not written to disk yet. And therefore I would get into trouble in my production environment. The script I want to test looks like this and is captured from this mailing list:
>>      
> 	The command "echo 3>  /proc/sys/vm/drop_caches" is safe to run
> in a production environment.  However, I don't think you want to do so.
> What it does is evicts all file data from cache.  Don't worry, you won't
> lose any data.  But you will not only evict the data from the backup
> volume, you will also evict data from any other file.  This may slow
> down some processes as they have to re-read their data.
> 	Why do you feel there is a problem?  Is it just that the free
> memory number shrinks?  That's not a problem, as stated above.  Is there
> some other affect on the system?
>
> Joel
>
>