<div>Alexei,</div> <div> </div> <div> I am not a Linux kernel specialist, but generally memory on the "buffer cache" is on a different "list" than memory "free", so the kernel has to reclaim the buffers from the buffer "list" to release them to the "free" "list" as memory pressure increases.</div> <div> </div> <div> This has a processing cost. And on large database servers is undesirable as the database does not use the buffer cache, so we have the processing cost without any benefit.</div> <div> </div> <div> Usually on a database environment I tune the kernel as to keep a large amount of the memory on the "free" list, so that it be readly available when spawning new database processes.</div> <div> </div> <div>Regards,</div> <div>Luis<BR><BR><B><I>Alexei_Roudnev <Alexei_Roudnev@exigengroup.com></I></B> wrote:</div> <BLOCKQUOTE class=replbq style="PADDING-LEFT: 5px;
MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid"> <META content="MSHTML 6.00.2800.1561" name=GENERATOR> <DIV><FONT face=Arial size=2>Suni, you do know it much better - is not is strange a little? (so many ext3 inode chache objects, togetrher with 3 GB of cached disk space).</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>But I dont see anything wrong below - big 'cached' memory means only 'you have so many unused memory that system cached files in it' and nothing more. Cached memory is in reality _free for immediate use_ memory (can be used by anyone immediately).</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>What ysstem is doing is _instead of freeing memory and filling it by zerous, free unused memory and fill it in by buffer cache data, so that if someone need these data he have them at once. If someone need more memory, system reuse any of 'chached' memory without any delay
(because these are buffers which are already written to the disk or which was never udpated).</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial size=2>Big 'cached' value means only _big free memory_.</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <BLOCKQUOTE style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px"> <DIV style="FONT: 10pt arial">----- O riginal Message ----- </DIV> <DIV style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B> <A title=lfreitas34@yahoo.com href="mailto:lfreitas34@yahoo.com">Luis Freitas</A> </DIV> <DIV style="FONT: 10pt arial"><B>To:</B> <A title=Alexei_Roudnev@exigengroup.com href="mailto:Alexei_Roudnev@exigengroup.com">Alexei_Roudnev</A> ; <A title=blsieler@gmail.com href="mailto:blsieler@gmail.com">Brian Sieler</A> ; <A title=ocfs2-users@oss.oracle.com
href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</A> </DIV> <DIV style="FONT: 10pt arial"><B>Sent:</B> Monday, April 09, 2007 4:32 PM</DIV> <DIV style="FONT: 10pt arial"><B>Subject:</B> Re: [Ocfs2-users] High on buffers and deep on swap</DIV> <DIV><BR></DIV> <DIV>Alexei,</DIV> <DIV> </DIV> <DIV> How can I relate the information on slabtop to the actual memory used by buffers?</DIV> <DIV> </DIV> <DIV> I see this on slabtop:</DIV> <DIV> </DIV> <DIV><FONT face=courier size=2> Active / Total Objects (% used) : 603822 / 649643 (92.9%)<BR> Active / Total Slabs (% used) : 47216 / 47216 (100.0%)<BR> Active / Total Caches (% used) : 97 / 133 (72.9%)<BR> Active / Total Size (% used) : 176601.08K / 181508.49K (97.3%)<BR> Minimum / Average / Maximum Object : 0.01K / 0.28K /
128.00K</FONT></DIV> <DIV><FONT face=courier size=2> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME<BR>202461 202451 99% 0.54K 28923 7 115692K ext3_inode_cache<BR>232206 232153 99% 0.15K 8931 26 35724K dentry_cache<BR> 29974 26092 87% 0.27K 2141 14 8564K radix_tree_node<BR> 68250 62400 91% 0.05K 910 75 3640K buffer_head<BR> 855 855 100% 4.00K 855 1 3420K pmd<BR> 647 647
100% 4.00K 647 1 2588K size-4096<BR> 8595 7694 89% 0.25K 573 15 2292K filp<BR> 20835 17331 83% 0.09K 463 45 1852K vm_area_struct<BR> 780 767 98% 2.00K 390 2 1560K size-2048<BR> 18849 7962 42% 0.06K 309 61 1236K size-64<BR> 2440 632 25% 0.50K 305
8 1220K size-512<BR> 2926 2889 98% 0.34K 266 11 1064K inode_cache<BR> 256 256 100% 3.00K 128 2 1024K biovec-(256)<BR> 600 592 98% 1.38K 120 5 960K task_struct<BR> 515 512 99% 1.38K 103 5 824K pirpIo<BR> 5084 2365 46% 0.12K 164 31 656K size-128<BR></FONT></DIV>
<DIV> </DIV> <DIV> The largest area is about 100Mb, but on free there are over 3Gb on the "cached" column:</DIV> <DIV> </DIV> <DIV><FONT face=courier size=1>[oracle@br001sv0431 ~]$ free<BR> total used free shared buffers cached<BR>Mem: 5190736 4461420 729316 0 141836 3265464<BR>-/+ buffers/cache: 1054120 4136616<BR>Swap: 2048248 0 2048248<BR>[oracle@br001sv0431 ~]$<BR></FONT></DIV> <DIV><FONT face=courier size=1></FONT> </DIV>
<DIV>Regards,</DIV> <DIV>Luis</DIV> <DIV> </DIV> <DIV><BR><BR><B><I>Alexei_Roudnev <Alexei_Roudnev@exigengroup.com></I></B> wrote:</DIV> <BLOCKQUOTE class=replbq style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid"> <META content="MSHTML 6.00.2800.1561" name=GENERATOR> <DIV><FONT face=Arial size=2>Did you run slabtop ? It can show unreleased buffers in the system.</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <BLOCKQUOTE style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px"> <DIV style="FONT: 10pt arial">----- Original Message ----- </DIV> <DIV style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B> <A title=lfreitas34@yahoo.com href="mailto:lfreitas34@yahoo.com">Luis Freitas</A> </DIV> <DIV style="FONT: 10pt arial"><B>To:</B> <A title=Alexei_Roudnev@exigengroup.com
href="mailto:Alexei_Roudnev@exigengroup.com">Alexei_Roudnev</A> ; <A title=blsieler@gmail.com href="mailto:blsieler@gmail.com">Brian Sieler</A> ; <A title=ocfs2-users@oss.oracle.com href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</A> </DIV> <DIV style="FONT: 10pt arial"><B>Sent:</B> Monday, April 09, 2007 2:29 PM</DIV> <DIV style="FONT: 10pt arial"><B>Subject:</B> Re: [Ocfs2-users] High on buffers and deep on swap</DIV> <DIV><BR></DIV> <DIV>Alexei,</DIV> <DIV> </DIV> <DIV> Yes, it seems to have no effect, which too is very strange. On 2.4 vm.freepages had a very easy to notice effect.</DIV> <DIV> </DIV> <DIV> There are other people having problems with buffers not being released on the list and some of them are forcing the kernel cache to be flushed with:</DIV> <DIV> </DIV> <DIV>echo 3 > /proc/sys/vm/drop_caches</DIV> <DIV> </DIV> <DIV> But I dont see this parameter on RHAS
4.0.</DIV> <DIV> </DIV> <DIV> Also, to be fair this seems to be a generic VM issue, I see this on servers that are not running ocfs2 too. And I only see this behavior on machines with more than 2Gb of memory.</DIV> <DIV> </DIV> <DIV>Regards,</DIV> <DIV>Luis</DIV> <DIV> </DIV> <DIV>Regards,<BR>Luis<BR><BR><B><I>Alexei_Roudnev <Alexei_Roudnev@exigengroup.com></I></B> wrote:</DIV> <BLOCKQUOTE class=replbq style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid"> <META content="MSHTML 6.00.2800.1561" name=GENERATOR> <STYLE>v\:* { BEHAVIOR: url(#default#VML) } o\:* { BEHAVIOR: url(#default#VML) } w\:* { BEHAVIOR: url(#default#VML) } .shape { BEHAVIOR: url(#default#VML) } </STYLE> <STYLE> <!-- /* Font Definitions */ @font-face {font-family:Courier; panose-1:2 7 4 9 2 2 5 2 4 4;} @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} /* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman";} a:link, span.MsoHyperlink {color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {color:blue; text-decoration:underline;} p.MsoPlainText, li.MsoPlainText, div.MsoPlainText {margin:0in; margin-bottom:.0001pt; font-size:10.0pt; font-family:"Courier New";} p {mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; font-size:12.0pt; font-family:"Times New Roman";} span.EmailStyle18 {mso-style-type:personal-reply; font-family:Arial; color:navy;} @page Section1 {size:8.5in 11.0in; margin:1.0in 1.25in 1.0in 1.25in;} div.Section1 {page:Section1;} --> </STYLE> <DIV><FONT face=Arial size=2>Did you tried vm.swappiness parameter?</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <DIV><FONT face=Arial
size=2>(/proc/sys/vm/swappiness)</FONT></DIV> <DIV><FONT face=Arial size=2></FONT> </DIV> <BLOCKQUOTE style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px"> <DIV style="FONT: 10pt arial">----- </DIV> <DIV style="FONT: 10pt arial">Original Message ----- </DIV> <DIV style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B> <A title=blsieler@gmail.com href="mailto:blsieler@gmail.com">Brian Sieler</A> </DIV> <DIV style="FONT: 10pt arial"><B>To:</B> <A title=lfreitas34@yahoo.com href="mailto:lfreitas34@yahoo.com">'Luis Freitas'</A> ; <A title=ocfs2-users@oss.oracle.com href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</A> </DIV> <DIV style="FONT: 10pt arial"><B>Sent:</B> Sunday, April 08, 2007 10:52 PM</DIV> <DIV style="FONT: 10pt arial"><B>Subject:</B> RE: [Ocfs2-users] High on buffers and deep on swap</DIV> <DIV><BR></DIV> <DIV class=Section1> <DIV
class=MsoNormal><FONT face="Courier New" color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">Luis, yes I am experiencing what appears to be a similar problem you are describing. See my post from just a few minutes ago on another thread.<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p></SPAN></FONT></DIV> <DIV class=MsoNormal><FONT face="Courier New" color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'"><o:p> </o:p></SPAN></FONT></DIV> <DIV class=MsoPlainText><FONT face="Courier New" color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy">I run a 2-node cluster with OCFS2/RAC on </SPAN></FONT>2.6.9-34.0.2.ELsmp (RHEL 4.0) as well.<o:p></o:p></DIV> <DIV class=MsoNormal><FONT face="Courier New" color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'"><o:p> </o:p></SPAN></FONT></DIV> <DIV class=MsoNormal><FONT
face="Courier New" color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'"> total used free shared buffers cached<o:p></o:p></SPAN></FONT></DIV> <DIV class=MsoNormal><FONT face="Courier New" color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">Mem: 4044496 4005516 38980 0 34108 2236636<o:p></o:p></SPAN></FONT></DIV> <DIV class=MsoNormal><FONT face="Courier New" color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">-/+ buffers/cache: 1734772
2309724<o:p></o:p></SPAN></FONT></DIV> <DIV class=MsoNormal><FONT face="Courier New" color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">Swap: 2097144 648244 1448900<o:p></o:p></SPAN></FONT></DIV> <DIV class=MsoNormal><FONT face="Courier New" color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'"><o:p> </o:p></SPAN></FONT></DIV> <DIV class=MsoNormal><FONT face="Courier New" color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">If you’ve uncovered anything since posting this message, please pass it along?<o:p></o:p></SPAN></FONT></DIV> <DIV class=MsoNormal><FONT face=Arial color=navy size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></DIV> <DIV> <DIV class=MsoNormal style="TEXT-ALIGN: center" align=center><FONT face="Times New Roman"
size=3><SPAN style="FONT-SIZE: 12pt"> <HR tabIndex=-1 align=center width="100%" SIZE=2> </SPAN></FONT></DIV> <DIV class=MsoNormal><B><FONT face=Tahoma size=2><SPAN style="FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">From:</SPAN></FONT></B><FONT face=Tahoma size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> ocfs2-users-bounces@oss.oracle.com [mailto:ocfs2-users-bounces@oss.oracle.com] <B><SPAN style="FONT-WEIGHT: bold">On Behalf Of </SPAN></B>Luis Freitas<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Friday, February 23, 2007 5:32 PM<BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> ocfs2-users@oss.oracle.com<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B> [Ocfs2-users] High on buffers and deep on swap</SPAN></FONT><o:p></o:p></DIV></DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"><o:p> </o:p></SPAN></FONT></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman"
size=3><SPAN style="FONT-SIZE: 12pt">Hi,<o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> This is a bit off topic, hope there is not a problem.<o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> Anyone out there experiencing high swapping with the kernel retaining a large amount of buffers? This used to be a problem on 2.4, and I usually changed /proc/sys/vm/freepages to fix it. But on 2.6 this parameter no longer exists.<o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New
Roman" size=3><SPAN style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> One of the servers here is holding over 3.5Gb of cache even when using over 700Mb of swap, and free memory is always low.<o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face=Courier size=3><SPAN style="FONT-SIZE: 12pt; FONT-FAMILY: Courier">[oracle@br001sv0432 ~]$ free<BR> total used free shared buffers cached<BR>Mem: 5190736
4810880 379856 0 143032 3583868<BR>-/+ buffers/cache: 1083980 4106756<BR>Swap: 2048248 723064 1325184<BR>[oracle@br001sv0432 ~]$</SPAN></FONT><o:p></o:p></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face=Arial size=3><SPAN style="FONT-SIZE: 12pt; FONT-FAMILY: Arial"> I am tuning /proc/sys/vm/swappiness, but this seems to have no effect at all. Changed from 60 to 10 and seems to have no effect. The server runs Oracle RAC with OCFS2.</SPAN></FONT><o:p></o:p></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE:
12pt">Regards,<o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt">Luis<o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV> <DIV><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV> <DIV class=MsoNormal style="TEXT-ALIGN: center" align=center><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"> <HR align=center width="100%" SIZE=1> </SPAN></FONT></DIV> <DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt"><A
href="http://answers.yahoo.com/dir/index;_ylc=X3oDMTFvbGNhMGE3BF9TAzM5NjU0NTEwOARfcwMzOTY1NDUxMDMEc2VjA21haWxfdGFnbGluZQRzbGsDbWFpbF90YWcx?link=ask&sid=396545367">Food fight?</A> Enjoy some healthy debate<BR>in the <A href="http://answers.yahoo.com/dir/index;_ylc=X3oDMTFvbGNhMGE3BF9TAzM5NjU0NTEwOARfcwMzOTY1NDUxMDMEc2VjA21haWxfdGFnbGluZQRzbGsDbWFpbF90YWcx?link=ask&sid=396545367">Yahoo! Answers Food & Drink Q&A.</A><o:p></o:p></SPAN></FONT></DIV></DIV> <DIV> <HR> <DIV></DIV>_______________________________________________<BR>Ocfs2-users mailing list<BR>Ocfs2-users@oss.oracle.com<BR>http://oss.oracle.com/mailman/listinfo/ocfs2-users</DIV></BLOCKQUOTE></BLOCKQUOTE><BR> <DIV> <HR SIZE=1> Be a PS3 game guru.<BR>Get your game face on with <A href="http://us.rd.yahoo.com/evt=49936/*http://videogames.yahoo.com">the latest PS3 news and previews at Yahoo! Games.</A></DIV></BLOCKQUOTE></BLOCKQUOTE><BR> <div> <HR SIZE=1> Finding fabulous fares is fun.<BR><A
href="http://farechase.yahoo.com/promo-generic-14795097;_ylc=X3oDMTFtNW45amVpBF9TAzk3NDA3NTg5BF9zAzI3MTk0ODEEcG9zAzEEc2VjA21haWx0YWdsaW5lBHNsawNxMS0wNw--">Let Yahoo! FareChase search your favorite travel sites</A> to find flight and hotel bargains.</BLOCKQUOTE></BLOCKQUOTE><BR><p> 
<hr size=1><a href="http://us.rd.yahoo.com/evt=49935/*http://games.yahoo.com">Bored stiff?</a> Loosen up...<br><a href="http://us.rd.yahoo.com/evt=49935/*http://games.yahoo.com">Download and play hundreds of games for free</a> on Yahoo! Games.