<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns:o = "urn:schemas-microsoft-com:office:office"><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.3059" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><SPAN class=535560718-10042007><FONT face=Arial
color=#0000ff size=2>Hello Luis,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=535560718-10042007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=535560718-10042007><FONT face=Arial
color=#0000ff size=2>the large usage of cache by inodes is not an issue in the
typical database setup, because you have only a very limited number of files
(used) on your filesystems. Also the database usually grows initially and then
does not request much more additional memory in operations.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=535560718-10042007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=535560718-10042007><FONT face=Arial
color=#0000ff size=2>However you are right, Oracle with Data Files on ocfsv2 has
no need to cache the data blocks on the FS layer. But I gues sthis is not done
(memadvice, fcntl or similiar methods can be used to enforce "directio" - i am
not sure which one is used by Oracle). I think the original problem reports with
large usage have been due to non-db workloads.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=535560718-10042007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=535560718-10042007><FONT face=Arial
color=#0000ff size=2>Gruss</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=535560718-10042007><FONT face=Arial
color=#0000ff size=2>Bernd</FONT></SPAN></DIV><BR>
<DIV class=OutlookMessageHeader lang=de dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> ocfs2-users-bounces@oss.oracle.com
[mailto:ocfs2-users-bounces@oss.oracle.com] <B>On Behalf Of </B>Luis
Freitas<BR><B>Sent:</B> Tuesday, April 10, 2007 7:59 PM<BR><B>To:</B>
Alexei_Roudnev; ocfs2-users@oss.oracle.com<BR><B>Subject:</B> Re: [Ocfs2-users]
High on buffers and deep on swap<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV>Alexei,</DIV>
<DIV> </DIV>
<DIV> I am not a Linux kernel specialist, but generally memory
on the "buffer cache" is on a different "list" than memory "free", so the kernel
has to reclaim the buffers from the buffer "list" to release them to the
"free" "list" as memory pressure increases.</DIV>
<DIV> </DIV>
<DIV> This has a processing cost. And on large database servers is
undesirable as the database does not use the buffer cache, so we have the
processing cost without any benefit.</DIV>
<DIV> </DIV>
<DIV> Usually on a database environment I tune the
kernel as to keep a large amount of the memory on the "free" list, so that it be
readly available when spawning new database processes.</DIV>
<DIV> </DIV>
<DIV>Regards,</DIV>
<DIV>Luis<BR><BR><B><I>Alexei_Roudnev
<Alexei_Roudnev@exigengroup.com></I></B> wrote:</DIV>
<BLOCKQUOTE class=replbq
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid">
<META content="MSHTML 6.00.2800.1561" name=GENERATOR>
<DIV><FONT face=Arial size=2>Suni, you do know it much better - is not is
strange a little? (so many ext3 inode chache objects, togetrher with 3 GB of
cached disk space).</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>But I dont see anything wrong below - big
'cached' memory means only 'you have so many unused memory that system cached
files in it' and nothing more. Cached memory is in reality _free for immediate
use_ memory (can be used by anyone immediately).</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>What ysstem is doing is _instead of freeing
memory and filling it by zerous, free unused memory and fill it in by buffer
cache data, so that if someone need these data he have them at once. If
someone need more memory, system reuse any of 'chached' memory without any
delay (because these are buffers which are already written to the disk or
which was never udpated).</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Big 'cached' value means only _big free
memory_.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<BLOCKQUOTE
style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV style="FONT: 10pt arial">----- O riginal Message ----- </DIV>
<DIV
style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B>
<A title=lfreitas34@yahoo.com href="mailto:lfreitas34@yahoo.com">Luis
Freitas</A> </DIV>
<DIV style="FONT: 10pt arial"><B>To:</B> <A
title=Alexei_Roudnev@exigengroup.com
href="mailto:Alexei_Roudnev@exigengroup.com">Alexei_Roudnev</A> ; <A
title=blsieler@gmail.com href="mailto:blsieler@gmail.com">Brian Sieler</A> ;
<A title=ocfs2-users@oss.oracle.com
href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</A>
</DIV>
<DIV style="FONT: 10pt arial"><B>Sent:</B> Monday, April 09, 2007 4:32
PM</DIV>
<DIV style="FONT: 10pt arial"><B>Subject:</B> Re: [Ocfs2-users] High on
buffers and deep on swap</DIV>
<DIV><BR></DIV>
<DIV>Alexei,</DIV>
<DIV> </DIV>
<DIV> How can I relate the information on slabtop to the actual
memory used by buffers?</DIV>
<DIV> </DIV>
<DIV> I see this on slabtop:</DIV>
<DIV> </DIV>
<DIV><FONT face=courier size=2> Active / Total Objects (%
used) : 603822 / 649643 (92.9%)<BR> Active / Total
Slabs (% used) : 47216 / 47216
(100.0%)<BR> Active / Total Caches (% used) :
97 / 133 (72.9%)<BR> Active / Total Size (%
used) : 176601.08K / 181508.49K
(97.3%)<BR> Minimum / Average / Maximum Object : 0.01K / 0.28K /
128.00K</FONT></DIV>
<DIV><FONT face=courier size=2> OBJS ACTIVE USE OBJ SIZE
SLABS OBJ/SLAB CACHE SIZE NAME<BR>202461 202451 99%
0.54K 28923
7 115692K ext3_inode_cache<BR>232206 232153
99% 0.15K
8931 26 35724K
dentry_cache<BR> 29974 26092 87%
0.27K 2141
14 8564K radix_tree_node<BR> 68250
62400 91% 0.05K
910 75
3640K buffer_head<BR> 855 855
100% 4.00K
855
1 3420K pmd<BR>
647 647 100% 4.00K
647
1 2588K size-4096<BR> 8595
7694 89% 0.25K
573 15
2292K filp<BR> 20835 17331 83%
0.09K 463
45 1852K vm_area_struct<BR>
780 767 98%
2.00K 390
2 1560K size-2048<BR> 18849
7962 42% 0.06K
309 61
1236K size-64<BR> 2440 632
25% 0.50K
305
8 1220K size-512<BR> 2926
2889 98% 0.34K
266 11
1064K inode_cache<BR> 256 256
100% 3.00K
128
2 1024K biovec-(256)<BR>
600 592 98%
1.38K 120
5 960K task_struct<BR>
515 512 99%
1.38K 103
5 824K pirpIo<BR> 5084
2365 46% 0.12K
164
31 656K size-128<BR></FONT></DIV>
<DIV> </DIV>
<DIV> The largest area is about 100Mb, but on free there are over 3Gb
on the "cached" column:</DIV>
<DIV> </DIV>
<DIV><FONT face=courier size=1>[oracle@br001sv0431 ~]$
free<BR>
total
used free
shared buffers
cached<BR>Mem: 5190736
4461420
729316
0 141836 3265464<BR>-/+
buffers/cache: 1054120
4136616<BR>Swap:
2048248
0 2048248<BR>[oracle@br001sv0431 ~]$<BR></FONT></DIV>
<DIV><FONT face=courier size=1></FONT> </DIV>
<DIV>Regards,</DIV>
<DIV>Luis</DIV>
<DIV> </DIV>
<DIV><BR><BR><B><I>Alexei_Roudnev
<Alexei_Roudnev@exigengroup.com></I></B> wrote:</DIV>
<BLOCKQUOTE class=replbq
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid">
<META content="MSHTML 6.00.2800.1561" name=GENERATOR>
<DIV><FONT face=Arial size=2>Did you run slabtop ? It can show
unreleased buffers in the system.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<BLOCKQUOTE
style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV style="FONT: 10pt arial">----- Original Message ----- </DIV>
<DIV
style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B>
<A title=lfreitas34@yahoo.com href="mailto:lfreitas34@yahoo.com">Luis
Freitas</A> </DIV>
<DIV style="FONT: 10pt arial"><B>To:</B> <A
title=Alexei_Roudnev@exigengroup.com
href="mailto:Alexei_Roudnev@exigengroup.com">Alexei_Roudnev</A> ; <A
title=blsieler@gmail.com href="mailto:blsieler@gmail.com">Brian
Sieler</A> ; <A title=ocfs2-users@oss.oracle.com
href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</A>
</DIV>
<DIV style="FONT: 10pt arial"><B>Sent:</B> Monday, April 09, 2007 2:29
PM</DIV>
<DIV style="FONT: 10pt arial"><B>Subject:</B> Re: [Ocfs2-users] High on
buffers and deep on swap</DIV>
<DIV><BR></DIV>
<DIV>Alexei,</DIV>
<DIV> </DIV>
<DIV> Yes, it seems to have no effect, which too is very strange.
On 2.4 vm.freepages had a very easy to notice effect.</DIV>
<DIV> </DIV>
<DIV> There are other people having problems with buffers
not being released on the list and some of them are forcing the kernel
cache to be flushed with:</DIV>
<DIV> </DIV>
<DIV>echo 3 > /proc/sys/vm/drop_caches</DIV>
<DIV> </DIV>
<DIV> But I dont see this parameter on RHAS 4.0.</DIV>
<DIV> </DIV>
<DIV> Also, to be fair this seems to be a
generic VM issue, I see this on servers that are not running ocfs2 too.
And I only see this behavior on machines with more than 2Gb of
memory.</DIV>
<DIV> </DIV>
<DIV>Regards,</DIV>
<DIV>Luis</DIV>
<DIV> </DIV>
<DIV>Regards,<BR>Luis<BR><BR><B><I>Alexei_Roudnev
<Alexei_Roudnev@exigengroup.com></I></B> wrote:</DIV>
<BLOCKQUOTE class=replbq
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid">
<META content="MSHTML 6.00.2800.1561" name=GENERATOR>
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<STYLE>@font-face {
font-family: Courier;
}
@font-face {
font-family: Tahoma;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline
}
A:visited {
COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlinkFollowed {
COLOR: blue; TEXT-DECORATION: underline
}
P.MsoPlainText {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
LI.MsoPlainText {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
DIV.MsoPlainText {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
P {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "Times New Roman"; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto
}
SPAN.EmailStyle18 {
COLOR: navy; FONT-FAMILY: Arial; mso-style-type: personal-reply
}
DIV.Section1 {
page: Section1
}
</STYLE>
<DIV><FONT face=Arial size=2>Did you tried vm.swappiness
parameter?</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>(/proc/sys/vm/swappiness)</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<BLOCKQUOTE
style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV style="FONT: 10pt arial">----- </DIV>
<DIV style="FONT: 10pt arial">Original Message ----- </DIV>
<DIV
style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B>
<A title=blsieler@gmail.com href="mailto:blsieler@gmail.com">Brian
Sieler</A> </DIV>
<DIV style="FONT: 10pt arial"><B>To:</B> <A
title=lfreitas34@yahoo.com href="mailto:lfreitas34@yahoo.com">'Luis
Freitas'</A> ; <A title=ocfs2-users@oss.oracle.com
href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</A>
</DIV>
<DIV style="FONT: 10pt arial"><B>Sent:</B> Sunday, April 08, 2007
10:52 PM</DIV>
<DIV style="FONT: 10pt arial"><B>Subject:</B> RE: [Ocfs2-users] High
on buffers and deep on swap</DIV>
<DIV><BR></DIV>
<DIV class=Section1>
<DIV class=MsoNormal><FONT face="Courier New" color=navy
size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">Luis,
yes I am experiencing what appears to be a similar problem you are
describing. See my post from just a few minutes ago on another
thread.<o:p></o:p></SPAN></FONT></DIV>
<DIV class=MsoNormal><FONT face="Courier New" color=navy
size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'"><o:p> </o:p></SPAN></FONT></DIV>
<DIV class=MsoPlainText><FONT face="Courier New" color=navy
size=2><SPAN style="FONT-SIZE: 10pt; COLOR: navy">I run a 2-node
cluster with OCFS2/RAC on </SPAN></FONT>2.6.9-34.0.2.ELsmp (RHEL
4.0) as well.<o:p></o:p></DIV>
<DIV class=MsoNormal><FONT face="Courier New" color=navy
size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'"><o:p> </o:p></SPAN></FONT></DIV>
<DIV class=MsoNormal><FONT face="Courier New" color=navy
size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">
total
used
free shared
buffers
cached<o:p></o:p></SPAN></FONT></DIV>
<DIV class=MsoNormal><FONT face="Courier New" color=navy
size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">Mem:
4044496 4005516
38980
0 34108
2236636<o:p></o:p></SPAN></FONT></DIV>
<DIV class=MsoNormal><FONT face="Courier New" color=navy
size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">-/+
buffers/cache: 1734772
2309724<o:p></o:p></SPAN></FONT></DIV>
<DIV class=MsoNormal><FONT face="Courier New" color=navy
size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">Swap:
2097144 648244
1448900<o:p></o:p></SPAN></FONT></DIV>
<DIV class=MsoNormal><FONT face="Courier New" color=navy
size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'"><o:p> </o:p></SPAN></FONT></DIV>
<DIV class=MsoNormal><FONT face="Courier New" color=navy
size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: 'Courier New'">If
you’ve uncovered anything since posting this message, please pass it
along?<o:p></o:p></SPAN></FONT></DIV>
<DIV class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></DIV>
<DIV>
<DIV class=MsoNormal style="TEXT-ALIGN: center" align=center><FONT
face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt">
<HR tabIndex=-1 align=center width="100%" SIZE=2>
</SPAN></FONT></DIV>
<DIV class=MsoNormal><B><FONT face=Tahoma size=2><SPAN
style="FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">From:</SPAN></FONT></B><FONT
face=Tahoma size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">
ocfs2-users-bounces@oss.oracle.com
[mailto:ocfs2-users-bounces@oss.oracle.com] <B><SPAN
style="FONT-WEIGHT: bold">On Behalf Of </SPAN></B>Luis
Freitas<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B>
Friday, February 23, 2007 5:32 PM<BR><B><SPAN
style="FONT-WEIGHT: bold">To:</SPAN></B>
ocfs2-users@oss.oracle.com<BR><B><SPAN
style="FONT-WEIGHT: bold">Subject:</SPAN></B> [Ocfs2-users] High on
buffers and deep on swap</SPAN></FONT><o:p></o:p></DIV></DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"><o:p> </o:p></SPAN></FONT></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt">Hi,<o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> This is a bit off topic, hope
there is not a problem.<o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> Anyone out there
experiencing high swapping with the kernel retaining a large
amount of buffers? This used to be a problem on 2.4, and I usually
changed /proc/sys/vm/freepages to fix it. But on 2.6 this parameter
no longer exists.<o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman"
size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> One of the servers here
is holding over 3.5Gb of cache even when using over 700Mb of swap,
and free memory is always low.<o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face=Courier size=3><SPAN
style="FONT-SIZE: 12pt; FONT-FAMILY: Courier">[oracle@br001sv0432
~]$
free<BR>
total
used
free shared
buffers
cached<BR>Mem:
5190736 4810880
379856
0 143032 3583868<BR>-/+
buffers/cache: 1083980
4106756<BR>Swap:
2048248 723064
1325184<BR>[oracle@br001sv0432
~]$</SPAN></FONT><o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face=Arial size=3><SPAN
style="FONT-SIZE: 12pt; FONT-FAMILY: Arial"> I am tuning
/proc/sys/vm/swappiness, but this seems to have no effect at all.
Changed from 60 to 10 and seems to have no effect. The server runs
Oracle RAC with OCFS2.</SPAN></FONT><o:p></o:p></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt">Regards,<o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt">Luis<o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></DIV>
<DIV class=MsoNormal style="TEXT-ALIGN: center" align=center><FONT
face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt">
<HR align=center width="100%" SIZE=1>
</SPAN></FONT></DIV>
<DIV class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"><A
href="http://answers.yahoo.com/dir/index;_ylc=X3oDMTFvbGNhMGE3BF9TAzM5NjU0NTEwOARfcwMzOTY1NDUxMDMEc2VjA21haWxfdGFnbGluZQRzbGsDbWFpbF90YWcx?link=ask&sid=396545367">Food
fight?</A> Enjoy some healthy debate<BR>in the <A
href="http://answers.yahoo.com/dir/index;_ylc=X3oDMTFvbGNhMGE3BF9TAzM5NjU0NTEwOARfcwMzOTY1NDUxMDMEc2VjA21haWxfdGFnbGluZQRzbGsDbWFpbF90YWcx?link=ask&sid=396545367">Yahoo!
Answers Food & Drink
Q&A.</A><o:p></o:p></SPAN></FONT></DIV></DIV>
<DIV>
<HR>
<DIV></DIV>_______________________________________________<BR>Ocfs2-users
mailing
list<BR>Ocfs2-users@oss.oracle.com<BR>http://oss.oracle.com/mailman/listinfo/ocfs2-users</DIV></BLOCKQUOTE></BLOCKQUOTE><BR>
<DIV>
<HR SIZE=1>
Be a PS3 game guru.<BR>Get your game face on with <A
href="http://us.rd.yahoo.com/evt=49936/*http://videogames.yahoo.com">the
latest PS3 news and previews at Yahoo!
Games.</A></DIV></BLOCKQUOTE></BLOCKQUOTE><BR>
<DIV>
<HR SIZE=1>
Finding fabulous fares is fun.<BR><A
href="http://farechase.yahoo.com/promo-generic-14795097;_ylc=X3oDMTFtNW45amVpBF9TAzk3NDA3NTg5BF9zAzI3MTk0ODEEcG9zAzEEc2VjA21haWx0YWdsaW5lBHNsawNxMS0wNw--">Let
Yahoo! FareChase search your favorite travel sites</A> to find flight and
hotel bargains.</DIV></BLOCKQUOTE></BLOCKQUOTE><BR>
<P>
<HR SIZE=1>
<A href="http://us.rd.yahoo.com/evt=49935/*http://games.yahoo.com">Bored
stiff?</A> Loosen up...<BR><A
href="http://us.rd.yahoo.com/evt=49935/*http://games.yahoo.com">Download and
play hundreds of games for free</A> on Yahoo! Games.
<DIV><P><HR>
SEEBURGER AG <BR>
Headquarters:<BR>
Edisonstraße 1 <BR>
D-75015 Bretten <BR>
Tel.: 0 72 52/96-0 <BR>
Fax: 0 72 52/96-2222 <BR>
Internet: http://www.seeburger.de <BR>
e-mail: info@seeburger.de <BR>
<BR>
Vorstand:<BR>
Bernd Seeburger, Axel Haas, Michael Kleeberg<BR>
<BR>
Vorsitzender des Aufsichtsrats:<BR>
Dr. Franz Scherer<BR>
<BR>
Handelsregister:<BR>
HRB 240708 Mannheim
</P></DIV>
</BODY></HTML>