<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Hi
Joel,<o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Many
thanks for the reply – actually, I did submit a few problems regarding
OCFS2 1.2.1, but there weren’t actually helpful thoughts regarding this
fencing mechanism which followed by panic and node hang making the entire RAC
unavailable.<o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Having
an SLA to maintain, the customer is now willing to replace OCFS2. From the
feedback I received from Oracle support, ASM seems to be a better alternative in
terms of stability under heavy load. <o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Anyway,
you’re right about all the aspects in the first point, we’ve got a
good RAID6 environment - so basically, unless there are potential
performance issues, I would maintain ASM with external redundancy. The storage
seems to be reliable, 5 corrupted disks in one year (total number of disks is 164
on both SANs) – nothing went down as a RAID6 disk group can survive
with up to 2 failed disks. <o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>One
question here is how to make sure the LUNs are aligned for ASM? Do you have any
guidelines on this? The LUNs are created using the SAN console and the OS recognize
them as partitions once these LUNs are assigned to the node(s). There is no
intervention with respect to the partitioning process on the OS level (each LUN
is one partition). <o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>As
per the second point, the only reason I would prefer to use /dev/mapper/* is
because dm changes if the assigned LUN list is altered, whilst /dev/mapper/* doesn’t
change. If ASMLib would really be an added value to the setup, then I will manage
with dm.<o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> <o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>We’ve
got AI64 SLES9 SP3; do you recommend any OS upgrade prior installing
anything? <o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>This
scanning code seems to be interesting, I’d appreciate sharing the
details.<o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Thanks
and best regards,<o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Karim
<o:p></o:p></span></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>-----Original Message-----<br>
From: Joel Becker [mailto:Joel.Becker@oracle.com] <br>
Sent: Wednesday, June 17, 2009 8:02 PM<br>
To: Karim Alkhayer<br>
Cc: oracleasm-users@oss.oracle.com<br>
Subject: Re: [Oracleasm-users] Shifting from OCFS2 to ASM</p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>On Wed, Jun 17, 2009 at 03:44:04AM +0300, Karim Alkhayer
wrote:<o:p></o:p></p>
<p class=MsoPlainText>> We're considering ASM with Oracle Clusterware
10.2.0.4 as a replacement to<o:p></o:p></p>
<p class=MsoPlainText>> OCFS2 in order to maintain a more reliable RAC.
We've been hitting stability<o:p></o:p></p>
<p class=MsoPlainText>> issues with our 2 node RAC running IA64 SLES9 SP3
and Oracle 10.1 and OCFS2<o:p></o:p></p>
<p class=MsoPlainText>> 1.2.1 - As the upgrade options are limited, ASM
seems to be a better<o:p></o:p></p>
<p class=MsoPlainText>> alternative and been recommended by Oracle support
as well.<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText> Wearing my ocfs2 hat, I
have to wonder if we've heard from you<o:p></o:p></p>
<p class=MsoPlainText>about these stability issues. 1.2 with 2 nodes
should be very stable,<o:p></o:p></p>
<p class=MsoPlainText>and just as viable as ASM.<o:p></o:p></p>
<p class=MsoPlainText> Now putting on my
oracleasm hat, because using ASM or ocfs2 or<o:p></o:p></p>
<p class=MsoPlainText>both (both is quite usable too) is up to you in the end.<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>> 1. The primary RAC is
hosting 5 databases with around 500 GB divided<o:p></o:p></p>
<p class=MsoPlainText>> into a few disk groups with RAID 6 hosted in two
SANs; these disk groups are<o:p></o:p></p>
<p class=MsoPlainText>> composed of disks of 36GB with 15K RPM and 146GB
with 10K RPM. I'd like to<o:p></o:p></p>
<p class=MsoPlainText>> know if we need to break the RAID6. If so, what is
the best recommended RAID<o:p></o:p></p>
<p class=MsoPlainText>> type? <o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>[ Caveat - I don't work on ASM, this mailing list is for
oracleasm, the<o:p></o:p></p>
<p class=MsoPlainText> Linux ASMLib. You may want to contact Oracle
support to get more<o:p></o:p></p>
<p class=MsoPlainText> direct communication with the ASM folks. ]<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText> You don't need to break
the raid6. The question is whether you<o:p></o:p></p>
<p class=MsoPlainText>want to. I assume that you don't mix your disks -
that some raid6<o:p></o:p></p>
<p class=MsoPlainText>groups are 15K RPM disks and other raid6 groups are 10K
RPM disks. I<o:p></o:p></p>
<p class=MsoPlainText>further assume that they are on some sort of smart
storage with a cache.<o:p></o:p></p>
<p class=MsoPlainText> I would start by keeping
the raid6 LUNs. When you give them<o:p></o:p></p>
<p class=MsoPlainText>to ASM, make sure that they are aligned. ASM
stripes on 1MB boundaries.<o:p></o:p></p>
<p class=MsoPlainText>So make sure that the partitions you create on these LNUs
start at an<o:p></o:p></p>
<p class=MsoPlainText>appropriate stripe boundary on the array. That way,
ASM's stripes line<o:p></o:p></p>
<p class=MsoPlainText>up with the array's stripes. Get this wrong and the
performance<o:p></o:p></p>
<p class=MsoPlainText>suffers.<o:p></o:p></p>
<p class=MsoPlainText> If you have a good raid6
environment, you probably want to use<o:p></o:p></p>
<p class=MsoPlainText>'external redundancy' when creating your
diskgroups. That's telling ASM<o:p></o:p></p>
<p class=MsoPlainText>not to handle redundancy because you trust your storage
to do it.<o:p></o:p></p>
<p class=MsoPlainText> See how the performance is
(I expect it to be fine). If there<o:p></o:p></p>
<p class=MsoPlainText>is a problem, the ASM team and Oracle support can help
you figure it out<o:p></o:p></p>
<p class=MsoPlainText>better.<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>> 2. We've got consistent
shared LUN listing across the cluster, an<o:p></o:p></p>
<p class=MsoPlainText>> example is /dev/mapper/2003013840ad20008 (we're
using this method to create<o:p></o:p></p>
<p class=MsoPlainText>> and mount OCFS2 volumes); is it possible to maintain
the same way of<o:p></o:p></p>
<p class=MsoPlainText>> discovery? There is a note in "Configuring
Oracle ASMLib on Multipath Disks"<o:p></o:p></p>
<p class=MsoPlainText>> saying that dm prefix must be used for the scan
order and exclude. Do we<o:p></o:p></p>
<p class=MsoPlainText>> still need ASMLib, and can we use the device mapper
instead of dm prefix?<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText> Here I can help. dm
is 'device mapper'. It's just that the<o:p></o:p></p>
<p class=MsoPlainText>current scan code relies on /proc/partitions, which
displays 'dm', not<o:p></o:p></p>
<p class=MsoPlainText>'mapper/'. The mapper/ names are just aliases for
the dm devices.<o:p></o:p></p>
<p class=MsoPlainText> Are your dm devices just
multipaths? Or do you do other things?<o:p></o:p></p>
<p class=MsoPlainText>I'm assuming multipath, as ASM will do the LVM for you
:-)<o:p></o:p></p>
<p class=MsoPlainText> You can, if you like, just
use regular paths and no ASMLib for<o:p></o:p></p>
<p class=MsoPlainText>your ASM devices. Make your diskstring
"/dev/mapper/*" or something<o:p></o:p></p>
<p class=MsoPlainText>like it. This will use POSIX I/O ops on your disks
instead of ASMLib.<o:p></o:p></p>
<p class=MsoPlainText>However, you lose the benefits of ASMLib.<o:p></o:p></p>
<p class=MsoPlainText> If you want to use ASMLib,
you can easily make it work with your<o:p></o:p></p>
<p class=MsoPlainText>multipath devices. First, if you are on RHEL5,
comment out the<o:p></o:p></p>
<p class=MsoPlainText>following line in /etc/udev/rules.d/50-udev.rules:<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>#KERNEL=="dm-[0-9]*", ACTION=="add",
OPTIONS+="ignore_device"<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Next, set 'dm' in your SCANORDER as specified in the
document on<o:p></o:p></p>
<p class=MsoPlainText>multipath configuration. This will detect your
device mapper multipath<o:p></o:p></p>
<p class=MsoPlainText>devices. Tada!<o:p></o:p></p>
<p class=MsoPlainText> If you have any furthur
problems, I have fancier scanning code<o:p></o:p></p>
<p class=MsoPlainText>in the works, and you could try it out if you like.<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Joel<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>-- <o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>"The nearest approach to immortality on Earth is a
government<o:p></o:p></p>
<p class=MsoPlainText> bureau."<o:p></o:p></p>
<p class=MsoPlainText> - James F. Byrnes<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Joel Becker<o:p></o:p></p>
<p class=MsoPlainText>Principal Software Developer<o:p></o:p></p>
<p class=MsoPlainText>Oracle<o:p></o:p></p>
<p class=MsoPlainText>E-mail: joel.becker@oracle.com<o:p></o:p></p>
<p class=MsoPlainText>Phone: (650) 506-8127<o:p></o:p></p>
</div>
</body>
</html>