<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE></TITLE>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.5512" name=GENERATOR></HEAD>
<BODY text=#000000 bgColor=#ffffff>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011>So, on the configuration using sdb directly, the
heartbeat device is also sdb. On the configuration using dm-multipath, it
is as follows: -</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face="Courier New" color=#0000ff size=2><SPAN
class=855425807-27072011># mount | grep fatwire<BR>/dev/mapper/fatwire on
/fatwire/shared type ocfs2 (rw,_netdev,heartbeat=local)</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011>The mount is using
/dev/mapper/fatwire</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011><FONT face="Courier New"># dmsetup ls | grep
fatwire<BR>fatwire (253, 13)</FONT><BR></SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011>The minor number for the fatwire device-mapper device
is 13</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011> </DIV></SPAN></FONT>
<DIV dir=ltr align=left><FONT face="Courier New" color=#0000ff size=2><SPAN
class=855425807-27072011># cat
/sys/kernel/config/cluster/ocfs2/heartbeat/E47A8B6EAEDC4BF1A28A7150D659C45F/dev<BR>dm-13</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=855425807-27072011>The
heartbeat is also using device-mapper device 13. So the heartbeat and
filesystem are using the same device in this case, but not deadlocking. It
looks as though the way device-mapper handles locks on devices is different
somehow to the way the underlying devices themselves are
handled?</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=855425807-27072011>Simon</DIV></SPAN></FONT>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV align=left>
<DIV
style="FONT-SIZE: 100%; COLOR: rgb(0,0,0); FONT-FAMILY: tahoma,verdana,sans-serif"
align=left><FONT style="COLOR: rgb(0,0,0)" face=tahoma,verdana,sans-serif
size=2>- <BR>Simon Hargrave <A
title="blocked::blocked::mailto:szhargrave@ybs.co.uk blocked::mailto:szhargrave@ybs.co.uk mailto:szhargrave@ybs.co.uk"
href="blocked::blocked::blocked::mailto:szhargrave@ybs.co.uk">szhargrave@ybs.co.uk</A> <BR>Enterprise
Systems Team Leader x2831</FONT><FONT style="COLOR: rgb(0,0,0)"
face=tahoma,verdana,sans-serif size=2><BR>Yorkshire Building Society 01274
472831</FONT></DIV>
<DIV
style="FONT-SIZE: 100%; COLOR: rgb(0,0,0); FONT-FAMILY: tahoma,verdana,sans-serif"
align=left><FONT face=Arial size=2><A title=http://wwwtech/sysint/tsgcore.asp
href="blocked::http://wwwtech/sysint/tsgcore.asp">http://wwwtech/sysint/tsgcore.asp</A></FONT></DIV></DIV>
<DIV> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Sunil Mushran
[mailto:sunil.mushran@oracle.com] <BR><B>Sent:</B> 26 July 2011
18:16<BR><B>To:</B> Simon Hargrave<BR><B>Cc:</B>
ocfs2-users@oss.oracle.com<BR><B>Subject:</B> Re: [Ocfs2-users] OCFS2 unmount
problems after online resize<BR></FONT><BR></DIV>
<DIV></DIV>cat
/sys/kernel/config/cluster/CLUSTERNAME/heartbeat/HBREGION/dev<BR><BR>Mount and
run the following. Replace CLUSTERNAME and HBREGION<BR>appropriately.<BR><BR>On
07/26/2011 01:12 AM, Simon Hargrave wrote:
<BLOCKQUOTE cite=mid:83B9398A54BE50429CD78BB4800885E710A6C319@EX-MB-VS2.YBS.com
type="cite">
<META content="MSHTML 6.00.2900.5512" name=GENERATOR>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2>Thanks Sunil</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2>Interestingly I seem to have found a potential
workaround.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2>Using sdb directly causes the problem I described.
However, if I allow sdb to be subsumed by dm-multipath and use the multipath
device directly, no such deadlocking occurs.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2>For example: -</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2> * Allow dm-multipath to subsume the paths (6 in
the case of a physical server, only 1 in the case of a VMware guest) under
/dev/mapper/mpathX</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2> * mkfs on the /dev/map/mpathX
device</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2> * Mount the filesytem</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2> * Resize the LUN</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2> * echo "1" > /dev/block/sdX/device/rescan for
each path for the multipath device</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2> * multipathd -k"resize map
mpathX"</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2> * tune2fs to resize the
filesystem</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2>I can then unmount the filesystem without issue.
</FONT></SPAN><SPAN class=234474807-26072011><FONT face=Arial color=#0000ff
size=2>The only problem with this approach seems to be I cannot mount by
label, since it tries to use the sdX devices so I have to specify the mpath
device directly - not really an issue since they are persistent across reboots
due to /var/lib/multipath/bindings.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2>Can you explain why the deadlock issue doesn't occur when
using dm-multipath?</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2>Simon</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=234474807-26072011><FONT face=Arial
color=#0000ff size=2>Incidentally, it also appears that when using
dm-multipath, I can actually use a partition rather than the raw device - I
can fdisk the mpath0 and rerun partprive and kpartx -a to reread the partition
table into the kernel and resize the partition. This cannot be done when
using the partition directly.</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV align=left>
<DIV
style="FONT-SIZE: 100%; COLOR: rgb(0,0,0); FONT-FAMILY: tahoma,verdana,sans-serif"
align=left><FONT style="COLOR: rgb(0,0,0)" face=tahoma,verdana,sans-serif
size=2>- <BR>Simon Hargrave <A
title="blocked::blocked::mailto:szhargrave@ybs.co.uk blocked::mailto:szhargrave@ybs.co.uk mailto:szhargrave@ybs.co.uk"
href="blocked::blocked::blocked::mailto:szhargrave@ybs.co.uk"
moz-do-not-send="true">szhargrave@ybs.co.uk</A> <BR>Enterprise
Systems Team Leader x2831</FONT><FONT style="COLOR: rgb(0,0,0)"
face=tahoma,verdana,sans-serif size=2><BR>Yorkshire Building Society 01274
472831</FONT></DIV>
<DIV
style="FONT-SIZE: 100%; COLOR: rgb(0,0,0); FONT-FAMILY: tahoma,verdana,sans-serif"
align=left><FONT face=Arial size=2><A title=http://wwwtech/sysint/tsgcore.asp
href="blocked::http://wwwtech/sysint/tsgcore.asp"
moz-do-not-send="true">http://wwwtech/sysint/tsgcore.asp</A></FONT></DIV></DIV>
<DIV> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Sunil Mushran [<A
class=moz-txt-link-freetext
href="mailto:sunil.mushran@oracle.com">mailto:sunil.mushran@oracle.com</A>]
<BR><B>Sent:</B> 25 July 2011 19:22<BR><B>To:</B> Simon Hargrave<BR><B>Cc:</B>
<A class=moz-txt-link-abbreviated
href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</A><BR><B>Subject:</B>
Re: [Ocfs2-users] OCFS2 unmount problems after online
resize<BR></FONT><BR></DIV>The umount and the hb stop threads are deadlocking
on the s_umount lock.<BR><BR>This problem is due to the local heartbeat scheme
employed in which the<BR>hb device is the same as the mounted one. umount
trigger hb stop which<BR>calls open() => ... => rescan_partitions()
=> ... => get_super() => down_read().<BR>The same lock should be held
by the umount thread.<BR><BR>Unfortunately there is no fix for this other than
using a different hb scheme.<BR><BR>Later this year we will be releasing
global heartbeat as part of the o2cb stack<BR>that will allow users to specify
different hb devices.<BR><BR>Another option is to move to sles11 and make use
of the pacemaker cluster<BR>stack.<BR><BR>On 07/25/2011 06:58 AM, Simon
Hargrave wrote:
<BLOCKQUOTE
cite=mid:83B9398A54BE50429CD78BB4800885E70FAEE94E@EX-MB-VS2.YBS.com
type="cite">
<META content="MSHTML 6.00.2900.5512" name=GENERATOR>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011>A further update, which simplifies the
situation. It appears to be more fundamental, and not actually
anything to do with the online resize.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011>Basically it appears that simply the act of
resizing the LUN and performing the scsi rescan is enough to make the next
unmount fail, i.e.:-</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011> * create filesystem</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011> * mount filesystem</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011> * unmounts and mounts
fine</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011> * extend LUN on storage</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011> * echo 1 to
/sys/block/sdb/device/rescan</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011> * unmount filesystem, which
hangs</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011>The above happens even if only one node is in the
cluster, so it doesn't appear to be a locking issue between the
hosts.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011>I have tried exactly the same with ext3 (one node
obviously!) and the same resize doesn't cause a hang.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011>I have also configure ocfs on a single physical
machine (to rule out VMware), and the symptoms are
identical.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011>So for whatever reason, the system call to umount()
for an ocfs2 filesystem hangs if the underlying block device has changed
size?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011>Simon</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=533375413-25072011></SPAN></FONT> </DIV>
<DIV align=left>
<DIV
style="FONT-SIZE: 100%; COLOR: rgb(0,0,0); FONT-FAMILY: tahoma,verdana,sans-serif"
align=left><FONT style="COLOR: rgb(0,0,0)" face=tahoma,verdana,sans-serif
size=2>- <BR>Simon Hargrave <A
title="blocked::blocked::mailto:szhargrave@ybs.co.uk blocked::mailto:szhargrave@ybs.co.uk mailto:szhargrave@ybs.co.uk"
href="blocked::blocked::blocked::mailto:szhargrave@ybs.co.uk"
moz-do-not-send="true">szhargrave@ybs.co.uk</A> <BR>Enterprise
Systems Team Leader x2831</FONT><FONT style="COLOR: rgb(0,0,0)"
face=tahoma,verdana,sans-serif size=2><BR>Yorkshire Building Society 01274
472831</FONT></DIV>
<DIV
style="FONT-SIZE: 100%; COLOR: rgb(0,0,0); FONT-FAMILY: tahoma,verdana,sans-serif"
align=left><FONT face=Arial size=2><A
title=http://wwwtech/sysint/tsgcore.asp
href="blocked::http://wwwtech/sysint/tsgcore.asp"
moz-do-not-send="true">http://wwwtech/sysint/tsgcore.asp</A></FONT></DIV></DIV>
<DIV> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> <A class=moz-txt-link-abbreviated
href="mailto:ocfs2-users-bounces@oss.oracle.com"
moz-do-not-send="true">ocfs2-users-bounces@oss.oracle.com</A> [<A
class=moz-txt-link-freetext href="mailto:ocfs2-users-bounces@oss.oracle.com"
moz-do-not-send="true">mailto:ocfs2-users-bounces@oss.oracle.com</A>] <B>On
Behalf Of </B>Simon Hargrave<BR><B>Sent:</B> 25 July 2011
13:50<BR><B>To:</B> <A class=moz-txt-link-abbreviated
href="mailto:ocfs2-users@oss.oracle.com"
moz-do-not-send="true">ocfs2-users@oss.oracle.com</A><BR><B>Subject:</B> Re:
[Ocfs2-users] OCFS2 unmount problems after online
resize<BR></FONT><BR></DIV>
<DIV dir=ltr align=left><SPAN class=500464912-25072011><FONT face=Arial
color=#0000ff size=2>Further to this, I get the following in dmesg every 120
seconds after the attempted unmount: -</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=500464912-25072011></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=500464912-25072011><FONT face=Arial
color=#0000ff size=2>INFO: task ocfs2_hb_ctl:3794 blocked for more than 120
seconds.<BR>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.<BR>ocfs2_hb_ctl D
ffff810003db6420 0 3794
3793
(NOTLB)<BR> ffff8100b9d05cf8 0000000000000086 00000000f000020a
ffffffff8002d0ee<BR> 0000000000000000 0000000000000007 ffff8100d801e820
ffffffff80310b60<BR> 000000887c712d88 000000000000791a ffff8100d801ea08
0000000080009852<BR>Call Trace:<BR> [<ffffffff8002d0ee>]
wake_up_bit+0x11/0x22<BR> [<ffffffff8006466c>]
__down_read+0x7a/0x92<BR> [<ffffffff800e68aa>]
get_super+0x48/0x95<BR> [<ffffffff800e387b>]
fsync_bdev+0xe/0x3b<BR> [<ffffffff8014a6f8>]
invalidate_partition+0x28/0x40<BR> [<ffffffff8010d6e7>]
rescan_partitions+0x37/0x279<BR> [<ffffffff800e78ec>]
do_open+0x231/0x30f<BR> [<ffffffff800e7c1e>]
blkdev_open+0x0/0x4f<BR> [<ffffffff800e7c41>]
blkdev_open+0x23/0x4f<BR> [<ffffffff8001eab6>]
__dentry_open+0xd9/0x1dc<BR> [<ffffffff8002751f>]
do_filp_open+0x2a/0x38<BR> [<ffffffff8002ae16>]
iput+0x4b/0x84<BR> [<ffffffff800dddf3>]
alternate_node_alloc+0x70/0x8c<BR> [<ffffffff80019f7e>]
do_sys_open+0x44/0xbe<BR> [<ffffffff8005d28d>]
tracesys+0xd5/0xe0<BR></FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=500464912-25072011></SPAN> </DIV>
<DIV align=left>
<DIV
style="FONT-SIZE: 100%; COLOR: rgb(0,0,0); FONT-FAMILY: tahoma,verdana,sans-serif"
align=left><FONT style="COLOR: rgb(0,0,0)" face=tahoma,verdana,sans-serif
size=2>- <BR>Simon Hargrave <A
title="blocked::blocked::mailto:szhargrave@ybs.co.uk blocked::mailto:szhargrave@ybs.co.uk mailto:szhargrave@ybs.co.uk"
href="blocked::blocked::blocked::mailto:szhargrave@ybs.co.uk"
moz-do-not-send="true">szhargrave@ybs.co.uk</A> <BR>Enterprise
Systems Team Leader x2831</FONT><FONT style="COLOR: rgb(0,0,0)"
face=tahoma,verdana,sans-serif size=2><BR>Yorkshire Building Society 01274
472831</FONT></DIV>
<DIV
style="FONT-SIZE: 100%; COLOR: rgb(0,0,0); FONT-FAMILY: tahoma,verdana,sans-serif"
align=left><FONT face=Arial size=2><A
title=http://wwwtech/sysint/tsgcore.asp
href="blocked::http://wwwtech/sysint/tsgcore.asp"
moz-do-not-send="true">http://wwwtech/sysint/tsgcore.asp</A></FONT></DIV></DIV>
<DIV> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> <A class=moz-txt-link-abbreviated
href="mailto:ocfs2-users-bounces@oss.oracle.com"
moz-do-not-send="true">ocfs2-users-bounces@oss.oracle.com</A> [<A
class=moz-txt-link-freetext href="mailto:ocfs2-users-bounces@oss.oracle.com"
moz-do-not-send="true">mailto:ocfs2-users-bounces@oss.oracle.com</A>] <B>On
Behalf Of </B>Simon Hargrave<BR><B>Sent:</B> 25 July 2011
13:26<BR><B>To:</B> <A class=moz-txt-link-abbreviated
href="mailto:ocfs2-users@oss.oracle.com"
moz-do-not-send="true">ocfs2-users@oss.oracle.com</A><BR><B>Subject:</B>
[Ocfs2-users] OCFS2 unmount problems after online
resize<BR></FONT><BR></DIV>
<P><SPAN style="FONT-SIZE: 8.2pt; FONT-FAMILY: 'MS Sans Serif'">Please read
the warning at the end of this email<BR></SPAN><SPAN
style="FONT-SIZE: 8pt; FONT-FAMILY: 'Arial'">________________________________________________</SPAN></P>
<P><SPAN style="FONT-SIZE: 8pt; FONT-FAMILY: 'Arial'"></SPAN> </P>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Hi</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>I'm doing some experimentation with OCFS2 (1.4 on RHEL5) with a view
to using as a 2-node clustered filesystem. I seem to be having issues
with online resize (which documentation suggests is supported under
1.4). I'm creating a LUN and publishing from a HP EVA6400 storage
array to the 2 nodes, and creating a filesystem which works fine.
However, it appears that if I online-increase the size of the LUN and
subsequently the filesystem, it hangs indefinately on unmount. Full
transcript of issue is as below: -</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>/etc/ocfs2/cluster.conf (created via
ocfs2console)</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>--------------------------------------------------</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>node:<BR> ip_port =
7777<BR> ip_address =
10.34.8.90<BR> number =
0<BR> name =
ybsxlx45<BR> cluster =
ocfs2</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>node:<BR> ip_port =
7777<BR> ip_address =
10.34.8.91<BR> number =
1<BR> name =
ybsxlx46<BR> cluster =
ocfs2</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>cluster:<BR> node_count =
2<BR> name =
ocfs2</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>/etc/sysconfig/o2cb (created via ocfs2console)</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>----------------------------------------------</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># O2CB_ENABLED: 'true' means to load the driver on
boot.<BR>O2CB_ENABLED=true</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># O2CB_STACK: The name of the cluster stack backing
O2CB.<BR>O2CB_STACK=o2cb</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># O2CB_BOOTCLUSTER: If not empty, the name of a cluster to
start.<BR>O2CB_BOOTCLUSTER=ocfs2</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered
dead.<BR>O2CB_HEARTBEAT_THRESHOLD=</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># O2CB_IDLE_TIMEOUT_MS: Time in ms before a network connection is
considered dead.<BR>O2CB_IDLE_TIMEOUT_MS=</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># O2CB_KEEPALIVE_DELAY_MS: Max time in ms before a keepalive packet
is sent<BR>O2CB_KEEPALIVE_DELAY_MS=</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face=Arial size=2><FONT
face="Courier New"># O2CB_RECONNECT_DELAY_MS: Min time in ms between
connection
attempts<BR>O2CB_RECONNECT_DELAY_MS=</FONT><BR></FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>2GB LUN published to both nodes and appears as
/dev/sdb</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>-------------------------------------------------------</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># grep sdb /proc/partitions<BR> 8
16 2097152 sdb</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Operating System</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>----------------</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Red Hat Enterprise Linux Server release 5.6 (Tikanga)<BR>Linux
ybsxlx45 2.6.18-238.1.1.el5 #1 SMP Tue Jan 4 13:32:19 EST 2011 x86_64 x86_64
x86_64 GNU/Linux</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>OCFS2 Packages</FONT></SPAN></DIV>
<DIV><FONT face="Courier New"><FONT size=2><SPAN
class=502530112-25072011>--------------</SPAN></FONT></FONT></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>ocfs2-2.6.18-238.1.1.el5-1.4.7-1.el5<BR>ocfs2console-1.4.4-1.el5<BR>ocfs2-tools-1.4.4-1.el5</FONT></SPAN><SPAN
class=502530112-25072011></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT size=2><FONT
face="Courier New">Create and exercise filesystem</FONT></FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT size=2><FONT
face="Courier New">------------------------------</FONT></FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># mkfs.ocfs2 -L "ocfstest" /dev/sdb<BR># mount -L ocfstest
/ocfstest<BR># dd if=/dev/zero of=/ocfstest/file1 bs=1024k count=500 (on
first node)<BR># dd if=/dev/zero of=/ocfstest/file2 bs=1024k count=500 (on
second node)</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># df -k
/ocfstest<BR>Filesystem
1K-blocks Used Available Use% Mounted
on<BR>/dev/sdb
2097152 1320836 776316 63%
/ocfstest</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Test unmount and remount</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>------------------------</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># strace -f -o before.txt umount /ocfstest<BR># mount -L ocfstest
/ocfstest</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>LUN resized to 3GB and rescan on each host</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>------------------------------------------</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># echo "1" > /sys/block/sdb/device/rescan<BR># grep sdb
/proc/partitions<BR> 8 16
3145728 sdb</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>(new device size showing)</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Online resize of filesystem</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>---------------------------</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># df -k
/ocfstest<BR>Filesystem
1K-blocks Used Available Use% Mounted
on<BR>/dev/sdb
2097152 1312644 784508 63%
/ocfstest</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># tunefs.ocfs2 -S /dev/sdb</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># df -k
/ocfstest<BR>Filesystem
1K-blocks Used Available Use% Mounted
on<BR>/dev/sdb
3145728 1312676 1833052 42%
/ocfstest</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>(new filesystem size shows on both nodes)</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Exercise filesystem</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>-------------------</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># dd if=/dev/zero of=/ocfstest/file3 bs=1024k count=500 (on first
node)<BR># dd if=/dev/zero of=/ocfstest/file4 bs=1024k count=500 (on second
node)</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># df -k
/ocfstest<BR>Filesystem
1K-blocks Used Available Use% Mounted
on<BR>/dev/sdb
3145728 2340772 804956 75%
/ocfstest</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>(filesystem continues to function and can be filled past old
size)</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Unmount filesystem</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>------------------</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2># strace -f -o after.txt umount /ocfstest</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>At this point, the unmount hangs forever and only a reboot will clear
it.</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Comparing the "strace" output, the second one hangs during the call
to umount() system call, after having checked that umount.ocfs2 doesn't
exist.</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Whilst hung, the filesystem still "appears" in /etc/mtab and df
output, but it is not mounted according to the kernel
(/proc/mounts).</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Other node continues to function whilst in this state, filesystem
does not hang.</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>So the question is, is this a bug, or am I doing something
wrong? The OCFS2 1.4 user guide does state: -</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>9. Online File system Resize<BR>Users can now grow the file system
without having to unmount it. This feature requires a compatible clustered
logical volume manager. Compatible volumes managers will be announced when
support is available.</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>However since I'm using the raw device, not LVM this should work,
provided the scsi device rescan has been performed on all nodes prior to
running tunefs.ocfs2?</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face=Arial><FONT
face="Courier New" size=2>I should finally point out that this is being
performed on 2 VMware guests, but the LUN is published directly to the
guests as a Raw Device Mapping in Physical Compatibility Mode (passthru), as
per the various VMware whitepapers. I don't have 2 spare SAN-attached
crash-and-burn hosts to test this out physically, but I don't believe this
should be a factor.</FONT></FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face=Arial><FONT
face="Courier New" size=2>Any help appreciated as online resize is a must in
a 24x7 clustered environment!</FONT></FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN> </DIV>
<DIV><SPAN class=502530112-25072011><FONT face="Courier New"
size=2>Thanks</FONT></SPAN></DIV>
<DIV><SPAN class=502530112-25072011></SPAN><SPAN
class=502530112-25072011></SPAN> </DIV>
<DIV align=left>
<DIV
style="FONT-SIZE: 100%; COLOR: rgb(0,0,0); FONT-FAMILY: tahoma,verdana,sans-serif"
align=left><FONT style="COLOR: rgb(0,0,0)" face=tahoma,verdana,sans-serif
size=2>- <BR>Simon Hargrave <A
title="blocked::blocked::mailto:szhargrave@ybs.co.uk blocked::mailto:szhargrave@ybs.co.uk mailto:szhargrave@ybs.co.uk"
href="blocked::blocked::blocked::mailto:szhargrave@ybs.co.uk"
moz-do-not-send="true">szhargrave@ybs.co.uk</A> <BR>Enterprise
Systems Team Leader x2831</FONT><FONT style="COLOR: rgb(0,0,0)"
face=tahoma,verdana,sans-serif size=2><BR>Yorkshire Building Society 01274
472831</FONT></DIV>
<DIV
style="FONT-SIZE: 100%; COLOR: rgb(0,0,0); FONT-FAMILY: tahoma,verdana,sans-serif"
align=left><FONT face=Arial size=2><A
title=http://wwwtech/sysint/tsgcore.asp
href="blocked::http://wwwtech/sysint/tsgcore.asp"
moz-do-not-send="true">http://wwwtech/sysint/tsgcore.asp</A></FONT></DIV></DIV>
<DIV> </DIV>
<P><SPAN
style="FONT-SIZE: 8pt; FONT-FAMILY: 'Arial'">________________________________________________</SPAN></P>
<P><SPAN style="FONT-SIZE: 8pt; FONT-FAMILY: 'Arial'">This email and any
attachments are confidential and may contain privileged
information.</SPAN></P>
<P><SPAN style="FONT-SIZE: 8pt; FONT-FAMILY: 'Arial'">If you are not the
person for whom they are intended please return the email and then delete
all material from any computer. You must not use the email or attachments
for any purpose, nor disclose its contents to anyone other than the intended
recipient.</SPAN></P>
<P><SPAN style="FONT-SIZE: 8pt; FONT-FAMILY: 'Arial'">Any statements made by
an individual in this email do not necessarily reflect the views of the
Yorkshire Building Society Group.</SPAN></P>
<P><SPAN
style="FONT-SIZE: 8pt; FONT-FAMILY: 'Arial'">________________________________________________<BR><BR>Yorkshire
Building Society, which is authorised and regulated by the Financial
Services Authority, chooses to introduce its customers to Legal &
General for the purposes of advising on and arranging life assurance and
investment products bearing Legal & General’s name.</SPAN></P>
<P><SPAN style="FONT-SIZE: 8pt; FONT-FAMILY: 'Arial'"><BR>We are entered in
the FSA Register and our FSA registration number is 106085 <A
class=moz-txt-link-freetext href="http://www.fsa.gov.uk/register"
moz-do-not-send="true">http://www.fsa.gov.uk/register</A><BR><BR>Head
Office: Yorkshire Building Society, Yorkshire House, Yorkshire Drive,
Bradford, BD5 8LJ<BR>Tel: 0845 1 200 100<BR><BR>Visit Our Website<BR><A
class=moz-txt-link-freetext href="http://www.ybs.co.uk"
moz-do-not-send="true">http://www.ybs.co.uk</A><BR><BR>All communications
with us may be monitored/recorded to improve the quality of our service and
for your protection and security.<BR></SPAN></P>
<P><SPAN
style="FONT-SIZE: 8pt; FONT-FAMILY: 'Arial'"></SPAN> </P><BR>________________________________________________________________________<BR>This
e-mail has been scanned for all viruses by Star. The<BR>service is powered
by MessageLabs. For more information on a proactive<BR>anti-virus service
working around the clock, around the globe, visit:<BR><A
class=moz-txt-link-freetext href="http://www.star.net.uk"
moz-do-not-send="true">http://www.star.net.uk</A><BR>________________________________________________________________________<BR><BR>________________________________________________________________________<BR>This
e-mail has been scanned for all viruses by Star. The<BR>service is powered
by MessageLabs. For more information on a proactive<BR>anti-virus service
working around the clock, around the globe, visit:<BR><A
class=moz-txt-link-freetext href="http://www.star.net.uk"
moz-do-not-send="true">http://www.star.net.uk</A><BR>________________________________________________________________________<BR><BR>________________________________________________________________________<BR>This
e-mail has been scanned for all viruses by Star. The<BR>service is powered
by MessageLabs. For more information on a proactive<BR>anti-virus service
working around the clock, around the globe, visit:<BR><A
class=moz-txt-link-freetext href="http://www.star.net.uk"
moz-do-not-send="true">http://www.star.net.uk</A><BR>________________________________________________________________________<BR><BR>________________________________________________________________________<BR>This
e-mail has been scanned for all viruses by Star. The<BR>service is powered
by MessageLabs. For more information on a proactive<BR>anti-virus service
working around the clock, around the globe, visit:<BR><A
class=moz-txt-link-freetext href="http://www.star.net.uk"
moz-do-not-send="true">http://www.star.net.uk</A><BR>________________________________________________________________________<BR><BR>________________________________________________________________________<BR>This
e-mail has been scanned for all viruses by Star. The<BR>service is powered
by MessageLabs. For more information on a proactive<BR>anti-virus service
working around the clock, around the globe, visit:<BR><A
class=moz-txt-link-freetext href="http://www.star.net.uk"
moz-do-not-send="true">http://www.star.net.uk</A><BR>________________________________________________________________________<BR><PRE wrap=""><FIELDSET class=mimeAttachmentHeader></FIELDSET>
_______________________________________________
Ocfs2-users mailing list
<A class=moz-txt-link-abbreviated href="mailto:Ocfs2-users@oss.oracle.com" moz-do-not-send="true">Ocfs2-users@oss.oracle.com</A>
<A class=moz-txt-link-freetext href="http://oss.oracle.com/mailman/listinfo/ocfs2-users" moz-do-not-send="true">http://oss.oracle.com/mailman/listinfo/ocfs2-users</A></PRE></BLOCKQUOTE><BR><BR>________________________________________________________________________<BR>This
e-mail has been scanned for all viruses by Star. The<BR>service is powered by
MessageLabs. For more information on a proactive<BR>anti-virus service working
around the clock, around the globe, visit:<BR><A class=moz-txt-link-freetext
href="http://www.star.net.uk">http://www.star.net.uk</A><BR>________________________________________________________________________<BR><BR>________________________________________________________________________<BR>This
e-mail has been scanned for all viruses by Star. The<BR>service is powered by
MessageLabs. For more information on a proactive<BR>anti-virus service working
around the clock, around the globe, visit:<BR><A class=moz-txt-link-freetext
href="http://www.star.net.uk">http://www.star.net.uk</A><BR>________________________________________________________________________<BR><PRE wrap=""><FIELDSET class=mimeAttachmentHeader></FIELDSET>
_______________________________________________
Ocfs2-users mailing list
<A class=moz-txt-link-abbreviated href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</A>
<A class=moz-txt-link-freetext href="http://oss.oracle.com/mailman/listinfo/ocfs2-users">http://oss.oracle.com/mailman/listinfo/ocfs2-users</A></PRE></BLOCKQUOTE><BR><BR>________________________________________________________________________<BR>This
e-mail has been scanned for all viruses by Star. The<BR>service is powered by
MessageLabs. For more information on a proactive<BR>anti-virus service working
around the clock, around the globe,
visit:<BR>http://www.star.net.uk<BR>________________________________________________________________________<BR><BR>
________________________________________________________________________<BR>
This e-mail has been scanned for all viruses by Star. The<BR>
service is powered by MessageLabs. For more information on a proactive<BR>
anti-virus service working around the clock, around the globe, visit:<BR>
http://www.star.net.uk<BR>
________________________________________________________________________<BR>
</BODY></HTML>