[Ocfs2-users] sanity check - Xen+iSCSI+LVM+OCFS2 at dom0/domU
Alok Dhir
adhir at symplicity.com
Fri Feb 8 10:25:06 PST 2008
Great call on the netconsole -- I had no idea it existed until your
advice - you learn something new every day :)
Here's my repeatable oops on Centos 5.1 Xen dom0, 2.6.18-53.1.6.el5xen
x86_64, using OSS packages 'ocfs2-2.6.18-53.1.6.el5xen-1.2.8-2.el5'.
As soon as I kick off 'iozone -A':
Kernel BUG at fs/inode.c:250
invalid opcode: 0000 [1] SMP
last sysfs file: /devices/
pci0000
:00/0000:00:02.0/0000:04:00.0/0000:05:00.0/0000:06:00.0/0000:07:00.0/irq
CPU 5
Modules linked in: ocfs2(U) netconsole netloop netbk blktap blkbk
ipt_MASQUERADE iptable_nat ip_nat xt_state ip_conntrack nfnetlink
ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge ipv6
autofs4 hidp rfcomm l2cap bluetooth ocfs2_dlmfs(U) ocfs2_dlm(U)
ocfs2_nodemanager(U) configfs sunrpc ib_iser rdma_cm ib_cm iw_cm
ib_addr ib_local_sa ib_sa ib_mad ib_core iscsi_tcp libiscsi
scsi_transport_iscsi dm_multipath video sbs backlight i2c_ec i2c_core
button battery asus_acpi ac parport_pc lp parport joydev sr_mod ide_cd
serial_core serio_raw cdrom pcspkr shpchp bnx2 dm_snapshot dm_zero
dm_mirror dm_mod mppVhba(U) usb_storage ata_piix libata megaraid_sas
mppUpper(U) sg(U) sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 31841, comm: iozone Not tainted 2.6.18-53.1.6.el5xen #1
RIP: e030:[<ffffffff80222b19>] [<ffffffff80222b19>] clear_inode+0x1b/
0x123
RSP: e02b:ffff8803c2803e28 EFLAGS: 00010202
RAX: ffff8803c3f2ff20 RBX: ffff8803c3f2fd98 RCX: 0000000000000000
RDX: ffffffffff578140 RSI: ffff8803c2803e48 RDI: ffff8803c3f2fd98
RBP: 0000000000000000 R08: ffff8803dbe1b6c0 R09: 0000000000000002
R10: 0000000000000001 R11: ffff880002cb8c00 R12: ffff8803c3f2fac0
R13: ffff8803d6e46000 R14: 0000000000000002 R15: 0000000000000000
FS: 00002aaaaaac5ee0(0000) GS:ffffffff80599280(0000) knlGS:
0000000000000000
CS: e033 DS: 0000 ES: 0000
Process iozone (pid: 31841, threadinfo ffff8803c2802000, task
ffff8803e345a7a0)
Stack: ffff8803c3f2fd98 ffffffff8869ac19 ffff8803c3f2fd98
ffff8803c3857130
0000000000000000 0000000000000002 ffffffffffffffff 0000000000000200
ffff8803c3f2fd98 ffffffff8869a4f0
Call Trace:
[<ffffffff8869ac19>] :ocfs2:ocfs2_delete_inode+0x729/0x79a
[<ffffffff8869a4f0>] :ocfs2:ocfs2_delete_inode+0x0/0x79a
[<ffffffff8022f811>] generic_delete_inode+0xc6/0x143
[<ffffffff88699df5>] :ocfs2:ocfs2_drop_inode+0x117/0x16e
[<ffffffff8023c6ea>] do_unlinkat+0xd5/0x141
[<ffffffff8025d291>] tracesys+0x47/0xb2
[<ffffffff8025d2f1>] tracesys+0xa7/0xb2
Code: 0f 0b 68 e3 7f 47 80 c2 fa 00 48 8b 83 08 02 00 00 a8 10 75
RIP [<ffffffff80222b19>] clear_inode+0x1b/0x123
RSP <ffff8803c2803e28>
<0>Kernel panic - not syncing: Fatal exception
On Feb 7, 2008, at 6:56 PM, Sunil Mushran wrote:
> Setup netconsole on the cluster members (domU?) to get a stack trace.
>
> Alok Dhir wrote:
>> Thanks again for your prompt assistance earlier today - we seem to
>> have gotten past the fs/inode.c bug at domU by using the OSS
>> packaged ocfs2 kernel modules. The cluster comes up and mounts on
>> all boxes, and appears to work.
>>
>> However, we have now run into a more prevalent issue - at dom0, any
>> of the cluster member servers will spontaneously reboot when I
>> start an 'iozone -A' in an ocfs2 filesystem. I am unable to check
>> the kernel panic message as the box reboots immediately, despite
>> the setting of 'kernel.panic=0' in sysctl (which is supposed to
>> mean 'do not reboot on panic'). There are also no entries in
>> messages when this happens.
>>
>> I realize there's not much debugging you can do without the panic
>> message, but I'm wondering if perhaps this new version has some bug
>> which was not in 1.2.7 (with our self-built 1.2.7 only domU servers
>> rebooted - dom0 were stable).
>>
>> Are others running this new version with success? Under RHEL/
>> Centos 5.1 Xen dom0/domU?
>>
>> On Feb 7, 2008, at 1:40 PM, Sunil Mushran wrote:
>>
>>> Is the ip address correct? If not, correct.
>>>
>>> # netstat -tan
>>> See if that port is already in use. If so, use another.
>>>
>>> Alok Dhir wrote:
>>>> Ah - thanks for the clarification.
>>>>
>>>> I'm left with one perplexing problem - on one of the hosts,
>>>> 'devxen0', o2cb refuses to start. The box is identically
>>>> configured to at least 2 other cluster hosts and all were imaged
>>>> the exact same way, except that devxen0 has 32GB RAM where the
>>>> others have 16 or less.
>>>>
>>>> Any clues where to look?
>>>>
>>>> --[root at devxen0:~] service o2cb enable
>>>> Writing O2CB configuration: OK
>>>> Starting O2CB cluster ocfs2: Failed
>>>> Cluster ocfs2 created
>>>> Node beast added
>>>> o2cb_ctl: Internal logic failure while adding node devxen0
>>>>
>>>> Stopping O2CB cluster ocfs2: OK
>>>> --
>>>> This is in syslog when this happens:
>>>>
>>>> Feb 7 13:26:50 devxen0 kernel:
>>>> (17194,6):o2net_open_listening_sock:1867 ERROR: unable to bind
>>>> socket at 196.168.1.72:7777, ret=-99
>>>>
>>>> --
>>>> Box config:
>>>>
>>>> [root at devxen0:~] uname -a
>>>> Linux devxen0.symplicity.com 2.6.18-53.1.6.el5xen #1 SMP Wed Jan
>>>> 23 11:59:21 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>> --
>>>> Here is cluster.conf:
>>>>
>>>> ---
>>>> node:
>>>> ip_port = 7777
>>>> ip_address = 192.168.1.62
>>>> number = 0
>>>> name = beast
>>>> cluster = ocfs2
>>>>
>>>> node:
>>>> ip_port = 7777
>>>> ip_address = 196.168.1.72
>>>> number = 1
>>>> name = devxen0
>>>> cluster = ocfs2
>>>>
>>>> node:
>>>> ip_port = 7777
>>>> ip_address = 192.168.1.73
>>>> number = 2
>>>> name = devxen1
>>>> cluster = ocfs2
>>>>
>>>> node:
>>>> ip_port = 7777
>>>> ip_address = 192.168.1.74
>>>> number = 3
>>>> name = devxen2
>>>> cluster = ocfs2
>>>>
>>>> node:
>>>> ip_port = 7777
>>>> ip_address = 192.168.1.70
>>>> number = 4
>>>> name = fs1
>>>> cluster = ocfs2
>>>>
>>>> node:
>>>> ip_port = 7777
>>>> ip_address = 192.168.1.71
>>>> number = 5
>>>> name = fs2
>>>> cluster = ocfs2
>>>>
>>>> node:
>>>> ip_port = 7777
>>>> ip_address = 192.168.1.80
>>>> number = 6
>>>> name = vdb1
>>>> cluster = ocfs2
>>>>
>>>> cluster:
>>>> node_count = 7
>>>> name = ocfs2
>>>> ---
>>>>
>>>>
>>>>
>>>> On Feb 7, 2008, at 1:23 PM, Sunil Mushran wrote:
>>>>
>>>>> Yes, but backported and released as ocfs2 1.4 which is yet to be
>>>>> released.
>>>>> You are on ocfs2 1.2.
>>>>>
>>>>> Alok Dhir wrote:
>>>>>> I've seen that -- I was under the impression that some of those
>>>>>> were being backported into the release kernels.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Alok
>>>>>>
>>>>>> On Feb 7, 2008, at 1:15 PM, Sunil Mushran wrote:
>>>>>>
>>>>>>> http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2-new-features.html
>>>>>>>
>>>>>>> Alok Dhir wrote:
>>>>>>>> We were indeed using a self-built module due to the lack of
>>>>>>>> an OSS one for the latest kernel. Thanks for your response,
>>>>>>>> I will test with the new version.
>>>>>>>>
>>>>>>>> What are we leaving on the table by not using the latest
>>>>>>>> mainline kernel?
>>>>>>>>
>>>>>>>> On Feb 7, 2008, at 12:56 PM, Sunil Mushran wrote:
>>>>>>>>
>>>>>>>>> Are you building ocfs2 with this kernel or are using the
>>>>>>>>> ones we
>>>>>>>>> provide for RHEL5?
>>>>>>>>>
>>>>>>>>> I am assuming you have built it yourself as we did not release
>>>>>>>>> packages for the latest 2.6.18-53.1.6 kernel till last night.
>>>>>>>>>
>>>>>>>>> If you are using your own, then use the one from oss.
>>>>>>>>>
>>>>>>>>> If you are using the one from oss, then file a bugzilla with
>>>>>>>>> the
>>>>>>>>> full oops trace.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Sunil
>>>>>>>>>
>>>>>>>>> Alok K. Dhir wrote:
>>>>>>>>>> Hello all - we're evaluating OCFS2 in our development
>>>>>>>>>> environment to see if it meets our needs.
>>>>>>>>>>
>>>>>>>>>> We're testing it with an iSCSI storage array (Dell MD3000i)
>>>>>>>>>> and 5 servers running Centos 5.1 (2.6.18-53.1.6.el5xen).
>>>>>>>>>>
>>>>>>>>>> 1) Each of the 5 servers is running the Centos 5.1 open-
>>>>>>>>>> iscsi initiator, and sees the volumes exposed by the array
>>>>>>>>>> just fine. So far so good.
>>>>>>>>>>
>>>>>>>>>> 2) Created a volume group using the exposed iscsi volumes
>>>>>>>>>> and created a few LVM2 logical volumes.
>>>>>>>>>>
>>>>>>>>>> 3) vgscan; vgchange -a y; on all the cluster members. all
>>>>>>>>>> see the "md3000vg" volume group. looking good. (we have no
>>>>>>>>>> intention of changing the LVM2 configurations much if at
>>>>>>>>>> all, and can make sure all such changes are done when the
>>>>>>>>>> volumes are off-line on all cluster members, so
>>>>>>>>>> theoretically this should not be a problem).
>>>>>>>>>>
>>>>>>>>>> 4) mkfs.ocfs2 /dev/md3000vg/testvol0 -- works great
>>>>>>>>>>
>>>>>>>>>> 5) mount on all Xen dom0 boxes in the cluster, works great.
>>>>>>>>>>
>>>>>>>>>> 6) create a VM on one of the cluster members, set up iscsi,
>>>>>>>>>> vgscan, md3000vg shows up -- looking good.
>>>>>>>>>>
>>>>>>>>>> 7) install ocfs2, 'service o2cb enable', starts up fine.
>>>>>>>>>> mount /dev/md3000vg/testvol0, works fine.
>>>>>>>>>>
>>>>>>>>>> ** Thanks for making it this far -- this is where is gets
>>>>>>>>>> interesting
>>>>>>>>>>
>>>>>>>>>> 8) run 'iozone' in domU against ocfs2 share - BANG -
>>>>>>>>>> immediate kernel panic, repeatable all day long.
>>>>>>>>>>
>>>>>>>>>> "kernel BUG at fs/inode.c"
>>>>>>>>>>
>>>>>>>>>> So my questions:
>>>>>>>>>>
>>>>>>>>>> 1) should this work?
>>>>>>>>>>
>>>>>>>>>> 2) if not, what should we do differently?
>>>>>>>>>>
>>>>>>>>>> 3) currently we're tracking the latest RHEL/Centos 5.1
>>>>>>>>>> kernels -- would we have better luck using the latest
>>>>>>>>>> mainline kernel?
>>>>>>>>>>
>>>>>>>>>> Thanks for any assistance.
>>>>>>>>>>
>>>>>>>>>> Alok Dhir
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Ocfs2-users mailing list
>>>>>>>>>> Ocfs2-users at oss.oracle.com
>>>>>>>>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Ocfs2-users mailing list
>>>>>>>> Ocfs2-users at oss.oracle.com
>>>>>>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
More information about the Ocfs2-users
mailing list