[Ocfs2-users] 2 node cluster with shared LUN via FC
Manuel Bogner
manuel.bogner at geizhals.at
Fri Nov 5 07:28:03 PDT 2010
Hi again,
your hint was great. I found a workaround by doing the following:
ocfs2_hb_ctl -S -d /dev/sdd1 ocfs2
mount /dev/sdd1 /shared/
The first command starts the network stack as the log says:
Nov 5 15:17:51 localhost kernel: [ 896.971993] o2net: connected to
node xen02a (num 0) at 192.168.100.100:7777
Then I can mount and unmount the fs without problems. Ok, so i have to
write a small startup-script for this, or is there another way to get
the network stack started correctly?
Regards,
Manuel
Am 2010-11-05 12:55, schrieb Manuel Bogner:
> Hi Sérgio,
>
> we have not changed the default behavior of the startin order and the
> share is mounted manually till now with mount /dev/sdd1 /shared. so i
> thought these parts (network stack, ...) are started automatically
> during mount.
>
> here are the links in the /etc/rc*
>
> xen02a:~# for i in `find /etc/rc* -name '*ocfs*'`; do ls -la $i; done
> lrwxrwxrwx 1 root root 15 2010-10-28 09:55 /etc/rc0.d/S02ocfs2 ->
> ../init.d/ocfs2
> lrwxrwxrwx 1 root root 15 2010-10-28 09:55 /etc/rc6.d/S02ocfs2 ->
> ../init.d/ocfs2
> lrwxrwxrwx 1 root root 15 2010-10-28 09:55 /etc/rcS.d/S65ocfs2 ->
> ../init.d/ocfs2
>
>
> xen02a:~# find /etc/rc* -name '*ocfs*'
> /etc/rc0.d/S02ocfs2
> /etc/rc6.d/S02ocfs2
> /etc/rcS.d/S65ocfs2
>
> xen02a:~# for i in `find /etc/rc* -name '*o2*'`; do ls -la $i; done
> lrwxrwxrwx 1 root root 14 2010-10-28 09:55 /etc/rc0.d/S06o2cb ->
> ../init.d/o2cb
> lrwxrwxrwx 1 root root 14 2010-10-28 09:55 /etc/rc6.d/S06o2cb ->
> ../init.d/o2cb
> lrwxrwxrwx 1 root root 14 2010-10-28 09:55 /etc/rcS.d/S60o2cb ->
> ../init.d/o2cb
>
>
> i returned to 2.6.26-2-xen-amd64 as the problem still existed in the
> 2.6.32 kernel installed via backports.
>
>
> xen02a:~# dpkg -l ocfs*
> Desired=Unknown/Install/Remove/Purge/Hold
> |
> Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err:
> uppercase=bad)
> ||/ Name Version Description
> +++-=====================-=====================-==========================================================
> ii ocfs2-tools 1.6.3-0ab1~lenny tools for managing OCFS2
> cluster filesystems
> pn ocfs2console <none> (no description available)
>
> i upgraded the tools yesterday without success. so do you have
> suggestions, how to influence the initialization order?
>
> regards,
> Manuel
>
> Am 2010-11-04 19:06, schrieb Sérgio Surkamp:
>> It seems that the initialization order is reversed.
>>
>> The init scripts should start o2net and then try to mount the
>> filesystem (I think o2cb init script should do it). But in your case,
>> the scripts are trying to mount the filesystem, then start the cluster
>> stack:
>>
>>> Nov 4 17:27:37 localhost kernel: [ 487.105327] ocfs2: Mounting
>>> device (8,49) on (node 0, slot 0) with ordered data mode.
>>> Nov 4 17:28:11 localhost kernel: [ 521.163897] o2net: accepted
>>> connection from node xen02b (num 1) at 192.168.100.101:7777
>>
>> So when the second node brings up, it checks the disk heartbeat and
>> detect the first node, but as the network stack isn't up yet, it
>> "thinks" that the first node is dead, due to the lack of network
>> connectivity and start the recovery procedure.
>>
>> Are you starting the ocfs2 script and then the o2cb script? If so,
>> change the order and bring up the o2cb first (heartbeat+network stack).
>> If not, check the actual init scripts and make sure that the latest
>> thing that the ocfs2 script do is mount the filesystem.
>>
>> Regards,
>> Sérgio
>>
>> Em Thu, 04 Nov 2010 17:31:41 +0100
>> Manuel Bogner <manuel.bogner at geizhals.at> escreveu:
>>
>>> Hi,
>>>
>>> I just upgraded to a bpo kernel 2.6.32-bpo.5-amd64 and now it logs the
>>> following:
>>>
>>> Nov 4 17:27:37 localhost kernel: [ 487.098196] ocfs2_dlm: Nodes in
>>> domain ("8CEAFACAAE3B4A9BB6AAC6A7664EE094"): 0
>>> Nov 4 17:27:37 localhost kernel: [ 487.105327] ocfs2: Mounting
>>> device (8,49) on (node 0, slot 0) with ordered data mode.
>>> Nov 4 17:28:11 localhost kernel: [ 521.163897] o2net: accepted
>>> connection from node xen02b (num 1) at 192.168.100.101:7777
>>>
>>>
>>> Nov 4 17:27:59 localhost kernel: [ 577.338311] ocfs2_dlm: Nodes in
>>> domain ("8CEAFACAAE3B4A9BB6AAC6A7664EE094"): 1
>>> Nov 4 17:27:59 localhost kernel: [ 577.351868] ocfs2: Mounting
>>> device (8,49) on (node 1, slot 1) with ordered data mode.
>>> Nov 4 17:27:59 localhost kernel: [ 577.352241]
>>> (2287,2):ocfs2_replay_journal:1607 Recovering node 0 from slot 0 on
>>> device (8,49)
>>> Nov 4 17:28:00 localhost kernel: [ 578.505783]
>>> (2287,0):ocfs2_begin_quota_recovery:376 Beginning quota recovery in
>>> slot 0 Nov 4 17:28:00 localhost kernel: [ 578.569121]
>>> (2241,0):ocfs2_finish_quota_recovery:569 Finishing quota recovery in
>>> slot 0 Nov 4 17:28:11 localhost kernel: [ 589.359996] o2net:
>>> connected to node xen02a (num 0) at 192.168.100.100:7777
>>>
>>> process description for the log:
>>>
>>> node1: mount
>>> node2: mount
>>>
>>> still the same but now it logs something about the quota.
>>>
>>> (i also changed the network port for the traffic. now they are
>>> directly attached to each other.)
>>>
>>> regards,
>>> Manuel
>>>
>>>
>>> Am 2010-11-04 15:49, schrieb Manuel Bogner:
>>>> Hi,
>>>>
>>>> this could also be interesting. I tried mount /dev/sdd1 /shared/ on
>>>> both nodes at the same time with the following log result:
>>>>
>>>> [ 331.158166] OCFS2 1.5.0
>>>> [ 336.155577] ocfs2_dlm: Nodes in domain
>>>> ("55A9D0B0050C484F97257788A3B9DDE0"): 0
>>>> [ 336.166327] kjournald starting. Commit interval 5 seconds
>>>> [ 336.166327] ocfs2: Mounting device (8,49) on (node 0, slot 1)
>>>> with ordered data mode.
>>>> [ 336.166664] (3239,0):ocfs2_replay_journal:1149 Recovering node 1
>>>> from slot 0 on device (8,49)
>>>> [ 337.350942] kjournald starting. Commit interval 5 seconds
>>>> [ 351.142229] o2net: accepted connection from node xen02b (num 1)
>>>> at 10.0.0.102:7777
>>>> [ 495.059065] o2net: no longer connected to node xen02b (num 1) at
>>>> 10.0.0.102:7777
>>>>
>>>>
>>>> [ 4841.036991] ocfs2_dlm: Nodes in domain
>>>> ("55A9D0B0050C484F97257788A3B9DDE0"): 1
>>>> [ 4841.039225] kjournald starting. Commit interval 5 seconds
>>>> [ 4841.039997] ocfs2: Mounting device (8,49) on (node 1, slot 0)
>>>> with ordered data mode.
>>>> [ 4862.033837] o2net: connected to node xen02a (num 0) at
>>>> 10.0.0.168:7777 [ 5005.996422] o2net: no longer connected to node
>>>> xen02a (num 0) at 10.0.0.168:7777
>>>> [ 5005.998393] ocfs2: Unmounting device (8,49) on (node 1)
>>>>
>>>>
>>>> at the end xen02a was the only one that had it mounted.
>>>>
>>>> regards,
>>>> Manuel
>>>>
>>>>
>>>> Am 2010-11-04 15:14, schrieb Manuel Bogner:
>>>>> Hi Sérgio,
>>>>>
>>>>> thanks for your quick answere.
>>>>>
>>>>> There are such lines after waiting a little bit, but still the same
>>>>> behavior.
>>>>>
>>>>> [ 2063.720211] o2net: connected to node xen02a (num 0) at
>>>>> 10.0.0.168:7777
>>>>>
>>>>> [ 1979.611076] o2net: accepted connection from node xen02b (num 1)
>>>>> at 10.0.0.102:7777
>>>>>
>>>>>
>>>>> xen02a:~# lsmod | egrep 'jbd|ocfs2|configfs'
>>>>> ocfs2 395816 1
>>>>> ocfs2_dlmfs 23696 1
>>>>> ocfs2_stack_o2cb 9088 1
>>>>> ocfs2_dlm 197824 2 ocfs2_dlmfs,ocfs2_stack_o2cb
>>>>> ocfs2_nodemanager 208744 8
>>>>> ocfs2,ocfs2_dlmfs,ocfs2_stack_o2cb,ocfs2_dlm
>>>>> ocfs2_stackglue 16432 2 ocfs2,ocfs2_stack_o2cb
>>>>> configfs 29736 2 ocfs2_nodemanager
>>>>> jbd 54696 2 ocfs2,ext3
>>>>>
>>>>> xen02a:~# netstat -an | grep 7777
>>>>> tcp 0 0 10.0.0.168:7777 0.0.0.0:*
>>>>> LISTEN
>>>>> tcp 0 0 10.0.0.168:7777 10.0.0.102:47547
>>>>> ESTABLISHED
>>>>>
>>>>> xen02b:~# lsmod | egrep 'jbd|ocfs2|configfs'
>>>>> ocfs2 395816 1
>>>>> ocfs2_dlmfs 23696 1
>>>>> ocfs2_stack_o2cb 9088 1
>>>>> ocfs2_dlm 197824 2 ocfs2_dlmfs,ocfs2_stack_o2cb
>>>>> ocfs2_nodemanager 208744 8
>>>>> ocfs2,ocfs2_dlmfs,ocfs2_stack_o2cb,ocfs2_dlm
>>>>> ocfs2_stackglue 16432 2 ocfs2,ocfs2_stack_o2cb
>>>>> configfs 29736 2 ocfs2_nodemanager
>>>>> jbd 54696 2 ocfs2,ext3
>>>>>
>>>>> xen02b:~# netstat -an | grep 7777
>>>>> tcp 0 0 10.0.0.102:7777 0.0.0.0:*
>>>>> LISTEN
>>>>> tcp 0 0 10.0.0.102:47547 10.0.0.168:7777
>>>>> ESTABLISHED
>>>>>
>>>>> There are no iptables-entries on both nodes as they are just
>>>>> test-servers.
>>>>>
>>>>> xen02a:~# uname -a
>>>>> Linux xen02a 2.6.26-2-xen-amd64 #1 SMP Thu Sep 16 16:32:15 UTC 2010
>>>>> x86_64 GNU/Linux
>>>>>
>>>>> xen02b:~# uname -a
>>>>> Linux xen02b 2.6.26-2-xen-amd64 #1 SMP Thu Sep 16 16:32:15 UTC 2010
>>>>> x86_64 GNU/Linux
>>>>>
>>>>> xen02b:~# cat /etc/default/o2cb
>>>>> #
>>>>> # This is a configuration file for automatic startup of the O2CB
>>>>> # driver. It is generated by running /etc/init.d/o2cb configure.
>>>>> # On Debian based systems the preferred method is running
>>>>> # 'dpkg-reconfigure ocfs2-tools'.
>>>>> #
>>>>>
>>>>> # O2CB_ENABLED: 'true' means to load the driver on boot.
>>>>> O2CB_ENABLED=true
>>>>>
>>>>> # O2CB_STACK: The name of the cluster stack backing O2CB.
>>>>> O2CB_STACK=o2cb
>>>>>
>>>>> # O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start.
>>>>> O2CB_BOOTCLUSTER=ocfs2
>>>>>
>>>>> # O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered
>>>>> dead. O2CB_HEARTBEAT_THRESHOLD=31
>>>>>
>>>>> # O2CB_IDLE_TIMEOUT_MS: Time in ms before a network connection is
>>>>> considered dead.
>>>>> O2CB_IDLE_TIMEOUT_MS=30000
>>>>>
>>>>> # O2CB_KEEPALIVE_DELAY_MS: Max time in ms before a keepalive
>>>>> packet is sent O2CB_KEEPALIVE_DELAY_MS=2000
>>>>>
>>>>> # O2CB_RECONNECT_DELAY_MS: Min time in ms between connection
>>>>> attempts O2CB_RECONNECT_DELAY_MS=2000
>>>>>
>>>>>
>>>>> xen02b:~# mount
>>>>> /dev/sda1 on / type ext3 (rw,errors=remount-ro)
>>>>> tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
>>>>> proc on /proc type proc (rw,noexec,nosuid,nodev)
>>>>> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
>>>>> procbususb on /proc/bus/usb type usbfs (rw)
>>>>> udev on /dev type tmpfs (rw,mode=0755)
>>>>> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
>>>>> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
>>>>> configfs on /sys/kernel/config type configfs (rw)
>>>>> ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
>>>>> /dev/sdd1 on /shared type ocfs2 (rw,_netdev,heartbeat=local)
>>>>>
>>>>>
>>>>> regards,
>>>>> Manuel
>>>>>
>>>>> Am 2010-11-04 15:03, schrieb Sérgio Surkamp:
>>>>>> It seems that the o2net (network stack) is not running as you
>>>>>> should see the network messages in dmesg. Something like:
>>>>>>
>>>>>> xen02a kernel: o2net: connected to node xen02b (num 0) at
>>>>>> 10.0.0.102:7777
>>>>>>
>>>>>> Check your firewall and network configurations, also check if
>>>>>> [o2net] kernel thread is running and the tcp port 7777 is
>>>>>> listening in both nodes. If the thread is not running, check if
>>>>>> you have all needed kernel modules loaded:
>>>>>>
>>>>>> ocfs2
>>>>>> jbd
>>>>>> ocfs2_dlm
>>>>>> ocfs2_dlmfs
>>>>>> ocfs2_nodemanager
>>>>>> configfs
>>>>>>
>>>>>> Regards,
>>>>>> Sérgio
>>>>>>
>>>>>> Em Thu, 04 Nov 2010 14:12:11 +0100
>>>>>> Manuel Bogner <manuel.bogner at geizhals.at> escreveu:
>>>>>>
>>>>>>> sorry for the repost, but just saw that i mixed german and
>>>>>>> english... here is the corrected version:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm trying to create a cluster out of 2 nodes. Both systems
>>>>>>> share the same LUN via FC and see it as /dev/sdd.
>>>>>>>
>>>>>>> /dev/sdd has one partition
>>>>>>>
>>>>>>> Disk /dev/sdd: 21.4 GB, 21474836480 bytes
>>>>>>> 64 heads, 32 sectors/track, 20480 cylinders
>>>>>>> Units = cylinders of 2048 * 512 = 1048576 bytes
>>>>>>> Disk identifier: 0xc29cb93d
>>>>>>>
>>>>>>> Device Boot Start End Blocks Id System
>>>>>>> /dev/sdd1 1 20480 20971504 83 Linux
>>>>>>>
>>>>>>> which is formated with
>>>>>>>
>>>>>>> mkfs.ocfs2 -L ocfs2 /dev/sdd1
>>>>>>>
>>>>>>>
>>>>>>> Here is my /etc/ocfs2/cluster.conf
>>>>>>>
>>>>>>> node:
>>>>>>> ip_port = 7777
>>>>>>> ip_address = 10.0.0.168
>>>>>>> number = 0
>>>>>>> name = xen02a
>>>>>>> cluster = ocfs2
>>>>>>>
>>>>>>> node:
>>>>>>> ip_port = 7777
>>>>>>> ip_address = 10.0.0.102
>>>>>>> number = 1
>>>>>>> name = xen02b
>>>>>>> cluster = ocfs2
>>>>>>>
>>>>>>> cluster:
>>>>>>> node_count = 2
>>>>>>> name = ocfs2
>>>>>>>
>>>>>>>
>>>>>>> Everything seems to be fine:
>>>>>>>
>>>>>>> xen02a:~# /etc/init.d/o2cb status
>>>>>>> Driver for "configfs": Loaded
>>>>>>> Filesystem "configfs": Mounted
>>>>>>> Stack glue driver: Loaded
>>>>>>> Stack plugin "o2cb": Loaded
>>>>>>> Driver for "ocfs2_dlmfs": Loaded
>>>>>>> Filesystem "ocfs2_dlmfs": Mounted
>>>>>>> Checking O2CB cluster ocfs2: Online
>>>>>>> Heartbeat dead threshold = 31
>>>>>>> Network idle timeout: 30000
>>>>>>> Network keepalive delay: 2000
>>>>>>> Network reconnect delay: 2000
>>>>>>> Checking O2CB heartbeat: Active
>>>>>>>
>>>>>>> And mounting the fs on each node works fine:
>>>>>>>
>>>>>>> /dev/sdd1 on /shared type ocfs2 (rw,_netdev,heartbeat=local)
>>>>>>>
>>>>>>> Both nodes can ping each other.
>>>>>>>
>>>>>>>
>>>>>>> xen02a:~# mounted.ocfs2 -d
>>>>>>> Device FS UUID
>>>>>>> Label /dev/sdd1 ocfs2
>>>>>>> 55a9d0b0-050c-484f-9725-7788a3b9dde0 ocfs2
>>>>>>>
>>>>>>> xen02b:~# mounted.ocfs2 -d
>>>>>>> Device FS UUID
>>>>>>> Label /dev/sdd1 ocfs2
>>>>>>> 55a9d0b0-050c-484f-9725-7788a3b9dde0 ocfs2
>>>>>>>
>>>>>>>
>>>>>>> Now the problem:
>>>>>>>
>>>>>>> I first mount the device on node1:
>>>>>>>
>>>>>>> xen02a:~# mount -L ocfs2 /shared/
>>>>>>> => /dev/sdd1 on /shared type ocfs2 (rw,_netdev,heartbeat=local)
>>>>>>> without any errors.
>>>>>>>
>>>>>>> dmesg says:
>>>>>>>
>>>>>>> [ 97.244054] ocfs2_dlm: Nodes in domain
>>>>>>> ("55A9D0B0050C484F97257788A3B9DDE0"): 0
>>>>>>> [ 97.245869] kjournald starting. Commit interval 5 seconds
>>>>>>> [ 97.247045] ocfs2: Mounting device (8,49) on (node 0, slot 0)
>>>>>>> with ordered data mode.
>>>>>>>
>>>>>>> xen02a:~# mounted.ocfs2 -f
>>>>>>> Device FS Nodes
>>>>>>> /dev/sdd1 ocfs2 xen02a
>>>>>>>
>>>>>>> xen02a:~# echo "slotmap" | debugfs.ocfs2 -n /dev/sdd1
>>>>>>> Slot# Node#
>>>>>>> 0 0
>>>>>>>
>>>>>>>
>>>>>>> Now I mount the device on the second node:
>>>>>>>
>>>>>>> xen02b:~# mount -L ocfs2 /shared/
>>>>>>> => /dev/sdd1 on /shared type ocfs2 (rw,_netdev,heartbeat=local)
>>>>>>>
>>>>>>> [ 269.741168] OCFS2 1.5.0
>>>>>>> [ 269.765171] ocfs2_dlm: Nodes in domain
>>>>>>> ("55A9D0B0050C484F97257788A3B9DDE0"): 1
>>>>>>> [ 269.779620] kjournald starting. Commit interval 5 seconds
>>>>>>> [ 269.779620] ocfs2: Mounting device (8,49) on (node 1, slot 1)
>>>>>>> with ordered data mode.
>>>>>>> [ 269.779620] (2953,0):ocfs2_replay_journal:1149 Recovering
>>>>>>> node 0 from slot 0 on device (8,49)
>>>>>>> [ 270.950540] kjournald starting. Commit interval 5 seconds
>>>>>>>
>>>>>>> xen02b:~# echo "slotmap" | debugfs.ocfs2 -n /dev/sdd1
>>>>>>> Slot# Node#
>>>>>>> 1 1
>>>>>>>
>>>>>>> xen02b:~# mounted.ocfs2 -f
>>>>>>> Device FS Nodes
>>>>>>> /dev/sdd1 ocfs2 xen02b
>>>>>>>
>>>>>>>
>>>>>>> So the first mount seems to be gone and any changes on the fs on
>>>>>>> that node are not distributed.
>>>>>>>
>>>>>>> At the moment I have no idea what this could be. I hope someone
>>>>>>> can help me.
>>>>>>>
>>>>>>> regards,
>>>>>>> Manuel
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Ocfs2-users mailing list
>>>>>>> Ocfs2-users at oss.oracle.com
>>>>>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Ocfs2-users mailing list
>>>>> Ocfs2-users at oss.oracle.com
>>>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>>>>
>>>>
>>>> _______________________________________________
>>>> Ocfs2-users mailing list
>>>> Ocfs2-users at oss.oracle.com
>>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>>>
>>>
>>> _______________________________________________
>>> Ocfs2-users mailing list
>>> Ocfs2-users at oss.oracle.com
>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>
>>
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>
More information about the Ocfs2-users
mailing list