<div dir="ltr">
<p class="MsoNormal">Dear OCFS2 guys,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">My name is Guy, and I'm testing ocfs2 due to its features as
a clustered filesystem that I need.</p>
<p class="MsoNormal">As part of the stability and reliability test I’ve
performed, I've encountered an issue with ocfs2 (format + mount + remove
disk...), that I wanted to make sure it is a real issue and not just a
mis-configuration.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">The main concern is that the stability of the whole system
is compromised when a single disk/volumes fails. It looks like the OCFS2 is not
handling the error correctly but stuck in an endless loop that interferes with
the work of the server.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I’ve test tested two cluster configurations – (1) Corosync/Pacemaker and (2) o2cb that react similarly. </p>
<p class="MsoNormal">Following the process and log entries:</p>
<p class="MsoNormal"><br></p><p class="MsoNormal">Also below additional configuration that were tested.<br></p><p class="MsoNormal"><br></p><p class="MsoNormal">Node 1:</p>
<p class="MsoNormal">======= </p>
<p class="MsoNormal">1. service corosync start</p>
<p class="MsoNormal">2. service dlm start</p>
<p class="MsoNormal">3. mkfs.ocfs2 -v -Jblock64 -b 4096
--fs-feature-level=max-features --cluster-=pcmk
--cluster-name=cluster-name -N 2 /dev/<path to device></p>
<p class="MsoNormal">4. mount -o
rw,noatime,nodiratime,data=writeback,heartbeat=none,cluster_stack=pcmk
/dev/<path to device> /mnt/ocfs2-mountpoint</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Node 2:</p>
<p class="MsoNormal">=======</p>
<p class="MsoNormal">5. service corosync start</p>
<p class="MsoNormal">6. service dlm start</p>
<p class="MsoNormal">7. mount -o
rw,noatime,nodiratime,data=writeback,heartbeat=none,cluster_stack=pcmk
/dev/<path to device> /mnt/ocfs2-mountpoint</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">So far all is working well, including reading and writing.</p>
<p class="MsoNormal">Next</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">8.
I’ve physically, pull out the disk at /dev/<path to device> to simulate a
hardware failure (that may occur…) , in real life the disk
is (hardware or software) protected. Nonetheless, I’m testing a hardware
failure that the one of the OCFS2 file systems in my server fails.</p>
<p class="MsoNormal">Following - messages observed in the system log
(see below) and <br></p>
<p class="MsoNormal">==> 9. kernel panic(!) ... in one of the nodes or on both, or reboot on one of the nodes or both.</p>
<p class="MsoNormal"></p>
<p class="MsoNormal"><br></p><p class="MsoNormal">Is there any configuration or set of parameters that will
enable the system to continue working, disabling the access to the failed disk
without compromising the system stability and not cause the kernel to panic?!</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">From my point of view it looks basics – when a hardware
failure occurs:</p>
<p><span style="font-size:11pt;font-family:"Calibri","sans-serif"">1. All
remaining hardware should continue working</span></p>
<p><span style="font-size:11pt;font-family:"Calibri","sans-serif"">2. The
failed disk/volume should be inaccessible – but not compromise the whole system
availability (Kernel panic).</span></p>
<p><span style="font-size:11pt;font-family:"Calibri","sans-serif"">3. OCFS2
“understands” there’s a failed disk and stop trying to access it.</span></p>
<p><span style="font-size:11pt;font-family:"Calibri","sans-serif"">3. All
disk commands such as mount/umount, df etc. should continue working.</span></p>
<p><span style="font-size:11pt;font-family:"Calibri","sans-serif"">4. When a
new/replacement drive is connected to the system, it can be accessed.</span></p>
<p class="MsoNormal">My settings:</p>
<p class="MsoNormal">ubuntu 14.04</p>
<p class="MsoNormal">linux: 3.16.0-46-generic</p>
<p class="MsoNormal">mkfs.ocfs2 1.8.4 (downloaded from git)</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Some other scenarios which also were tested:</p>
<p class="MsoNormal">1. Remove the max-features in the mkfs (i.e. mkfs.ocfs2 -v
-Jblock64 -b 4096 --cluster-stack=pcmk --cluster-name=cluster-name -N 2
/dev/<path to device>)</p>
<p class="MsoNormal">This improved in some of the cases with no kernel panic but
still the stability of the system was compromised, the syslog indicates that
something unrecoverable is going on (See below - Appendix A1). Furthermore,
System is hanging when trying to software reboot.</p>
<p class="MsoNormal">2. Also tried with the o2cb stack, with similar outcomes.</p>
<p class="MsoNormal">3. The configuration was also tested with (1,2 and 3) Local and Global
heartbeat(s) that were NOT on the simulated failed disk, but on other
physical disks.</p><p class="MsoNormal">4. Also tested: </p>
<p class="MsoNormal">Ubuntu 15.15</p>
<p class="MsoNormal">Kernel: 4.2.0-23-generic</p>
<p class="MsoNormal">mkfs.ocfs2 1.8.4 (git clone git://<a href="http://oss.oracle.com/git/ocfs2-tools.git" target="_blank">oss.oracle.com/git/ocfs2-tools.git</a>)</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">==============</p>
<p class="MsoNormal">Appendix A1:</p>
<p class="MsoNormal">==============</p>
<p class="MsoNormal">from syslog:</p>
<p class="MsoNormal">[ 1676.608123] (ocfs2cmt,5316,14):ocfs2_commit_thread:2195
ERROR: status = -5, journal is already aborted.</p>
<p class="MsoNormal">[ 1677.611827] (ocfs2cmt,5316,14):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1678.616634] (ocfs2cmt,5316,15):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1679.621419] (ocfs2cmt,5316,15):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1680.626175] (ocfs2cmt,5316,15):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1681.630981] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1682.107356] INFO: task kworker/u64:0:6 blocked for more
than 120 seconds.</p>
<p class="MsoNormal">[ 1682.108440] Not tainted
3.16.0-46-generic #62~14.04.1</p>
<p class="MsoNormal">[ 1682.109388] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.</p>
<p class="MsoNormal">[ 1682.110381] kworker/u64:0 D ffff88103fcb30c0
0 6 2 0x00000000</p>
<p class="MsoNormal">[ 1682.110401] Workqueue: fw_event0 _firmware_event_work
[mpt3sas]</p>
<p class="MsoNormal">[ 1682.110405] ffff88102910b8a0 0000000000000046
ffff88102977b2f0 00000000000130c0</p>
<p class="MsoNormal">[ 1682.110411] ffff88102910bfd8 00000000000130c0
ffff88102928c750 ffff88201db284b0</p>
<p class="MsoNormal">[ 1682.110415] ffff88201db28000 ffff881028cef000
ffff88201db28138 ffff88201db28268</p>
<p class="MsoNormal">[ 1682.110419] Call Trace:</p>
<p class="MsoNormal">[ 1682.110427] [<ffffffff8176a8b9>]
schedule+0x29/0x70</p>
<p class="MsoNormal">[ 1682.110458] [<ffffffffc08d6c11>]
ocfs2_clear_inode+0x3b1/0xa30 [ocfs2]</p>
<p class="MsoNormal">[ 1682.110464] [<ffffffff810b4de0>] ?
prepare_to_wait_event+0x100/0x100</p>
<p class="MsoNormal">[ 1682.110487] [<ffffffffc08d8c7e>]
ocfs2_evict_inode+0x6e/0x730 [ocfs2]</p>
<p class="MsoNormal">[ 1682.110493] [<ffffffff811eee04>]
evict+0xb4/0x180</p>
<p class="MsoNormal">[ 1682.110498] [<ffffffff811eef09>]
dispose_list+0x39/0x50</p>
<p class="MsoNormal">[ 1682.110501] [<ffffffff811efdb4>]
invalidate_inodes+0x134/0x150</p>
<p class="MsoNormal">[ 1682.110506] [<ffffffff8120a64a>]
__invalidate_device+0x3a/0x60</p>
<p class="MsoNormal">[ 1682.110510] [<ffffffff81367e81>]
invalidate_partition+0x31/0x50</p>
<p class="MsoNormal">[ 1682.110513] [<ffffffff81368f45>]
del_gendisk+0xf5/0x290</p>
<p class="MsoNormal">[ 1682.110519] [<ffffffff815177a1>]
sd_remove+0x61/0xc0</p>
<p class="MsoNormal">[ 1682.110524] [<ffffffff814baf7f>]
__device_release_driver+0x7f/0xf0</p>
<p class="MsoNormal">[ 1682.110529] [<ffffffff814bb013>]
device_release_driver+0x23/0x30</p>
<p class="MsoNormal">[ 1682.110534] [<ffffffff814ba918>]
bus_remove_device+0x108/0x180</p>
<p class="MsoNormal">[ 1682.110538] [<ffffffff814b7169>]
device_del+0x129/0x1c0</p>
<p class="MsoNormal">[ 1682.110543] [<ffffffff815123a5>]
__scsi_remove_device+0xd5/0xe0</p>
<p class="MsoNormal">[ 1682.110547] [<ffffffff815123d6>]
scsi_remove_device+0x26/0x40</p>
<p class="MsoNormal">[ 1682.110551] [<ffffffff81512590>]
scsi_remove_target+0x170/0x230</p>
<p class="MsoNormal">[ 1682.110561] [<ffffffffc03551e5>]
sas_rphy_remove+0x65/0x80 [scsi_transport_sas]</p>
<p class="MsoNormal">[ 1682.110570] [<ffffffffc035707d>]
sas_port_delete+0x2d/0x170 [scsi_transport_sas]</p>
<p class="MsoNormal">[ 1682.110575] [<ffffffff8124a6f9>] ?
sysfs_remove_link+0x19/0x30</p>
<p class="MsoNormal">[ 1682.110588] [<ffffffffc03f1599>]
mpt3sas_transport_port_remove+0x1c9/0x1e0 [mpt3sas]</p>
<p class="MsoNormal">[ 1682.110598] [<ffffffffc03e60b5>]
_scsih_remove_device+0x55/0x80 [mpt3sas]</p>
<p class="MsoNormal">[ 1682.110610] [<ffffffffc03e6159>]
_scsih_device_remove_by_handle.part.21+0x79/0xa0 [mpt3sas]</p>
<p class="MsoNormal">[ 1682.110619] [<ffffffffc03eca97>]
_firmware_event_work+0x1337/0x1690 [mpt3sas]</p>
<p class="MsoNormal">[ 1682.110626] [<ffffffff8101c315>] ?
native_sched_clock+0x35/0x90</p>
<p class="MsoNormal">[ 1682.110630] [<ffffffff8101c379>] ?
sched_clock+0x9/0x10</p>
<p class="MsoNormal">[ 1682.110636] [<ffffffff81011574>] ?
__switch_to+0xe4/0x580</p>
<p class="MsoNormal">[ 1682.110640] [<ffffffff81087bc9>] ?
pwq_activate_delayed_work+0x39/0x80</p>
<p class="MsoNormal">[ 1682.110644] [<ffffffff8108a302>]
process_one_work+0x182/0x450</p>
<p class="MsoNormal">[ 1682.110648] [<ffffffff8108aa71>]
worker_thread+0x121/0x570</p>
<p class="MsoNormal">[ 1682.110652] [<ffffffff8108a950>] ?
rescuer_thread+0x380/0x380</p>
<p class="MsoNormal">[ 1682.110657] [<ffffffff81091309>]
kthread+0xc9/0xe0</p>
<p class="MsoNormal">[ 1682.110662] [<ffffffff81091240>] ?
kthread_create_on_node+0x1c0/0x1c0</p>
<p class="MsoNormal">[ 1682.110667] [<ffffffff8176e818>]
ret_from_fork+0x58/0x90</p>
<p class="MsoNormal">[ 1682.110672] [<ffffffff81091240>] ?
kthread_create_on_node+0x1c0/0x1c0</p>
<p class="MsoNormal">[ 1682.635761] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1683.640549] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1684.645336] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1685.650114] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1686.654911] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1687.659684] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1688.664466] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1689.669252] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1690.674026] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1691.678810] (ocfs2cmt,5316,9):ocfs2_commit_cache:324
ERROR: status = -5</p>
<p class="MsoNormal">[ 1691.679920] (ocfs2cmt,5316,9):ocfs2_commit_thread:2195
ERROR: status = -5, journal is already aborted.</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks
in advance,</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Guy</p>
</div>