[Ocfs2-users] issues with my ocfs2 cluster

Thu Jan 25 15:29:21 PST 2018

hi list,

I upgraded all nodes to kernel 4.10 and things seemed stable for a time.
But we have had light load since. Today load ramped up somewhat and we had
an issue.
One node did shutdown for some reason and then shortly after some nodes
started getting kernel prints like I've posted here before.
I can post more details soon, but I first wanted to ask. What is the best
way to tell if a node shutdowns because of a ocfs2 kernel fault?
because I have
/etc/sysctl.conf
kernel.panic_on_oops = 1
kernel.panic = 30

thanks!

-- Jim

On Wed, Jan 10, 2018 at 1:47 PM, Luis Freitas <lfreitas34 at yahoo.com> wrote:

> Hi Jim,
>
>       Have you tried to update the kernel as suggested by Changwei?
>
>       The messages seem to indicate kernel 4.4.0, a google search shows
> this ubuntu version should be able to use kernel 4.10.
>
> Best Regards,
> Luis
>
>
> On Wednesday, January 10, 2018 4:12 PM, Jim Okken <jim at jokken.com> wrote:
>
>
> hello again list,
>
> We seem to be having issues on more servers where according to the linux
> developers here: "the kernel is stuck in a spin lock during a disk
> operation."
>
> The call traces are below, I see a lot of ocfs in the call traces, but I
> don't know how to read them, please tell me does the issue come from ocfs?
> thanks
> --Jim
>
> 2018-01-06T17:10:02.194362+00:00 node-115 kernel: [87885.155288] Modules
> linked in: vhost_net vhost macvtap macvlan ip6table_raw xt_mac xt_tcpudp
> xt_physdev br_netfilter veth ebtable_filter ebtables openvswitch ocfs2
> quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
> ocfs2_stackglue configfs ip6table_filter ip6_tables xt_multiport
> xt_conntrack iptable_filter xt_comment xt_CT iptable_raw ip_tables x_tables
> xfs bridge 8021q garp mrp stp llc intel_rapl x86_pkg_temp_thermal
> intel_powerclamp coretemp crct10dif_pclmul ipmi_ssif crc32_pclmul
> ghash_clmulni_intel kvm_intel aesni_intel aes_x86_64 kvm lrw gf128mul
> glue_helper ablk_helper irqbypass cryptd hpilo 8250_fintek serio_raw
> ioatdma ipmi_si sb_edac edac_core ipmi_msghandler shpchp dca
> acpi_power_meter lpc_ich mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad
> ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> nf_conntrack_proto_gre nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4
> nf_defrag_ipv4 nf_conntrack autofs4 raid10 raid456 async_raid6_recov
> async_memcpy async_pq async_xor async_tx xor dm_round_robin ses enclosure
> scsi_transport_sas raid6_pq libcrc32c raid1 raid0 multipath linear uas
> usb_storage psmouse lpfc be2net vxlan ip6_udp_tunnel scsi_transport_fc
> udp_tunnel wmi fjes scsi_dh_emc scsi_dh_rdac scsi_dh_alua dm_multipath
> 2018-01-06T17:10:02.194364+00:00 node-115 kernel: [87885.157143] CPU: 15
> PID: 11936 Comm: qemu-system-x86 Not tainted 4.4.0-98-generic #121-Ubuntu
> 2018-01-06T17:10:02.194366+00:00 node-115 kernel: [87885.157144] Hardware
> name: HP ProLiant BL460c Gen9, BIOS I36 02/17/2017
> 2018-01-06T17:10:02.194367+00:00 node-115 kernel: [87885.157280] task:
> ffff882036ff0000 ti: ffff881f80ca0000 task.ti: ffff881f80ca0000
> 2018-01-06T17:10:02.194400+00:00 node-115 kernel: [87885.157281] RIP:
> 0010:[<ffffffff810cb27c>]  [<ffffffff810cb27c>] native_queued_spin_lock_
> slowpath+0x15c/0x170
> 2018-01-06T17:10:02.194414+00:00 node-115 kernel: [87885.157566] RSP:
> 0018:ffff88203f143c30  EFLAGS: 00000202
> 2018-01-06T17:10:02.194416+00:00 node-115 kernel: [87885.157567] RAX:
> 0000000000000101 RBX: ffff8820046c83f0 RCX: 0000000000000001
> 2018-01-06T17:10:02.194418+00:00 node-115 kernel: [87885.157705] RDX:
> 0000000000000101 RSI: 0000000000000001 RDI: ffff8820046c83ec
> 2018-01-06T17:10:02.194440+00:00 node-115 kernel: [87885.157705] RBP:
> ffff88203f143c30 R08: 0000000000000101 R09: ffffffff811924a7
> 2018-01-06T17:10:02.194442+00:00 node-115 kernel: [87885.157706] R10:
> ffffea0040d6d680 R11: 0000000000000800 R12: ffff8820046c83ec
> 2018-01-06T17:10:02.194443+00:00 node-115 kernel: [87885.157707] R13:
> 0000000000000800 R14: 000000004c63ee00 R15: 0000000000000800
> 2018-01-06T17:10:02.194444+00:00 node-115 kernel: [87885.157708] FS:
> 00007fbcbb7eec00(0000) GS:ffff88203f140000(0000) knlGS:0000000000000000
> 2018-01-06T17:10:02.194444+00:00 node-115 kernel: [87885.157709] CS:
> 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2018-01-06T17:10:02.194445+00:00 node-115 kernel: [87885.157710] CR2:
> 00007f54266a8000 CR3: 0000000fcc2f2000 CR4: 00000000001426e0
> 2018-01-06T17:10:02.194446+00:00 node-115 kernel: [87885.157711] Stack:
> 2018-01-06T17:10:02.194448+00:00 node-115 kernel: [87885.157712]
> ffff88203f143c40 ffffffff81844421 ffff88203f143c60 ffffffff81842535
> 2018-01-06T17:10:02.194449+00:00 node-115 kernel: [87885.157714]
> ffff881e88a9ca80 ffff8820046c84b0 ffff88203f143c70 ffffffff8184257b
> 2018-01-06T17:10:02.194450+00:00 node-115 kernel: [87885.157716]
> ffff88203f143ca0 ffffffffc074158d ffff881e5d3beb80 0000000000000800
> 2018-01-06T17:10:02.194450+00:00 node-115 kernel: [87885.157717] Call
> Trace:
> 2018-01-06T17:10:02.194451+00:00 node-115 kernel: [87885.157718]  <IRQ>
> 2018-01-06T17:10:02.194453+00:00 node-115 kernel: [87885.157725]
> [<ffffffff81844421>] _raw_spin_lock+0x21/0x30
> 2018-01-06T17:10:02.194454+00:00 node-115 kernel: [87885.157727]
> [<ffffffff81842535>] __mutex_unlock_slowpath+0x25/0x50
> 2018-01-06T17:10:02.194456+00:00 node-115 kernel: [87885.157729]
> [<ffffffff8184257b>] mutex_unlock+0x1b/0x20
> 2018-01-06T17:10:02.194457+00:00 node-115 kernel: [87885.157766]
> [<ffffffffc074158d>] ocfs2_dio_end_io+0x6d/0x80 [ocfs2]
> 2018-01-06T17:10:02.194458+00:00 node-115 kernel: [87885.157770]
> [<ffffffff8124e57c>] dio_complete+0x11c/0x1c0
> 2018-01-06T17:10:02.194460+00:00 node-115 kernel: [87885.157771]
> [<ffffffff8124e693>] dio_bio_end_aio+0x73/0x100
> 2018-01-06T17:10:02.194461+00:00 node-115 kernel: [87885.157774]
> [<ffffffff813c3edf>] bio_endio+0x3f/0x60
> 2018-01-06T17:10:02.194463+00:00 node-115 kernel: [87885.157777]
> [<ffffffff813cb897>] blk_update_request+0x87/0x310
> 2018-01-06T17:10:02.194464+00:00 node-115 kernel: [87885.157780]
> [<ffffffff816bbd66>] end_clone_bio+0x46/0x70
> 2018-01-06T17:10:02.194465+00:00 node-115 kernel: [87885.157782]
> [<ffffffff813c3edf>] bio_endio+0x3f/0x60
> 2018-01-06T17:10:02.194465+00:00 node-115 kernel: [87885.157783]
> [<ffffffff813cb897>] blk_update_request+0x87/0x310
> 2018-01-06T17:10:02.194467+00:00 node-115 kernel: [87885.157786]
> [<ffffffff815c52f3>] scsi_end_request+0x33/0x1d0
> 2018-01-06T17:10:02.194468+00:00 node-115 kernel: [87885.157788]
> [<ffffffff815c8a26>] scsi_io_completion+0x1b6/0x690
> 2018-01-06T17:10:02.194469+00:00 node-115 kernel: [87885.157792]
> [<ffffffff810beb46>] ? rebalance_domains+0x166/0x2d0
> 2018-01-06T17:10:02.194470+00:00 node-115 kernel: [87885.157795]
> [<ffffffff815bf64f>] scsi_finish_command+0xcf/0x120
> 2018-01-06T17:10:02.194471+00:00 node-115 kernel: [87885.157796]
> [<ffffffff815c81b4>] scsi_softirq_done+0x124/0x150
> 2018-01-06T17:10:02.194471+00:00 node-115 kernel: [87885.157799]
> [<ffffffff813d3787>] blk_done_softirq+0x87/0xb0
> 2018-01-06T17:10:02.194480+00:00 node-115 kernel: [87885.157803]
> [<ffffffff81085dc1>] __do_softirq+0x101/0x290
> 2018-01-06T17:10:02.194481+00:00 node-115 kernel: [87885.157805]
> [<ffffffff810860c3>] irq_exit+0xa3/0xb0
> 2018-01-06T17:10:02.194483+00:00 node-115 kernel: [87885.157809]
> [<ffffffff81050e93>] smp_call_function_single_interrupt+0x33/0x40
> 2018-01-06T17:10:02.194483+00:00 node-115 kernel: [87885.157811]
> [<ffffffff81845ae2>] call_function_single_interrupt+0x82/0x90
> 2018-01-06T17:10:02.194484+00:00 node-115 kernel: [87885.157812]  <EOI>
> 2018-01-06T17:10:02.194484+00:00 node-115 kernel: [87885.157814]
> [<ffffffff81844414>] ? _raw_spin_lock+0x14/0x30
> 2018-01-06T17:10:02.194486+00:00 node-115 kernel: [87885.157815]
> [<ffffffff81842422>] __mutex_lock_slowpath+0x72/0x130
> 2018-01-06T17:10:02.194487+00:00 node-115 kernel: [87885.157829]
> [<ffffffffc0758099>] ? ocfs2_inode_unlock+0x119/0x120 [ocfs2]
> 2018-01-06T17:10:02.194488+00:00 node-115 kernel: [87885.157831]
> [<ffffffff818424ff>] mutex_lock+0x1f/0x30
> 2018-01-06T17:10:02.194489+00:00 node-115 kernel: [87885.157843]
> [<ffffffffc076177a>] ocfs2_file_write_iter+0x95a/0xdf0 [ocfs2]
> 2018-01-06T17:10:02.194490+00:00 node-115 kernel: [87885.157847]
> [<ffffffff812252c0>] ? poll_select_copy_remaining+0x140/0x140
> 2018-01-06T17:10:02.194490+00:00 node-115 kernel: [87885.157858]
> [<ffffffffc0760e20>] ? ocfs2_check_range_for_refcount+0x150/0x150 [ocfs2]
> 2018-01-06T17:10:02.194492+00:00 node-115 kernel: [87885.157862]
> [<ffffffff812613ea>] aio_run_iocb+0x26a/0x2d0
> 2018-01-06T17:10:02.194493+00:00 node-115 kernel: [87885.157865]
> [<ffffffff8122e8e5>] ? __fget_light+0x25/0x60
> 2018-01-06T17:10:02.194495+00:00 node-115 kernel: [87885.157867]
> [<ffffffff8122e933>] ? __fdget+0x13/0x20
> 2018-01-06T17:10:02.194496+00:00 node-115 kernel: [87885.157868]
> [<ffffffff812622cf>] do_io_submit+0x25f/0x500
> 2018-01-06T17:10:02.194497+00:00 node-115 kernel: [87885.157871]
> [<ffffffff81262580>] SyS_io_submit+0x10/0x20
> 2018-01-06T17:10:02.194499+00:00 node-115 kernel: [87885.157873]
> [<ffffffff818446b2>] entry_SYSCALL_64_fastpath+0x16/0x71
> 2018-01-06T17:10:02.194500+00:00 node-115 kernel: [87885.157874] Code: 01
> 48 8b 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0 74 f6 c7 40 08 01 00 00 00
> e9 63 ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90 <8b> 07 84 c0 75 f8 b8
> 01 00 00 00 66 89 07 5d c3 0f 1f 40 00 0f
> 2018-01-06T17:10:30.192979+00:00 node-115 kernel: [87913.154413] Modules
> linked in: vhost_net vhost macvtap macvlan ip6table_raw xt_mac xt_tcpudp
> xt_physdev br_netfilter veth ebtable_filter ebtables openvswitch ocfs2
> quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
> ocfs2_stackglue configfs ip6table_filter ip6_tables xt_multiport
> xt_conntrack iptable_filter xt_comment xt_CT iptable_raw ip_tables x_tables
> xfs bridge 8021q garp mrp stp llc intel_rapl x86_pkg_temp_thermal
> intel_powerclamp coretemp crct10dif_pclmul ipmi_ssif crc32_pclmul
> ghash_clmulni_intel kvm_intel aesni_intel aes_x86_64 kvm lrw gf128mul
> glue_helper ablk_helper irqbypass cryptd hpilo 8250_fintek serio_raw
> ioatdma ipmi_si sb_edac edac_core ipmi_msghandler shpchp dca
> acpi_power_meter lpc_ich mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad
> ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> nf_conntrack_proto_gre nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4
> nf_defrag_ipv4 nf_conntrack autofs4 raid10 raid456 async_raid6_recov
> async_memcpy async_pq async_xor async_tx xor dm_round_robin ses enclosure
> scsi_transport_sas raid6_pq libcrc32c raid1 raid0 multipath linear uas
> usb_storage psmouse lpfc be2net vxlan ip6_udp_tunnel scsi_transport_fc
> udp_tunnel wmi fjes scsi_dh_emc scsi_dh_rdac scsi_dh_alua dm_multipath
> 2018-01-06T17:10:30.192984+00:00 node-115 kernel: [87913.155150] CPU: 15
> PID: 11936 Comm: qemu-system-x86 Tainted: G             L  4.4.0-98-generic
> #121-Ubuntu
> 2018-01-06T17:10:30.192987+00:00 node-115 kernel: [87913.155151] Hardware
> name: HP ProLiant BL460c Gen9, BIOS I36 02/17/2017
> 2018-01-06T17:10:30.192988+00:00 node-115 kernel: [87913.155153] task:
> ffff882036ff0000 ti: ffff881f80ca0000 task.ti: ffff881f80ca0000
> 2018-01-06T17:10:30.192990+00:00 node-115 kernel: [87913.155154] RIP:
> 0010:[<ffffffff810cb27e>]  [<ffffffff810cb27e>] native_queued_spin_lock_
> slowpath+0x15e/0x170
> 2018-01-06T17:10:30.192992+00:00 node-115 kernel: [87913.155160] RSP:
> 0018:ffff88203f143c30  EFLAGS: 00000202
> 2018-01-06T17:10:30.192994+00:00 node-115 kernel: [87913.155161] RAX:
> 0000000000000101 RBX: ffff8820046c83f0 RCX: 0000000000000001
> 2018-01-06T17:10:30.192996+00:00 node-115 kernel: [87913.155162] RDX:
> 0000000000000101 RSI: 0000000000000001 RDI: ffff8820046c83ec
> 2018-01-06T17:10:30.193019+00:00 node-115 kernel: [87913.155163] RBP:
> ffff88203f143c30 R08: 0000000000000101 R09: ffffffff811924a7
> 2018-01-06T17:10:30.193023+00:00 node-115 kernel: [87913.155164] R10:
> ffffea0040d6d680 R11: 0000000000000800 R12: ffff8820046c83ec
> 2018-01-06T17:10:30.193024+00:00 node-115 kernel: [87913.155165] R13:
> 0000000000000800 R14: 000000004c63ee00 R15: 0000000000000800
> 2018-01-06T17:10:30.193026+00:00 node-115 kernel: [87913.155166] FS:
> 00007fbcbb7eec00(0000) GS:ffff88203f140000(0000) knlGS:0000000000000000
> 2018-01-06T17:10:30.193028+00:00 node-115 kernel: [87913.155167] CS:
> 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2018-01-06T17:10:30.193030+00:00 node-115 kernel: [87913.155168] CR2:
> 00007f54266a8000 CR3: 0000000fcc2f2000 CR4: 00000000001426e0
> 2018-01-06T17:10:30.193032+00:00 node-115 kernel: [87913.155169] Stack:
> 2018-01-06T17:10:30.193034+00:00 node-115 kernel: [87913.155170]
> ffff88203f143c40 ffffffff81844421 ffff88203f143c60 ffffffff81842535
> 2018-01-06T17:10:30.193036+00:00 node-115 kernel: [87913.155172]
> ffff881e88a9ca80 ffff8820046c84b0 ffff88203f143c70 ffffffff8184257b
> 2018-01-06T17:10:30.193037+00:00 node-115 kernel: [87913.155173]
> ffff88203f143ca0 ffffffffc074158d ffff881e5d3beb80 0000000000000800
> 2018-01-06T17:10:30.193039+00:00 node-115 kernel: [87913.155175] Call
> Trace:
> 2018-01-06T17:10:30.193040+00:00 node-115 kernel: [87913.155176]  <IRQ>
> 2018-01-06T17:10:30.193042+00:00 node-115 kernel: [87913.155183]
> [<ffffffff81844421>] _raw_spin_lock+0x21/0x30
> 2018-01-06T17:10:30.193044+00:00 node-115 kernel: [87913.155186]
> [<ffffffff81842535>] __mutex_unlock_slowpath+0x25/0x50
> 2018-01-06T17:10:30.193046+00:00 node-115 kernel: [87913.155187]
> [<ffffffff8184257b>] mutex_unlock+0x1b/0x20
> 2018-01-06T17:10:30.193047+00:00 node-115 kernel: [87913.155224]
> [<ffffffffc074158d>] ocfs2_dio_end_io+0x6d/0x80 [ocfs2]
> 2018-01-06T17:10:30.193049+00:00 node-115 kernel: [87913.155228]
> [<ffffffff8124e57c>] dio_complete+0x11c/0x1c0
> 2018-01-06T17:10:30.193051+00:00 node-115 kernel: [87913.155230]
> [<ffffffff8124e693>] dio_bio_end_aio+0x73/0x100
> 2018-01-06T17:10:30.193053+00:00 node-115 kernel: [87913.155233]
> [<ffffffff813c3edf>] bio_endio+0x3f/0x60
> 2018-01-06T17:10:30.193055+00:00 node-115 kernel: [87913.155235]
> [<ffffffff813cb897>] blk_update_request+0x87/0x310
> 2018-01-06T17:10:30.193058+00:00 node-115 kernel: [87913.155239]
> [<ffffffff816bbd66>] end_clone_bio+0x46/0x70
> 2018-01-06T17:10:30.193060+00:00 node-115 kernel: [87913.155240]
> [<ffffffff813c3edf>] bio_endio+0x3f/0x60
> 2018-01-06T17:10:30.193062+00:00 node-115 kernel: [87913.155242]
> [<ffffffff813cb897>] blk_update_request+0x87/0x310
> 2018-01-06T17:10:30.193064+00:00 node-115 kernel: [87913.155245]
> [<ffffffff815c52f3>] scsi_end_request+0x33/0x1d0
> 2018-01-06T17:10:30.193066+00:00 node-115 kernel: [87913.155247]
> [<ffffffff815c8a26>] scsi_io_completion+0x1b6/0x690
> 2018-01-06T17:10:30.193068+00:00 node-115 kernel: [87913.155251]
> [<ffffffff810beb46>] ? rebalance_domains+0x166/0x2d0
> 2018-01-06T17:10:30.193069+00:00 node-115 kernel: [87913.155254]
> [<ffffffff815bf64f>] scsi_finish_command+0xcf/0x120
> 2018-01-06T17:10:30.193070+00:00 node-115 kernel: [87913.155256]
> [<ffffffff815c81b4>] scsi_softirq_done+0x124/0x150
> 2018-01-06T17:10:30.193071+00:00 node-115 kernel: [87913.155258]
> [<ffffffff813d3787>] blk_done_softirq+0x87/0xb0
> 2018-01-06T17:10:30.193087+00:00 node-115 kernel: [87913.155263]
> [<ffffffff81085dc1>] __do_softirq+0x101/0x290
> 2018-01-06T17:10:30.193090+00:00 node-115 kernel: [87913.155265]
> [<ffffffff810860c3>] irq_exit+0xa3/0xb0
> 2018-01-06T17:10:30.193092+00:00 node-115 kernel: [87913.155269]
> [<ffffffff81050e93>] smp_call_function_single_interrupt+0x33/0x40
> 2018-01-06T17:10:30.193094+00:00 node-115 kernel: [87913.155270]
> [<ffffffff81845ae2>] call_function_single_interrupt+0x82/0x90
> 2018-01-06T17:10:30.193095+00:00 node-115 kernel: [87913.155271]  <EOI>
> 2018-01-06T17:10:30.193096+00:00 node-115 kernel: [87913.155273]
> [<ffffffff81844414>] ? _raw_spin_lock+0x14/0x30
> 2018-01-06T17:10:30.193098+00:00 node-115 kernel: [87913.155275]
> [<ffffffff81842422>] __mutex_lock_slowpath+0x72/0x130
> 2018-01-06T17:10:30.193099+00:00 node-115 kernel: [87913.155289]
> [<ffffffffc0758099>] ? ocfs2_inode_unlock+0x119/0x120 [ocfs2]
> 2018-01-06T17:10:30.193101+00:00 node-115 kernel: [87913.155291]
> [<ffffffff818424ff>] mutex_lock+0x1f/0x30
> 2018-01-06T17:10:30.193102+00:00 node-115 kernel: [87913.155303]
> [<ffffffffc076177a>] ocfs2_file_write_iter+0x95a/0xdf0 [ocfs2]
> 2018-01-06T17:10:30.193104+00:00 node-115 kernel: [87913.155306]
> [<ffffffff812252c0>] ? poll_select_copy_remaining+0x140/0x140
> 2018-01-06T17:10:30.193105+00:00 node-115 kernel: [87913.155317]
> [<ffffffffc0760e20>] ? ocfs2_check_range_for_refcount+0x150/0x150 [ocfs2]
> 2018-01-06T17:10:30.193106+00:00 node-115 kernel: [87913.155321]
> [<ffffffff812613ea>] aio_run_iocb+0x26a/0x2d0
> 2018-01-06T17:10:30.193107+00:00 node-115 kernel: [87913.155324]
> [<ffffffff8122e8e5>] ? __fget_light+0x25/0x60
> 2018-01-06T17:10:30.193108+00:00 node-115 kernel: [87913.155325]
> [<ffffffff8122e933>] ? __fdget+0x13/0x20
> 2018-01-06T17:10:30.193109+00:00 node-115 kernel: [87913.155327]
> [<ffffffff812622cf>] do_io_submit+0x25f/0x500
> 2018-01-06T17:10:30.193109+00:00 node-115 kernel: [87913.155329]
> [<ffffffff81262580>] SyS_io_submit+0x10/0x20
> 2018-01-06T17:10:30.193110+00:00 node-115 kernel: [87913.155331]
> [<ffffffff818446b2>] entry_SYSCALL_64_fastpath+0x16/0x71
> 2018-01-06T17:10:30.193111+00:00 node-115 kernel: [87913.155332] Code: 8b
> 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0 74 f6 c7 40 08 01 00 00 00 e9 63
> ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90 8b 07 <84> c0 75 f8 b8 01 00
> 00 00 66 89 07 5d c3 0f 1f 40 00 0f 1f 44
>
>
>
>
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20180125/99d5a5b3/attachment-0001.html 