[Ocfs2-users] issues with my ocfs2 cluster

Wed Jan 10 11:03:25 PST 2018

Changwei explained to me that 4.4.0 did not have the bug that he was
referencing with 4.4.9.
I would consider using 4.4.10. I just need to be wary of breaking anything
in the openstack/kvm infrastructure by making a kernel change.

the most recent occurrence of this spin lock I definitely know there were
no servers in the cluster that were rebooted or lost power...

thanks

-- Jim

On Wed, Jan 10, 2018 at 1:47 PM, Luis Freitas <lfreitas34 at yahoo.com> wrote:

> Hi Jim,
>
>       Have you tried to update the kernel as suggested by Changwei?
>
>       The messages seem to indicate kernel 4.4.0, a google search shows
> this ubuntu version should be able to use kernel 4.10.
>
> Best Regards,
> Luis
>
>
> On Wednesday, January 10, 2018 4:12 PM, Jim Okken <jim at jokken.com> wrote:
>
>
> hello again list,
>
> We seem to be having issues on more servers where according to the linux
> developers here: "the kernel is stuck in a spin lock during a disk
> operation."
>
> The call traces are below, I see a lot of ocfs in the call traces, but I
> don't know how to read them, please tell me does the issue come from ocfs?
> thanks
> --Jim
>
> 2018-01-06T17:10:02.194362+00:00 node-115 kernel: [87885.155288] Modules
> linked in: vhost_net vhost macvtap macvlan ip6table_raw xt_mac xt_tcpudp
> xt_physdev br_netfilter veth ebtable_filter ebtables openvswitch ocfs2
> quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
> ocfs2_stackglue configfs ip6table_filter ip6_tables xt_multiport
> xt_conntrack iptable_filter xt_comment xt_CT iptable_raw ip_tables x_tables
> xfs bridge 8021q garp mrp stp llc intel_rapl x86_pkg_temp_thermal
> intel_powerclamp coretemp crct10dif_pclmul ipmi_ssif crc32_pclmul
> ghash_clmulni_intel kvm_intel aesni_intel aes_x86_64 kvm lrw gf128mul
> glue_helper ablk_helper irqbypass cryptd hpilo 8250_fintek serio_raw
> ioatdma ipmi_si sb_edac edac_core ipmi_msghandler shpchp dca
> acpi_power_meter lpc_ich mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad
> ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> nf_conntrack_proto_gre nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4
> nf_defrag_ipv4 nf_conntrack autofs4 raid10 raid456 async_raid6_recov
> async_memcpy async_pq async_xor async_tx xor dm_round_robin ses enclosure
> scsi_transport_sas raid6_pq libcrc32c raid1 raid0 multipath linear uas
> usb_storage psmouse lpfc be2net vxlan ip6_udp_tunnel scsi_transport_fc
> udp_tunnel wmi fjes scsi_dh_emc scsi_dh_rdac scsi_dh_alua dm_multipath
> 2018-01-06T17:10:02.194364+00:00 node-115 kernel: [87885.157143] CPU: 15
> PID: 11936 Comm: qemu-system-x86 Not tainted 4.4.0-98-generic #121-Ubuntu
> 2018-01-06T17:10:02.194366+00:00 node-115 kernel: [87885.157144] Hardware
> name: HP ProLiant BL460c Gen9, BIOS I36 02/17/2017
> 2018-01-06T17:10:02.194367+00:00 node-115 kernel: [87885.157280] task:
> ffff882036ff0000 ti: ffff881f80ca0000 task.ti: ffff881f80ca0000
> 2018-01-06T17:10:02.194400+00:00 node-115 kernel: [87885.157281] RIP:
> 0010:[<ffffffff810cb27c>]  [<ffffffff810cb27c>] native_queued_spin_lock_
> slowpath+0x15c/0x170
> 2018-01-06T17:10:02.194414+00:00 node-115 kernel: [87885.157566] RSP:
> 0018:ffff88203f143c30  EFLAGS: 00000202
> 2018-01-06T17:10:02.194416+00:00 node-115 kernel: [87885.157567] RAX:
> 0000000000000101 RBX: ffff8820046c83f0 RCX: 0000000000000001
> 2018-01-06T17:10:02.194418+00:00 node-115 kernel: [87885.157705] RDX:
> 0000000000000101 RSI: 0000000000000001 RDI: ffff8820046c83ec
> 2018-01-06T17:10:02.194440+00:00 node-115 kernel: [87885.157705] RBP:
> ffff88203f143c30 R08: 0000000000000101 R09: ffffffff811924a7
> 2018-01-06T17:10:02.194442+00:00 node-115 kernel: [87885.157706] R10:
> ffffea0040d6d680 R11: 0000000000000800 R12: ffff8820046c83ec
> 2018-01-06T17:10:02.194443+00:00 node-115 kernel: [87885.157707] R13:
> 0000000000000800 R14: 000000004c63ee00 R15: 0000000000000800
> 2018-01-06T17:10:02.194444+00:00 node-115 kernel: [87885.157708] FS:
> 00007fbcbb7eec00(0000) GS:ffff88203f140000(0000) knlGS:0000000000000000
> 2018-01-06T17:10:02.194444+00:00 node-115 kernel: [87885.157709] CS:
> 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2018-01-06T17:10:02.194445+00:00 node-115 kernel: [87885.157710] CR2:
> 00007f54266a8000 CR3: 0000000fcc2f2000 CR4: 00000000001426e0
> 2018-01-06T17:10:02.194446+00:00 node-115 kernel: [87885.157711] Stack:
> 2018-01-06T17:10:02.194448+00:00 node-115 kernel: [87885.157712]
> ffff88203f143c40 ffffffff81844421 ffff88203f143c60 ffffffff81842535
> 2018-01-06T17:10:02.194449+00:00 node-115 kernel: [87885.157714]
> ffff881e88a9ca80 ffff8820046c84b0 ffff88203f143c70 ffffffff8184257b
> 2018-01-06T17:10:02.194450+00:00 node-115 kernel: [87885.157716]
> ffff88203f143ca0 ffffffffc074158d ffff881e5d3beb80 0000000000000800
> 2018-01-06T17:10:02.194450+00:00 node-115 kernel: [87885.157717] Call
> Trace:
> 2018-01-06T17:10:02.194451+00:00 node-115 kernel: [87885.157718]  <IRQ>
> 2018-01-06T17:10:02.194453+00:00 node-115 kernel: [87885.157725]
> [<ffffffff81844421>] _raw_spin_lock+0x21/0x30
> 2018-01-06T17:10:02.194454+00:00 node-115 kernel: [87885.157727]
> [<ffffffff81842535>] __mutex_unlock_slowpath+0x25/0x50
> 2018-01-06T17:10:02.194456+00:00 node-115 kernel: [87885.157729]
> [<ffffffff8184257b>] mutex_unlock+0x1b/0x20
> 2018-01-06T17:10:02.194457+00:00 node-115 kernel: [87885.157766]
> [<ffffffffc074158d>] ocfs2_dio_end_io+0x6d/0x80 [ocfs2]
> 2018-01-06T17:10:02.194458+00:00 node-115 kernel: [87885.157770]
> [<ffffffff8124e57c>] dio_complete+0x11c/0x1c0
> 2018-01-06T17:10:02.194460+00:00 node-115 kernel: [87885.157771]
> [<ffffffff8124e693>] dio_bio_end_aio+0x73/0x100
> 2018-01-06T17:10:02.194461+00:00 node-115 kernel: [87885.157774]
> [<ffffffff813c3edf>] bio_endio+0x3f/0x60
> 2018-01-06T17:10:02.194463+00:00 node-115 kernel: [87885.157777]
> [<ffffffff813cb897>] blk_update_request+0x87/0x310
> 2018-01-06T17:10:02.194464+00:00 node-115 kernel: [87885.157780]
> [<ffffffff816bbd66>] end_clone_bio+0x46/0x70
> 2018-01-06T17:10:02.194465+00:00 node-115 kernel: [87885.157782]
> [<ffffffff813c3edf>] bio_endio+0x3f/0x60
> 2018-01-06T17:10:02.194465+00:00 node-115 kernel: [87885.157783]
> [<ffffffff813cb897>] blk_update_request+0x87/0x310
> 2018-01-06T17:10:02.194467+00:00 node-115 kernel: [87885.157786]
> [<ffffffff815c52f3>] scsi_end_request+0x33/0x1d0
> 2018-01-06T17:10:02.194468+00:00 node-115 kernel: [87885.157788]
> [<ffffffff815c8a26>] scsi_io_completion+0x1b6/0x690
> 2018-01-06T17:10:02.194469+00:00 node-115 kernel: [87885.157792]
> [<ffffffff810beb46>] ? rebalance_domains+0x166/0x2d0
> 2018-01-06T17:10:02.194470+00:00 node-115 kernel: [87885.157795]
> [<ffffffff815bf64f>] scsi_finish_command+0xcf/0x120
> 2018-01-06T17:10:02.194471+00:00 node-115 kernel: [87885.157796]
> [<ffffffff815c81b4>] scsi_softirq_done+0x124/0x150
> 2018-01-06T17:10:02.194471+00:00 node-115 kernel: [87885.157799]
> [<ffffffff813d3787>] blk_done_softirq+0x87/0xb0
> 2018-01-06T17:10:02.194480+00:00 node-115 kernel: [87885.157803]
> [<ffffffff81085dc1>] __do_softirq+0x101/0x290
> 2018-01-06T17:10:02.194481+00:00 node-115 kernel: [87885.157805]
> [<ffffffff810860c3>] irq_exit+0xa3/0xb0
> 2018-01-06T17:10:02.194483+00:00 node-115 kernel: [87885.157809]
> [<ffffffff81050e93>] smp_call_function_single_interrupt+0x33/0x40
> 2018-01-06T17:10:02.194483+00:00 node-115 kernel: [87885.157811]
> [<ffffffff81845ae2>] call_function_single_interrupt+0x82/0x90
> 2018-01-06T17:10:02.194484+00:00 node-115 kernel: [87885.157812]  <EOI>
> 2018-01-06T17:10:02.194484+00:00 node-115 kernel: [87885.157814]
> [<ffffffff81844414>] ? _raw_spin_lock+0x14/0x30
> 2018-01-06T17:10:02.194486+00:00 node-115 kernel: [87885.157815]
> [<ffffffff81842422>] __mutex_lock_slowpath+0x72/0x130
> 2018-01-06T17:10:02.194487+00:00 node-115 kernel: [87885.157829]
> [<ffffffffc0758099>] ? ocfs2_inode_unlock+0x119/0x120 [ocfs2]
> 2018-01-06T17:10:02.194488+00:00 node-115 kernel: [87885.157831]
> [<ffffffff818424ff>] mutex_lock+0x1f/0x30
> 2018-01-06T17:10:02.194489+00:00 node-115 kernel: [87885.157843]
> [<ffffffffc076177a>] ocfs2_file_write_iter+0x95a/0xdf0 [ocfs2]
> 2018-01-06T17:10:02.194490+00:00 node-115 kernel: [87885.157847]
> [<ffffffff812252c0>] ? poll_select_copy_remaining+0x140/0x140
> 2018-01-06T17:10:02.194490+00:00 node-115 kernel: [87885.157858]
> [<ffffffffc0760e20>] ? ocfs2_check_range_for_refcount+0x150/0x150 [ocfs2]
> 2018-01-06T17:10:02.194492+00:00 node-115 kernel: [87885.157862]
> [<ffffffff812613ea>] aio_run_iocb+0x26a/0x2d0
> 2018-01-06T17:10:02.194493+00:00 node-115 kernel: [87885.157865]
> [<ffffffff8122e8e5>] ? __fget_light+0x25/0x60
> 2018-01-06T17:10:02.194495+00:00 node-115 kernel: [87885.157867]
> [<ffffffff8122e933>] ? __fdget+0x13/0x20
> 2018-01-06T17:10:02.194496+00:00 node-115 kernel: [87885.157868]
> [<ffffffff812622cf>] do_io_submit+0x25f/0x500
> 2018-01-06T17:10:02.194497+00:00 node-115 kernel: [87885.157871]
> [<ffffffff81262580>] SyS_io_submit+0x10/0x20
> 2018-01-06T17:10:02.194499+00:00 node-115 kernel: [87885.157873]
> [<ffffffff818446b2>] entry_SYSCALL_64_fastpath+0x16/0x71
> 2018-01-06T17:10:02.194500+00:00 node-115 kernel: [87885.157874] Code: 01
> 48 8b 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0 74 f6 c7 40 08 01 00 00 00
> e9 63 ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90 <8b> 07 84 c0 75 f8 b8
> 01 00 00 00 66 89 07 5d c3 0f 1f 40 00 0f
> 2018-01-06T17:10:30.192979+00:00 node-115 kernel: [87913.154413] Modules
> linked in: vhost_net vhost macvtap macvlan ip6table_raw xt_mac xt_tcpudp
> xt_physdev br_netfilter veth ebtable_filter ebtables openvswitch ocfs2
> quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
> ocfs2_stackglue configfs ip6table_filter ip6_tables xt_multiport
> xt_conntrack iptable_filter xt_comment xt_CT iptable_raw ip_tables x_tables
> xfs bridge 8021q garp mrp stp llc intel_rapl x86_pkg_temp_thermal
> intel_powerclamp coretemp crct10dif_pclmul ipmi_ssif crc32_pclmul
> ghash_clmulni_intel kvm_intel aesni_intel aes_x86_64 kvm lrw gf128mul
> glue_helper ablk_helper irqbypass cryptd hpilo 8250_fintek serio_raw
> ioatdma ipmi_si sb_edac edac_core ipmi_msghandler shpchp dca
> acpi_power_meter lpc_ich mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad
> ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> nf_conntrack_proto_gre nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4
> nf_defrag_ipv4 nf_conntrack autofs4 raid10 raid456 async_raid6_recov
> async_memcpy async_pq async_xor async_tx xor dm_round_robin ses enclosure
> scsi_transport_sas raid6_pq libcrc32c raid1 raid0 multipath linear uas
> usb_storage psmouse lpfc be2net vxlan ip6_udp_tunnel scsi_transport_fc
> udp_tunnel wmi fjes scsi_dh_emc scsi_dh_rdac scsi_dh_alua dm_multipath
> 2018-01-06T17:10:30.192984+00:00 node-115 kernel: [87913.155150] CPU: 15
> PID: 11936 Comm: qemu-system-x86 Tainted: G             L  4.4.0-98-generic
> #121-Ubuntu
> 2018-01-06T17:10:30.192987+00:00 node-115 kernel: [87913.155151] Hardware
> name: HP ProLiant BL460c Gen9, BIOS I36 02/17/2017
> 2018-01-06T17:10:30.192988+00:00 node-115 kernel: [87913.155153] task:
> ffff882036ff0000 ti: ffff881f80ca0000 task.ti: ffff881f80ca0000
> 2018-01-06T17:10:30.192990+00:00 node-115 kernel: [87913.155154] RIP:
> 0010:[<ffffffff810cb27e>]  [<ffffffff810cb27e>] native_queued_spin_lock_
> slowpath+0x15e/0x170
> 2018-01-06T17:10:30.192992+00:00 node-115 kernel: [87913.155160] RSP:
> 0018:ffff88203f143c30  EFLAGS: 00000202
> 2018-01-06T17:10:30.192994+00:00 node-115 kernel: [87913.155161] RAX:
> 0000000000000101 RBX: ffff8820046c83f0 RCX: 0000000000000001
> 2018-01-06T17:10:30.192996+00:00 node-115 kernel: [87913.155162] RDX:
> 0000000000000101 RSI: 0000000000000001 RDI: ffff8820046c83ec
> 2018-01-06T17:10:30.193019+00:00 node-115 kernel: [87913.155163] RBP:
> ffff88203f143c30 R08: 0000000000000101 R09: ffffffff811924a7
> 2018-01-06T17:10:30.193023+00:00 node-115 kernel: [87913.155164] R10:
> ffffea0040d6d680 R11: 0000000000000800 R12: ffff8820046c83ec
> 2018-01-06T17:10:30.193024+00:00 node-115 kernel: [87913.155165] R13:
> 0000000000000800 R14: 000000004c63ee00 R15: 0000000000000800
> 2018-01-06T17:10:30.193026+00:00 node-115 kernel: [87913.155166] FS:
> 00007fbcbb7eec00(0000) GS:ffff88203f140000(0000) knlGS:0000000000000000
> 2018-01-06T17:10:30.193028+00:00 node-115 kernel: [87913.155167] CS:
> 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2018-01-06T17:10:30.193030+00:00 node-115 kernel: [87913.155168] CR2:
> 00007f54266a8000 CR3: 0000000fcc2f2000 CR4: 00000000001426e0
> 2018-01-06T17:10:30.193032+00:00 node-115 kernel: [87913.155169] Stack:
> 2018-01-06T17:10:30.193034+00:00 node-115 kernel: [87913.155170]
> ffff88203f143c40 ffffffff81844421 ffff88203f143c60 ffffffff81842535
> 2018-01-06T17:10:30.193036+00:00 node-115 kernel: [87913.155172]
> ffff881e88a9ca80 ffff8820046c84b0 ffff88203f143c70 ffffffff8184257b
> 2018-01-06T17:10:30.193037+00:00 node-115 kernel: [87913.155173]
> ffff88203f143ca0 ffffffffc074158d ffff881e5d3beb80 0000000000000800
> 2018-01-06T17:10:30.193039+00:00 node-115 kernel: [87913.155175] Call
> Trace:
> 2018-01-06T17:10:30.193040+00:00 node-115 kernel: [87913.155176]  <IRQ>
> 2018-01-06T17:10:30.193042+00:00 node-115 kernel: [87913.155183]
> [<ffffffff81844421>] _raw_spin_lock+0x21/0x30
> 2018-01-06T17:10:30.193044+00:00 node-115 kernel: [87913.155186]
> [<ffffffff81842535>] __mutex_unlock_slowpath+0x25/0x50
> 2018-01-06T17:10:30.193046+00:00 node-115 kernel: [87913.155187]
> [<ffffffff8184257b>] mutex_unlock+0x1b/0x20
> 2018-01-06T17:10:30.193047+00:00 node-115 kernel: [87913.155224]
> [<ffffffffc074158d>] ocfs2_dio_end_io+0x6d/0x80 [ocfs2]
> 2018-01-06T17:10:30.193049+00:00 node-115 kernel: [87913.155228]
> [<ffffffff8124e57c>] dio_complete+0x11c/0x1c0
> 2018-01-06T17:10:30.193051+00:00 node-115 kernel: [87913.155230]
> [<ffffffff8124e693>] dio_bio_end_aio+0x73/0x100
> 2018-01-06T17:10:30.193053+00:00 node-115 kernel: [87913.155233]
> [<ffffffff813c3edf>] bio_endio+0x3f/0x60
> 2018-01-06T17:10:30.193055+00:00 node-115 kernel: [87913.155235]
> [<ffffffff813cb897>] blk_update_request+0x87/0x310
> 2018-01-06T17:10:30.193058+00:00 node-115 kernel: [87913.155239]
> [<ffffffff816bbd66>] end_clone_bio+0x46/0x70
> 2018-01-06T17:10:30.193060+00:00 node-115 kernel: [87913.155240]
> [<ffffffff813c3edf>] bio_endio+0x3f/0x60
> 2018-01-06T17:10:30.193062+00:00 node-115 kernel: [87913.155242]
> [<ffffffff813cb897>] blk_update_request+0x87/0x310
> 2018-01-06T17:10:30.193064+00:00 node-115 kernel: [87913.155245]
> [<ffffffff815c52f3>] scsi_end_request+0x33/0x1d0
> 2018-01-06T17:10:30.193066+00:00 node-115 kernel: [87913.155247]
> [<ffffffff815c8a26>] scsi_io_completion+0x1b6/0x690
> 2018-01-06T17:10:30.193068+00:00 node-115 kernel: [87913.155251]
> [<ffffffff810beb46>] ? rebalance_domains+0x166/0x2d0
> 2018-01-06T17:10:30.193069+00:00 node-115 kernel: [87913.155254]
> [<ffffffff815bf64f>] scsi_finish_command+0xcf/0x120
> 2018-01-06T17:10:30.193070+00:00 node-115 kernel: [87913.155256]
> [<ffffffff815c81b4>] scsi_softirq_done+0x124/0x150
> 2018-01-06T17:10:30.193071+00:00 node-115 kernel: [87913.155258]
> [<ffffffff813d3787>] blk_done_softirq+0x87/0xb0
> 2018-01-06T17:10:30.193087+00:00 node-115 kernel: [87913.155263]
> [<ffffffff81085dc1>] __do_softirq+0x101/0x290
> 2018-01-06T17:10:30.193090+00:00 node-115 kernel: [87913.155265]
> [<ffffffff810860c3>] irq_exit+0xa3/0xb0
> 2018-01-06T17:10:30.193092+00:00 node-115 kernel: [87913.155269]
> [<ffffffff81050e93>] smp_call_function_single_interrupt+0x33/0x40
> 2018-01-06T17:10:30.193094+00:00 node-115 kernel: [87913.155270]
> [<ffffffff81845ae2>] call_function_single_interrupt+0x82/0x90
> 2018-01-06T17:10:30.193095+00:00 node-115 kernel: [87913.155271]  <EOI>
> 2018-01-06T17:10:30.193096+00:00 node-115 kernel: [87913.155273]
> [<ffffffff81844414>] ? _raw_spin_lock+0x14/0x30
> 2018-01-06T17:10:30.193098+00:00 node-115 kernel: [87913.155275]
> [<ffffffff81842422>] __mutex_lock_slowpath+0x72/0x130
> 2018-01-06T17:10:30.193099+00:00 node-115 kernel: [87913.155289]
> [<ffffffffc0758099>] ? ocfs2_inode_unlock+0x119/0x120 [ocfs2]
> 2018-01-06T17:10:30.193101+00:00 node-115 kernel: [87913.155291]
> [<ffffffff818424ff>] mutex_lock+0x1f/0x30
> 2018-01-06T17:10:30.193102+00:00 node-115 kernel: [87913.155303]
> [<ffffffffc076177a>] ocfs2_file_write_iter+0x95a/0xdf0 [ocfs2]
> 2018-01-06T17:10:30.193104+00:00 node-115 kernel: [87913.155306]
> [<ffffffff812252c0>] ? poll_select_copy_remaining+0x140/0x140
> 2018-01-06T17:10:30.193105+00:00 node-115 kernel: [87913.155317]
> [<ffffffffc0760e20>] ? ocfs2_check_range_for_refcount+0x150/0x150 [ocfs2]
> 2018-01-06T17:10:30.193106+00:00 node-115 kernel: [87913.155321]
> [<ffffffff812613ea>] aio_run_iocb+0x26a/0x2d0
> 2018-01-06T17:10:30.193107+00:00 node-115 kernel: [87913.155324]
> [<ffffffff8122e8e5>] ? __fget_light+0x25/0x60
> 2018-01-06T17:10:30.193108+00:00 node-115 kernel: [87913.155325]
> [<ffffffff8122e933>] ? __fdget+0x13/0x20
> 2018-01-06T17:10:30.193109+00:00 node-115 kernel: [87913.155327]
> [<ffffffff812622cf>] do_io_submit+0x25f/0x500
> 2018-01-06T17:10:30.193109+00:00 node-115 kernel: [87913.155329]
> [<ffffffff81262580>] SyS_io_submit+0x10/0x20
> 2018-01-06T17:10:30.193110+00:00 node-115 kernel: [87913.155331]
> [<ffffffff818446b2>] entry_SYSCALL_64_fastpath+0x16/0x71
> 2018-01-06T17:10:30.193111+00:00 node-115 kernel: [87913.155332] Code: 8b
> 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0 74 f6 c7 40 08 01 00 00 00 e9 63
> ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90 8b 07 <84> c0 75 f8 b8 01 00
> 00 00 66 89 07 5d c3 0f 1f 40 00 0f 1f 44
>
>
>
>
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20180110/1fd5baf2/attachment-0001.html 