[rds-devel] soft lockups with rds

[Richard Frank]

Can you provide the rds-stress incantations that produced these errors 
along with rds-info -c output...

Rick

Or Gerlitz wrote:
> Hi Andy,
>
> Doing some rds-stress runs with a RH5 system that uses ofed 1.3.1 I got 
> bunch of soft lockups e.
> g the one below, any idea? does this ring any bell?
>
> Or.
>
>   
>> BUG: soft lockup detected on CPU#0!
>>
>> Call Trace:
>>  <IRQ>  [<ffffffff800b2c85>] softlockup_tick+0xdb/0xed
>>  [<ffffffff800933d1>] update_process_times+0x42/0x68
>>  [<ffffffff80073d97>] smp_local_timer_interrupt+0x23/0x47
>>  [<ffffffff80074459>] smp_apic_timer_interrupt+0x41/0x47
>>  [<ffffffff8005bcc2>] apic_timer_interrupt+0x66/0x6c
>>  [<ffffffff88709bc2>] :rds:rds_ib_send_cq_comp_handler+0x0/0x2a6
>>  [<ffffffff88709ca3>] :rds:rds_ib_send_cq_comp_handler+0xe1/0x2a6
>>  [<ffffffff88709c88>] :rds:rds_ib_send_cq_comp_handler+0xc6/0x2a6
>>  [<ffffffff88309a6c>] :mlx4_core:mlx4_eq_int+0x3b/0x26f
>>  [<ffffffff88309caf>] :mlx4_core:mlx4_msi_x_interrupt+0xf/0x17
>>  [<ffffffff80010705>] handle_IRQ_event+0x29/0x58
>>  [<ffffffff800b2fc4>] __do_IRQ+0xa4/0x105
>>  [<ffffffff8006a193>] do_IRQ+0xe7/0xf5
>>  [<ffffffff8005b649>] ret_from_intr+0x0/0xa
>>  [<ffffffff8006267e>] _read_lock+0x4/0xc
>>  [<ffffffff800127d9>] sock_def_readable+0x10/0x5f
>>  [<ffffffff8001b329>] tcp_rcv_established+0x62c/0x917
>>  [<ffffffff8003aaf8>] tcp_v4_do_rcv+0x2a/0x300
>>  [<ffffffff800370f8>] ip_route_input+0xc5d/0xc8a
>>  [<ffffffff80026d1e>] tcp_v4_rcv+0x9f6/0xa5f
>>  [<ffffffff80033f5b>] ip_local_deliver+0x19d/0x263
>>  [<ffffffff80035033>] ip_rcv+0x49c/0x4df
>>  [<ffffffff8001fd92>] netif_receive_skb+0x33c/0x3ba
>>  [<ffffffff88584d9f>] :ib_ipoib:ipoib_ib_handle_rx_wc+0x407/0x43c
>>  [<ffffffff88585d8d>] :ib_ipoib:ipoib_poll+0x9f/0x18b
>>  [<ffffffff8000c39b>] net_rx_action+0xa4/0x1a5
>>  [<ffffffff80011c19>] __do_softirq+0x5e/0xd5
>>  [<ffffffff8005c330>] call_softirq+0x1c/0x28
>>  <EOI>  [<ffffffff8006a310>] do_softirq+0x2c/0x85
>>  [<ffffffff8008e619>] local_bh_enable_ip+0x48/0x59
>>  [<ffffffff8003ccaa>] rt_run_flush+0x7f/0xb8
>>  [<ffffffff8023c376>] ip_mc_dec_group+0x86/0xab
>>  [<ffffffff8023d8ff>] ip_mc_drop_socket+0x4d/0x8b
>>  [<ffffffff8023a447>] inet_release+0x1a/0x55
>>  [<ffffffff80052c24>] sock_release+0x19/0x99
>>  [<ffffffff80052e1e>] sock_close+0x2c/0x30
>>  [<ffffffff80012281>] __fput+0xae/0x198
>>  [<ffffffff8002362c>] filp_close+0x5c/0x64
>>  [<ffffffff8001d5b2>] sys_close+0x88/0xa2
>>  [<ffffffff8005f013>] sysenter_do_call+0x1b/0x67
>>
>>   
>> BUG: soft lockup detected on CPU#1!
>>
>> Call Trace:
>>  <IRQ>  [<ffffffff800b50fa>] softlockup_tick+0xd5/0xe7
>>  [<ffffffff800930e2>] update_process_times+0x42/0x68
>>  [<ffffffff800746e3>] smp_local_timer_interrupt+0x23/0x47
>>  [<ffffffff80074da5>] smp_apic_timer_interrupt+0x41/0x47
>>  [<ffffffff8005bc8e>] apic_timer_interrupt+0x66/0x6c
>>  [<ffffffff80062b70>] _write_unlock_irqrestore+0x9/0xa
>>  [<ffffffff88582b07>] :rds:rds_recv_incoming+0x1dc/0x1ed
>>  [<ffffffff8858818f>] :rds:rds_ib_recv_cq_comp_handler+0x4b5/0x752
>>  [<ffffffff88588e49>] :rds:rds_ib_send_cq_comp_handler+0x28f/0x2a6
>>  [<ffffffff88188a7f>] :mlx4_core:mlx4_eq_int+0x3b/0x26f
>>  [<ffffffff88188cc2>] :mlx4_core:mlx4_msi_x_interrupt+0xf/0x17
>>  [<ffffffff800107a0>] handle_IRQ_event+0x29/0x58
>>  [<ffffffff800b5482>] __do_IRQ+0xa4/0x105
>>  [<ffffffff8006a3bd>] do_IRQ+0xe7/0xf5
>>  [<ffffffff8005b615>] ret_from_intr+0x0/0xa
>>  [<ffffffff880229e5>] :ehci_hcd:ehci_work+0x264/0x6e3
>>  [<ffffffff880229d8>] :ehci_hcd:ehci_work+0x257/0x6e3
>>  [<ffffffff8005b615>] ret_from_intr+0x0/0xa
>>  [<ffffffff8802567c>] :ehci_hcd:ehci_irq+0x13d/0x156
>>  [<ffffffff801dadd4>] usb_hcd_irq+0x27/0x55
>>  [<ffffffff800107a0>] handle_IRQ_event+0x29/0x58
>>  [<ffffffff800b5482>] __do_IRQ+0xa4/0x105
>>  [<ffffffff880243f3>] :ehci_hcd:ehci_watchdog+0x0/0x61
>>  [<ffffffff8006a3bd>] do_IRQ+0xe7/0xf5
>>  [<ffffffff8005b615>] ret_from_intr+0x0/0xa
>>  [<ffffffff88587cda>] :rds:rds_ib_recv_cq_comp_handler+0x0/0x752
>>  [<ffffffff800928e1>] run_timer_softirq+0x12a/0x1b0
>>  [<ffffffff80011cb4>] __do_softirq+0x5e/0xd5
>>  [<ffffffff80151593>] end_msi_irq_w_maskbit+0xf/0x1c
>>  [<ffffffff8005c2fc>] call_softirq+0x1c/0x28
>>  [<ffffffff8006a53a>] do_softirq+0x2c/0x85
>>  [<ffffffff8006a3c2>] do_IRQ+0xec/0xf5
>>  [<ffffffff8005b615>] ret_from_intr+0x0/0xa
>>  <EOI>  [<ffffffff800c2f8b>] __kzalloc+0x1a/0x21
>>  [<ffffffff800c2f7a>] __kzalloc+0x9/0x21
>>  [<ffffffff88581e0b>] :rds:rds_message_alloc+0x16/0x56
>>  [<ffffffff8011c70b>] avc_has_perm+0x43/0x55
>>  [<ffffffff885820cc>] :rds:rds_message_copy_from_user+0x29/0x135
>>  [<ffffffff88583377>] :rds:rds_sendmsg+0xe6/0x4ca
>>  [<ffffffff80052bd5>] sock_sendmsg+0xf3/0x110
>>  [<ffffffff800884ac>] default_wake_function+0x0/0xe
>>  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
>>  [<ffffffff88587ba8>] :rds:rds_ib_inc_free+0x0/0x51
>>  [<ffffffff80209069>] sys_sendmsg+0x217/0x28a
>>  [<ffffffff80060fd6>] thread_return+0xad/0xeb
>>  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
>>
>>   
>>     
>
>
>
> _______________________________________________
> rds-devel mailing list
> rds-devel at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/rds-devel
>