I tried to use two pairs of rds-sink/gen on two nodes (ie each node runs
both sink and generator) to see how things are going under bidi scheme. It
turns out that at some point one of the couple stops moving data, eg see below
that at some point traffic stops going on this session. The other seesion
keeps working fine and I don't see much evidence for this "data hang" in
the rds-info output, see also below.
I am using the ofed 1.3 beta, etc.
Or.
Script started on Sun 02 Dec 2007 03:01:03 PM IST
# rds-gen -s 192.168.3.62:1234 -d 192.168.3.61:1234 -m 64k -f /tmp/x -i 1
Bytes sent/s Packets sent/s Bytes recv/s Packets recv/s
Bytes read/s Bytes written/s
590807040 9015 0 0 104857600 0
592183296 9036 0 0 0 0
592117760 9035 0 0 0 0
611123200 9325 0 0 0 0
621674496 9486 0 0 0 0
624492544 9529 0 0 0 0
627572736 9576 0 0 0 0
627441664 9574 0 0 0 0
627834880 9580 0 0 0 0
628162560 9585 0 0 0 0
627965952 9582 0 0 0 0
627965952 9582 0 0 0 0
628293632 9587 0 0 0 0
611975168 9338 0 0 0 0
599785472 9152 0 0 0 0
598867968 9138 0 0 0 0
594149376 9066 0 0 0 0
593952768 9063 0 0 0 0
594411520 9070 0 0 0 0
610992128 9323 0 0 0 0
600440832 9162 0 0 0 0
596443136 9101 0 0 0 0
592314368 9038 0 0 0 0
613810176 9366 0 0 0 0
623575040 9515 0 0 0 0
626917376 9566 0 0 0 0
628555776 9591 0 0 0 0
627769344 9579 0 0 0 0
628686848 9593 0 0 0 0
628228096 9586 0 0 0 0
628883456 9596 0 0 0 0
628490240 9590 0 0 0 0
628359168 9588 0 0 0 0
612368384 9344 0 0 0 0
614137856 9371 0 0 0 0
272039936 4151 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
Total:
21782790144 332379 0 0 104857600 0
Script done on Sun 02 Dec 2007 03:02:07 PM IST
Script started on Sun 02 Dec 2007 03:44:20 PM IST
# rds-sink -s 192.168.3.61:1234 -f /dev/null -i 1
Bytes sent/s Packets sent/s Bytes recv/s Packets recv/s
Bytes read/s Bytes written/s
0 0 623575040 9515 0 623575040
0 0 627900416 9581 0 627834880
0 0 628293632 9587 0 628293632
0 0 628031488 9583 0 628031488
0 0 628883456 9596 0 628883456
0 0 628490240 9590 0 628490240
0 0 628752384 9594 0 628817920
0 0 628424704 9589 0 628424704
0 0 628555776 9591 0 628490240
0 0 605290496 9236 0 605356032
0 0 607911936 9276 0 607911936
0 0 105840640 1615 0 105840640
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
Total:
0 0 6969950208 106353 0 6969950208
Script done on Sun 02 Dec 2007 03:45:13 PM IST
TCP Connections:
LocalAddr LPort RemoteAddr RPort HdrRemain DataRemain SentNxt ExpectUna SeenUna
Counters:
CounterName Value
conn_reset 19
recv_drop_bad_checksum 0
recv_drop_old_seq 0
recv_drop_no_sock 47566
recv_drop_dead_sock 0
recv_deliver_raced 0
recv_delivered 3469600
recv_queued 1734800
recv_immediate_retry 0
recv_delayed_retry 0
recv_ack_required 1781857
send_queue_empty 2506815
send_queue_full 2513650
send_sem_contention 3
send_sem_queue_raced 0
send_immediate_retry 0
send_delayed_retry 0
send_drop_acked 0
send_ack_required 2552556
page_remainder_hit 0
page_remainder_miss 0
cong_update_queued 2951
cong_update_received 18228
cong_send_error 0
cong_send_blocked 3504
ib_connect_raced 0
ib_listen_closed_stale 0
ib_tx_cq_call 6821551
ib_tx_cq_event 6845295
ib_tx_ring_full 0
ib_tx_sg_mapping_failure 0
ib_tx_stalled 0
ib_rx_cq_call 6100793
ib_rx_cq_event 31066338
ib_rx_ring_empty 1
ib_rx_refill_from_cq 6100792
ib_rx_refill_from_thread 3
ib_rx_alloc_limit 0
ib_ack_sent 1781740
ib_ack_send_failure 0
ib_ack_send_delayed 108
ib_ack_received 2523650
tcp_data_ready_calls 0
tcp_write_space_calls 0
tcp_sndbuf_full 0
tcp_connect_raced 0
tcp_listen_closed_stale 0
RDS Sockets:
BoundAddr BPort ConnAddr CPort SndBuf RcvBuf
192.168.3.62 1234 0.0.0.0 0 67584 67584
192.168.3.62 5678 0.0.0.0 0 67584 67584
0.0.0.0 0 0.0.0.0 0 67584 67584
RDS Connections:
LocalAddr RemoteAddr NextTX NextRX Flg
192.168.3.62 192.168.3.61 2557553 1782370 --C
Receive Message Queue:
LocalAddr LPort RemoteAddr RPort Seq Bytes
192.168.3.62 5678 192.168.3.61 5678 1782371 65536
Send Message Queue:
LocalAddr LPort RemoteAddr RPort Seq Bytes
Retransmit Message Queue:
LocalAddr LPort RemoteAddr RPort Seq Bytes
192.168.3.62 1234 192.168.3.61 1234 2557552 65536
TCP Connections:
LocalAddr LPort RemoteAddr RPort HdrRemain DataRemain SentNxt ExpectUna SeenUna
Counters:
CounterName Value
conn_reset 0
recv_drop_bad_checksum 0
recv_drop_old_seq 0
recv_drop_no_sock 383406
recv_drop_dead_sock 0
recv_deliver_raced 0
recv_delivered 4348227
recv_queued 2174146
recv_immediate_retry 0
recv_delayed_retry 0
recv_ack_required 2552556
send_queue_empty 1499401
send_queue_full 1489160
send_sem_contention 3
send_sem_queue_raced 0
send_immediate_retry 0
send_delayed_retry 0
send_drop_acked 0
send_ack_required 1499618
page_remainder_hit 0
page_remainder_miss 0
cong_update_queued 18226
cong_update_received 2588
cong_send_error 0
cong_send_blocked 642
ib_connect_raced 0
ib_listen_closed_stale 0
ib_tx_cq_call 5538479
ib_tx_cq_event 5541211
ib_tx_ring_full 0
ib_tx_sg_mapping_failure 0
ib_tx_stalled 0
ib_rx_cq_call 6542318
ib_rx_cq_event 41704812
ib_rx_ring_empty 0
ib_rx_refill_from_cq 6542318
ib_rx_refill_from_thread 1
ib_rx_alloc_limit 0
ib_ack_sent 2523650
ib_ack_send_failure 0
ib_ack_send_delayed 25417
ib_ack_received 1499500
tcp_data_ready_calls 0
tcp_write_space_calls 0
tcp_sndbuf_full 0
tcp_connect_raced 0
tcp_listen_closed_stale 0
RDS Sockets:
BoundAddr BPort ConnAddr CPort SndBuf RcvBuf
192.168.3.61 1234 0.0.0.0 0 67584 67584
192.168.3.61 5678 0.0.0.0 0 67584 67584
0.0.0.0 0 0.0.0.0 0 67584 67584
RDS Connections:
LocalAddr RemoteAddr NextTX NextRX Flg
192.168.3.61 192.168.3.62 1500129 2557553 --C
Receive Message Queue:
LocalAddr LPort RemoteAddr RPort Seq Bytes
Send Message Queue:
LocalAddr LPort RemoteAddr RPort Seq Bytes
Retransmit Message Queue:
LocalAddr LPort RemoteAddr RPort Seq Bytes