[rds-devel] Has anyone tried running RDS over 10GE / IWARP NICs ?

[Richard Frank]

Steve, we would like to get RDS running over the Chelsio 10GE NICs .

It looks like there is some sort of a problem - we are assuming it is 
something simple - as RDS does not do much more than connect over an RC 
and do send / recvs.

Can you help us move this forward - either by looking at this yourself 
and or putting in contact with folks that can ?


Scott Weitzenkamp (sweitzen) wrote:
> Chelsio S310E-SR+ (see
> http://www.chelsio.com/products_10g_adapters.html), I'm sure they would
> be happy to sell you a few to use.
>
> Perhaps RDS does not use RDMA CM in a way that iWARP needs?
>
> Scott
>
>  
>
>   
>> -----Original Message-----
>> From: Richard Frank [mailto:richard.frank at oracle.com] 
>> Sent: Wednesday, April 02, 2008 11:52 AM
>> To: Scott Weitzenkamp (sweitzen)
>> Cc: rds-devel at oss.oracle.com
>> Subject: Re: [rds-devel] Has anyone tried running RDS over 
>> 10GE / IWARP NICs ?
>>
>> Who's cards are these - perhaps they will want to get RDS running and 
>> help out ?
>>
>> Scott Weitzenkamp (sweitzen) wrote:
>>     
>>> Sorry, but I doubt I will have time to rebuild RDS and try this.
>>>
>>> Scott
>>>  
>>>
>>>   
>>>       
>>>> -----Original Message-----
>>>> From: Richard Frank [mailto:richard.frank at oracle.com] 
>>>> Sent: Wednesday, April 02, 2008 11:23 AM
>>>> To: Scott Weitzenkamp (sweitzen)
>>>> Cc: rds-devel at oss.oracle.com
>>>> Subject: Re: [rds-devel] Has anyone tried running RDS over 
>>>> 10GE / IWARP NICs ?
>>>>
>>>> OK - well - I guess we are not too surprised...
>>>>
>>>> Short of us getting some these NICs to sort this out ourselves..
>>>>
>>>> Can you enable rds debug in configure.mk and send us the dmesg + 
>>>> /var/log/messages
>>>>
>>>> You will need to unload/load rds after building it with debug..
>>>>
>>>> Scott Weitzenkamp (sweitzen) wrote:
>>>>     
>>>>         
>>>>> Yes, it's an iWARP NIC, and the OFED 1.3 perftest 
>>>>>       
>>>>>           
>>>> ib_rdma_lat program is
>>>>     
>>>>         
>>>>> working.
>>>>>
>>>>> Scott
>>>>>
>>>>>
>>>>>   
>>>>>       
>>>>>           
>>>>>> -----Original Message-----
>>>>>> From: Richard Frank [mailto:richard.frank at oracle.com] 
>>>>>> Sent: Wednesday, April 02, 2008 11:04 AM
>>>>>> To: Scott Weitzenkamp (sweitzen)
>>>>>> Cc: rds-devel at oss.oracle.com; [ofa_general]
>>>>>> Subject: Re: [rds-devel] Has anyone tried running RDS over 
>>>>>> 10GE / IWARP NICs ?
>>>>>>
>>>>>> RDS does not run over regular 10G NICs - that appear as 
>>>>>>         
>>>>>>             
>>>> simple NICS - 
>>>>     
>>>>         
>>>>>> this was disabled in 1.3.
>>>>>>
>>>>>> For now we  are interested in RDS over IWARP NICS - 
>>>>>>             
>> configured as 
>>     
>>>>>> accessible via the verbs interfaces.
>>>>>>
>>>>>> Richard Frank wrote:
>>>>>>     
>>>>>>         
>>>>>>             
>>>>>>> is the rds driver loaded (modprobe rds)
>>>>>>>
>>>>>>> Scott Weitzenkamp (sweitzen) wrote:
>>>>>>>   
>>>>>>>       
>>>>>>>           
>>>>>>>               
>>>>>>>> Does't appear to work with Chelsio and OFED 1.3:
>>>>>>>>
>>>>>>>> [root at svbu-qa2950-1 counters]# ethtool -i eth2
>>>>>>>> driver: cxgb3
>>>>>>>> version: 1.0-ofed
>>>>>>>> firmware-version: T 5.0.0 TP 1.1.0
>>>>>>>> bus-info: 0000:0b:00.0
>>>>>>>> [root at svbu-qa2950-1 counters]# ifconfig eth2
>>>>>>>> eth2      Link encap:Ethernet  HWaddr 00:07:43:05:43:9F
>>>>>>>>           inet addr:192.168.0.198  Bcast:192.168.0.255
>>>>>>>> Mask:255.255.255.0
>>>>>>>>           inet6 addr: fe80::207:43ff:fe05:439f/64 Scope:Link
>>>>>>>>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>>>>>>>           RX packets:144770 errors:0 dropped:0 
>>>>>>>>                 
>> overruns:0 frame:0
>>     
>>>>>>>>           TX packets:144781 errors:0 dropped:0 
>>>>>>>>             
>>>>>>>>                 
>>>> overruns:0 carrier:0
>>>>     
>>>>         
>>>>>>>>           collisions:0 txqueuelen:1000
>>>>>>>>           RX bytes:207891512 (198.2 MiB)  TX bytes:9348152 
>>>>>>>>         
>>>>>>>>             
>>>>>>>>                 
>>>>>> (8.9 MiB)
>>>>>>     
>>>>>>         
>>>>>>             
>>>>>>>>           Interrupt:169 Memory:fceff000-fcefffff
>>>>>>>>
>>>>>>>> [root at svbu-qa2950-1 counters]# rds-sink -s 
>>>>>>>>             
>>>>>>>>                 
>>>> 192.168.0.198:22222 -i 1
>>>>     
>>>>         
>>>>>>>> rds-sink: Unable to bind socket: Cannot assign 
>>>>>>>>                 
>> requested address
>>     
>>>>>>>> Scott Weitzenkamp
>>>>>>>> SQA and Release Manager
>>>>>>>> Data Center Access Engineering
>>>>>>>> Cisco Systems
>>>>>>>>
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>>   
>>>>>>>>     
>>>>>>>>         
>>>>>>>>             
>>>>>>>>                 
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: rds-devel-bounces at oss.oracle.com 
>>>>>>>>> [mailto:rds-devel-bounces at oss.oracle.com] On Behalf Of 
>>>>>>>>>           
>>>>>>>>>               
>>>>>>>>>                   
>>>>>> Richard Frank
>>>>>>     
>>>>>>         
>>>>>>             
>>>>>>>>> Sent: Wednesday, April 02, 2008 10:31 AM
>>>>>>>>> To: rds-devel at oss.oracle.com; [ofa_general]
>>>>>>>>> Subject: [rds-devel] Has anyone tried running RDS over 10GE / 
>>>>>>>>> IWARP NICs ?
>>>>>>>>>
>>>>>>>>> We'd appreciate some feed back on your experience and would 
>>>>>>>>> like to sort 
>>>>>>>>> out any issues ASAP.
>>>>>>>>>
>>>>>>>>> Rick
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> rds-devel mailing list
>>>>>>>>> rds-devel at oss.oracle.com
>>>>>>>>> http://oss.oracle.com/mailman/listinfo/rds-devel
>>>>>>>>>
>>>>>>>>>     
>>>>>>>>>       
>>>>>>>>>           
>>>>>>>>>               
>>>>>>>>>                   
>>>>>>> _______________________________________________
>>>>>>> rds-devel mailing list
>>>>>>> rds-devel at oss.oracle.com
>>>>>>> http://oss.oracle.com/mailman/listinfo/rds-devel
>>>>>>>   
>>>>>>>       
>>>>>>>           
>>>>>>>