[rds-devel] Re: process death - with outstanding rdma operations

[Richard Frank]

Or Gerlitz wrote:
> Richard Frank wrote:
>>> From the rdsV3 interface.h "
>>> * Process death: Keys obtained by a process must be invalidated 
>>> before any
>>> * memory referenced by the keys is released from the dieing process and
>>> * before anyother process can detect the death of the dead process. 
>>> Local
>>> * memory pins must also be released.
>>> We need to cleanup the in-progress rdma ops - and or let them run 
>>> down before letting the process go away !
>
> Rick (and Olaf),
>
> I have been following the discussion and Q&A between Olaf and Roland 
> over the lists, however, I was not able to see a clear design to take 
> care of process death (or bug etc, whatever user space flow that can 
> make RDS to leak kernel/HW resources) beyond all the details discussed 
> there.
>
> Looking on the v3 document posted by Rick to this list on November 5th 
> @ http://oss.oracle.com/pipermail/rds-devel/2007-November/000139.html
> I saw the below references, but I definitely don't think we have a 
> closed design here, some not covered issues are:
>
Good points Or - rdsv3.h describes the user interface and expected 
behavior - not the driver implementation..
We need to update this doc..

> 1. death of a remote process holding keys sent by local process/es
>
The rdma server does not hold the key - it's given permission to use it 
- and the permission can be revoked at anytime 1) explicitly by the 
client 2) if the client process dies - by the RDS driver...

It's up to the client side app to decide if / when to revoke keys 
(release the keys).

> 2. whether an IB RC connection should be broken b/c one of the 
> processes using it has died or leaked resources
>
When a process dies all resources held by the driver must be cleaned up 
- this should not need to be doc'd ?
If a client process dies - and it's keys are released - and then 
subsequent use by the rdma server gets an access error - that's what we 
want - right ?

Breaking the RC is a function of the transport - not the RDS ULP / 
interface ?

One addition recently proposed (with patch) is to have per GID RCs - 
this isolates behavior to processes that agree to play together....

> and also
>
> 3. no treatment for resource leaks (process that calls get_mr and 
> never calls free_mr or remote process getting a keys but never sends 
> them back)
>
Why would the RDS driver / ULP deal with a poorly behaved client / rdma 
server ? Beyond the resource (key) quota - what else would you do ? 
Seems like a app issue ?

> 4. no limitation on how many registrations a process can do
>
We need to limit the keys allocated by a process - one proposal 
discussed is to overload mlock - an alternative is to have a per socket 
limit similar to so_snd/rcv... via a new ioctl... I prefer the later - 
as the key pool is a very limited resource - or is it ?

> 5. etc
>
> I would be very productive, I think, to have some text with a design 
> or sketch of design to handle at least 1 && 2 above, my feeling is 
> that now its more of a designing while coding/debugging...
>
> Or
>
>>    182     * Pinning of VA for the RDMA buffer for which the client 
>> obtains an FMR key -    183     * is done within rds_get_mr(va). This 
>> memory must stay pinned until the client
>>    184     * calls rds_free_mr(va) - or the process dies - in which 
>> case    185     * the driver must clean up the pins. 
>
>>    193     * Process death: Keys obtained by a process must be 
>> invalidated before any    194     * memory referenced by the keys is 
>> released from the dieing process and
>>    195     * before another process can detect the death of the dead 
>> process. Local    196     * memory pins must also be released.
>
>>    256     *     Cleanup at process death. The RDS driver must detect 
>> process death
>>    257     *     and free memory regions plus unpin memory held by 
>> the dieing process.    258     *     All resources must be cleaned up 
>> before any other process can detect
>>    259     *     the death of the dieing process. Further, the memory 
>> regions freed
>>    260     *     must be invalidated and the invalidations must be 
>> flushed thru to
>>    261     *     HCA. 
>