[rds-devel] The meaning of MR invalidation

[Richard Frank]

Is this a real issue blocking us from checking in the latest code for 
OFED 1.3 ?

If not can we please go ahead and complete the check in ?


Or Gerlitz wrote:
> Or Gerlitz wrote:
>> On 2/13/08, Olaf Kirch <olaf.kirch at oracle.com> wrote:
>>> That's one of the things I have no real understanding of. What is the
>>> actual difference in performance when you use an FMR exactly once?
>
>> Let me think about this and check with the Mellanox architects,
>
> Hi Olaf,
>
> For every incoming RDMA IB packet the HCA does TPT cache lookup.
>
> Hence, if the I/Os served by specific mappings (rkey) are large in 
> size, such that they span m >> 1 IB MTU sized packets (for example the 
> IB MTU is 2K and the I/O is 1M so 256 IB packets are needed to serve 
> the RDMA operation) after one cache miss under which the HCA have to 
> issue a look up in its network MMU, you might have all the other 
> packets being served by the cache.
>
> When there are multiple I/Os are running in parallel, and each being 
> served by different FMR --> different rkey --> different cache slots, 
> first, they all compete on the cache and second, since fmr_unmap does 
> SYNC_TPT which flushes the cache, one have to try and avoid calling 
> fmr_unmap when possible.
>
> So basically, when each fmr is remapped n times, over time, you get 
> less SYNC_TPT calls compared to the case where each fmr is mapped once 
> before moved to the unmap queue. However, if you use enough FMRs such 
> that you don't call SYNC_TPT "too much" the use-once design should 
> function quite well compared to use-n design.
>
> For example, a scheme where you have to serve 1000 1MB IOs/sec, and 
> you alloc 5k FMRs and once every 4 seconds you unmap 4K FMRs from a 
> background thread, might work quite good, but this has to be validated 
> ofcourse.
>
>
>> the flow I see for rds in that case would be something like:
>
>> rds_pool_start: alloc N FMRs
>
>> rds_pool_get: get FMR from the free list and map it
>> rds_pool_put: put the used FMR in the dirty list
>
>> rds_unmap_background_thread:  if the dirty list size > M call
>> fmr_unmap on the M FMRs in the dirty list and then return them to the 
>> free list
>
>> rds_pool_stop: unalloc N FMRS
>
> Now, if you are willing to go with that approach, it means that in 
> case core fmr pool API is enhanced such that you can --specify-- how 
> many times an fmr can be mapped before its queued for unmap, RDS 
> should be able to use this cache again and not have one of its own!
>
> Or
>
>
>
> _______________________________________________
> rds-devel mailing list
> rds-devel at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/rds-devel