[Tmem-devel] Tmem [PATCH 0/5] (Take 3): Transcendent memory

[Dan Magenheimer]

> >> Swapping to hypervisor is mainly useful to overcome
> >> 'static partitioning' problem you mentioned in article:
> >> http://oss.oracle.com/projects/tmem/
> >> ...such 'para-swap' can shrink/expand outside of VM constraints.
> > 
> > Frontswap is very different than "hypervisor swapping" as what's
> > done by VMware as a side-effect of transparent page-sharing.  With
> > frontswap, the kernel still decides which pages are swapped out.
> > If frontswap says there is space, the swap goes "fast" to tmem;
> > if not, the kernel writes it to its own swapdisk.  So there's
> > no "double paging" or random page selection/swapping.  On
> > the downside, kernels must have real swap configured and,
> > to avoid DoS issues, frontswap is limited by the same constraint
> > as ballooning (ie. can NOT expand outside of VM constraints).
> > 
> 
> I think I did not explain my point regarding "para-swap" correctly.
> What I meant was a virtual swap device which appears as swap disk
> to kernel. Kernel swaps to this disk as usual (hence the 
> kernel decides
> what pages to swap out). This device tries to send these 
> pages to hvisor
> but if that fails, it will fall back to swapping inside guest 
> only. There
> is no double swapping. I think this correctly explains 
> purpose of Frontswap.

I suppose this is a question of policy rather than mechanism,
but since tmem "charges" pages in frontswap to the guest (because
they are persistent and can't be discarded by tmem unless the
domain dies), it is most (maybe only) useful when the guest
has ballooned down to less than its maximum memory constraint
and the memory can't be immediately given back to the guest
(e.g. because ballooning isn't instantaneous).  If the memory
CAN be given back to the guest, I think it SHOULD be given back
to the guest... with one exception being if the virtual swap
disk can be transparently compressed.

Since compression can be done either by ramzswap or by frontswap,
I'm not sure I see any advantage in implementing ramzswap if
frontswap is already implemented in the same guest.

But... I guess I will withhold further judgement until
you have a chance to prototype it.

> >> ...such 'para-swap' can shrink/expand outside of VM constraints.
> <snip>
> > frontswap is limited by the same constraint
> > as ballooning (ie. can NOT expand outside of VM constraints).
> 
> What I meant was: A VM can have 512M of memory while this 
> "para swap" disk
> can have any size, say 1G. Now kernel can swapout 1G worth of 
> data to this
> swap which can (potentially) send all these pages to hvisor. 
> In future, if
> we want even more RAM for this VM, we can add another such 
> swap device to guest.
> Thus, in a way, we are able to overcome rigid static 
> partitioning of VMs w.r.t.
> memory resource.

But the memory in the virtual swap disk must be persistent.
So you quickly get into DoS issues, where one guest essentially
grabs all the spare memory in the physical system, leaving
none for other guests or for the hypervisor to create more
guests.  That's why tmem "policy" "charges" persistent pool
pages against the guests maximum memory.

> If we find virtswap to be feasible with virtio, we can go for fs-cache
> backend we talked about.

I'm actually much more interested in the fs-cache backend
as, if that can be done, there is likely much more end-user
value (e.g. NFS/AFS support).

Thanks,
Dan