[Tmem-devel] [RFC] Transcendent Memory ("tmem"): a new approach to physical memory management

[Dan Magenheimer]

> I didn't want to pollute xen-devel with this since it's totally KVM 

I hope its OK if I cc tmem-devel just to archive this discussion.
You'll have to subscribe first if you cc tmem-devel (see
oss.oracle.com/projects/tmem), or I can re-post (or if you prefer
to keep this offlist, let me know).

I've read Martin's CMM2 paper from OLS'06 and have been meaning to
contact him to see if there's been any further work.  I was
unaware that it had been abandoned.  However, I think tmem is
much simpler while achieving similar goals and also has potential
to be used in additionally beneficial ways.

Although I'm far from an expert in KVM memory management, I think
you are wrong and that there is still much value for tmem on KVM,
though possibly not quite as much as on Xen or VMware or HyperV.

> In KVM, guest's don't "own" there memory.  The model is the same for 
> s390 too.  Since they never know the real physical page, they VMM can 
> remove memory from the guest at any time as long as it can guarantee 
> that it can recreate the page when the guest needs it.

The ownership and addressing differences are relevant, but I don't
think they eliminate the value of tmem.

> from Xen's.  A guest balloons memory by issuing effectively a 
> hypercall 
> that will tell the VMM that the guest doesn't care about the memory's 
> contents anymore.  A guest is free to use that memory 
> whenever it wants 
> but the page will be all zeros.

If the kernel is absolutely certain that a page will not be used
again, certainly it doesn't care, but then it wouldn't put the
page into tmem either.  However a kernel that has been aggressively
reduced in memory size (via ballooning or equivalent) will
usually only regretfully evict a page and will often have
to go fetch that page from disk again (a "false negative
eviction").

On the reverse side, how quickly can KVM feed more memory to
a needy VM, e.g. when it is on the verge of swapping?  (Or does
only the host swap?)  Tmem handles this very efficiently.

After I post the Linux patch and you've had a chance to look at
some of the other materials, please let me know if you still
feel KVM won't benefit.

Looking forward to more discussion...

Thanks,
Dan

> -----Original Message-----
> From: Anthony Liguori [mailto:anthony at codemonkey.ws]
> Sent: Thursday, January 08, 2009 3:03 PM
> To: Dan Magenheimer
> Subject: Re: [RFC] Transcendent Memory ("tmem"): a new approach to
> physical memory management
> 
> 
> Hi Dan,
> 
> Dan Magenheimer wrote:
> > At last year's Xen North America Summit in Boston, I gave a talk
> > about memory overcommitment in Xen.  I showed that the basic
> > mechanisms for moving memory between domains were already present
> > in Xen and that, with a few scripts, it was possible to roughly
> > load-balance memory between domains.  During this effort, I
> > discovered that "ballooning" had a lot of weaknesses, even
> > though it is the foundation for "time-sharing" physical
> > memory in every major virtualization system existing today.
> > These weaknesses have led in many cases to unacceptable performance
> > issues when VMs are densely packed; as a result, memory is becoming
> > the bottleneck in many deployments of virtualization.
> > 
> > Transcendent Memory -- or "tmem" for short -- is phase II of that
> > work and it essentially augments ballooning and "fixes" many of
> > its weaknesses.  It requires paravirtualization, but the changes
> > (to Linux) are fairly small and minimally-invasive.  The changes
> > to Xen are larger, but also fairly non-invasive.  (No shell scripts
> > this time! :-)  The concept and code is modular and may easily
> > port to Windows, as well as KVM.  It may even be useful in
> > containers and in a native physical operating system. And,
> > yes, it is machine-independent so should be easily portable
> > to ia64 too!
> >
> 
> I didn't want to pollute xen-devel with this since it's totally KVM 
> specific, but I took a look at the info you have and believe 
> that tmem 
> is not really applicable to KVM.
> 
> In KVM, guest's don't "own" there memory.  The model is the same for 
> s390 too.  Since they never know the real physical page, they VMM can 
> remove memory from the guest at any time as long as it can guarantee 
> that it can recreate the page when the guest needs it.
> 
> Right now, KVM feeds information from the guest's shadow 
> paging to the 
> host's mm LRU which allows the Linux mm to effectively 
> determine which 
> portions of memory are not in use and it can swap those to disk.
> 
> Additionally, our "ballooning" mechanism behaves totally differently 
> from Xen's.  A guest balloons memory by issuing effectively a 
> hypercall 
> that will tell the VMM that the guest doesn't care about the memory's 
> contents anymore.  A guest is free to use that memory 
> whenever it wants 
> but the page will be all zeros.  In the VMM, we blow away the 
> page and 
> replace it with a CoW reference to the zero page.
> 
> The s390 guys took it a lot further.  They actually updated the mm to 
> give the VMM a ton of information about guest pages including which 
> pages where resident in memory but also resident on disk.  That means 
> that the VMM could blow away that page and deliver a special fault to 
> the guest when it tried to access this page which would then 
> result in 
> the guest pulling it in again from disk.
> 
> Unfortunately, the CMM2 work was deemed too invasive so it was 
> abandoned.  However, if you're not familiar with it, you should take 
> look at it.
> 
> Regards,
> 
> Anthony Liguori
> 
>