[Tmem-users] Experiments with Tmem and trying to find out Tmem USAGE statistics

Wed May 12 07:09:09 PDT 2010

Hi Icarus(?) -

Tmem has very limited functionality on a 32-bit Xen/CPU because allocation from the Xen heap works differently.  As a result, I haven't tested it on 32-bit in a long time, though it may work for some things.  Is your CPU really 32-bit-only?  Or are you just looking at your dom0?

Also, tmem wasn't really designed to work in dom0 though, again, it may work with some functionality limitations.  It is intended to run in PV guests.

And, the linux-side tmem code is getting old... I need to update it and also provide better user documentation.

To answer your questions: 

1.        You should not see an increase in system time (in a PV guest) from tmem because nearly all the work done in tmem is done in the hypervisor.  So I suspect the increase you are seeing is due to other factors within the dom0 kernel.

2.       The output you provided shows that tmem is not really being used at all.  Assuming you are using Xen-4.0.0 tools, then xentop should also show summary numbers for tmem (see the new "t" keyboard command), but only if tmem is actually in use.  You can also use "xm tmem-list --all -long | xen-tmem-list-parse" to see more detailed information.

Hope that helps!

Dan

From: Icarus Sonic [mailto:icarus.sonic at gmail.com] 
Sent: Wednesday, May 12, 2010 1:05 AM
To: tmem-users at oss.oracle.com
Subject: [Tmem-users] Experiments with Tmem and trying to find out Tmem USAGE statistics

1. Experiment data - does it conform to your expectation?

Hi, I've installed on a Dell Optilex GX260(32bit x86 , 1G Memory) Xen-4.0.0 and linux-2.6.18.8-xen kernel dom0 with tmem patched and PRECACHE/PRESWAP opened.

I did the following experiments and found that tmem does work: I wrote a c program that touch each page of a 512M 'unsigned char' array, and measured its running time in dom0. I tried it with three different settings. The result shows that tmem works but introduce a little bit overhead. Experiment result is listed as followed:

Setting 1: tmem off(by booting xen without 'tmem' argument), dom0 with all available memory close to 1024GB.
Setting 2: tmem off(by booting xen without 'tmem' argument), dom0 with only 256MB memory. xen argument: dom0_mem=256M
Setting 3: tmem on, dom0 with only 256MB memory. xen argument: dom0_mem=256M tmem

With each setting, I ran it three times. Run time is listed in [real],[user],[sys] format

Results 
==========================================
Setting1: round1(1.503,0.025,1.466), round2(1.487,0.030,1.456), round3(1.486,0.039,1.444)
Setting2: round1(16.591,0.036,2.779), round2(10.434,0.022,2.368) round3(13.262,0.033,2.305)
Setting3: round1(8.739,0.035,4.616), round2(4.582,0.036,2.852), round3(4.604,0.043,3.592)

With setting 2,run time is 10x of that with setting 1, which is caused by swapping. Results in setting 3 demostrate that tmem alleviate swapping problem but introduced overhead. 

2. Question: Where can i found how much tmem is used by a domain, and how much tmem is available?
Tried xm tmem-list -a | xen-tmem-list-parse to see statistics of tmem usage, its output:

total tmem ops=93859 (errors=91550) -- tmem pages avail=0
datastructs: objs=0 (max=0) pgps=0 (max=0) nodes=0 (max=0)
misc: failed_copies=0 alloc_failed=0 alloc_page_failed=0 low_mem=0 evicted=0/0 relinq=0/0, max_evicts_per_relinq=0, flush_pools=0, eph_count=0, eph_max=0
domid0: weight=0,cap=0,compress=0,frozen=0,total_cycles=61882224,succ_eph_gets=0,succ_pers_puts=0,succ_pers_gets=0,eph_count=0,max_eph=0,compression ratio=0% (samples=0,poor=0,nomem=0)
domid0,id0[EP]:pgp=0(max=0) obj=0(0) objnode=0(0) puts=0/0/0(dup=0/0) gets=0/0(0%) flush=0/0 flobj=0/0
domid0,id1[EP]:pgp=0(max=0) obj=0(0) objnode=0(0) puts=0/0/0(dup=0/0) gets=0/0(0%) flush=0/0 flobj=0/0
domid0,id2[PP]:pgp=0(max=0) obj=0(0) objnode=0(0) puts=0/0/0(dup=0/0) gets=0/0(0%) flush=0/0 flobj=0/0
failed get cycles: avg=642, max=18856, min=384, samples=91550
flush cycles: avg=785, max=2812, min=668, samples=1966
flush_obj cycles: avg=4623, max=12508, min=696, samples=334

But it does not tell me how much tmem is used by a domain or how much tmem is available in hypervisor?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/tmem-users/attachments/20100512/cf0140d5/attachment.html 