[DTrace-devel] [PATCH v2 2/2] unittest/sched: remove dtv2 xfail

Tue Oct 8 18:17:51 UTC 2024

On 07/10/2024 20:10, Kris Van Hees wrote:
> Any progress on this?
> 

The first approach I investigated - using a perf event BPF program to
catch sched switch software perf events - seems to be a dead end because
there is no easy way I could find to access the perf event payload to
determine if the event is a sched_in or sched_out event. There's nothing
I can see in the perf sample to provide that info, and even if we could
access it we would have to do a series of expensive
bpf_probe_read_kernel()s in a trampoline for sched;::on-cpu to determine
if it is the right event. There is no way I can see to limit perf event
firing to oncpu events, so we'd have to filter and that's expensive
since we know 50% of the time we will be in an off-cpu event. I can't
see any other obvious context we could draw upon reliably to distinguish
on- and off-cpu events in this context.

An alternative - that has other benefits aside from this specific
problem - is to support attach to .isra.0-suffixed functions. If we can
do that, we can attach to fbt::finish_task_switch[.isra.0:return to
trace the right location for on-cpu. This works on UEK7 also.

The .isra.0 functions are in available_filter_functions, and have BTF
representations (as long as they do not violate register expectations
with parameters), so having support for these would be valuable in
expanding the traceable surface of the system. There are a few wrinkles,
specifically:

- kprobe event names do not support a "." so we need to remove it
- BTF representations do not have the ".isra.0" suffix (in line with
DWARF) so we need to drop it to do BTF function lookup

Both these can be accomplished by creating a string which drops the
suffix and using that at the appropriate times. I have a rough
implementation working for kprobes, and will test fprobes shortly. What
do you think? It would be nice to expand the tracing surface, and if we
did add that, supporting sched:::on-cpu would be much easier (and
backwards-compatible to 5.15 at least).

Thanks!

Alan

>>> not allow an FBT probe on __perf_event_task_sched_in.  The failure is silent,
>>> causing the probe to simply never get enabled and no error reported, so the
>>> probe does not fire and causes the test to fail.
>>>
>>> In other words... the approach in 1/2 of this series does *not* seem to work
>>> for OL7 kernels.  That is a problem.
>>>
>>
>> Thanks for the report! I've root-caused the absence of the function from
>> available_filter_functions in UEK7 and earlier (upstream works fine).
>> Prior to
>>
>> commit 79df45731da68772d2285265864a52c900b8c65f
>> Author: Song Liu <songliubraving at fb.com>
>> Date:   Wed Oct 6 14:07:32 2021 -0700
>>
>>     perf/core: Allow ftrace for functions in kernel/event/core.c
>>
>>     It is useful to trace functions in kernel/event/core.c. Allow ftrace for
>>     them by removing $(CC_FLAGS_FTRACE) from Makefile.
>>
>>     Signed-off-by: Song Liu <songliubraving at fb.com>
>>     Signed-off-by: Peter Zijlstra (Intel) <peterz at infradead.org>
>>     Link:
>> https://lkml.kernel.org/r/20211006210732.2826289-1-songliubraving@fb.com
>>
>>
>> ...kernel/events/Makefile removed the ftrace compile flags which mark
>> function entry for the code in kernel/events. In UEK7 we see
>>
>> ifdef CONFIG_FUNCTION_TRACER
>> CFLAGS_REMOVE_core.o = $(CC_FLAGS_FTRACE)
>> endif
>>
>> I'll try to find a workaround that doesn't rely on function boundary
>> tracing in these files to work..
>>
>> Alan
>>
>>> On Fri, Aug 16, 2024 at 03:33:00PM -0400, Kris Van Hees wrote:
>>>> Running a full testsuite run (and then also individual test) I found this
>>>> test to fail on my OL9 VM with 5.15.0-205.149.5.1.el9uek.x86_64 kernel
>>>> while it works on my Debian VM with a 6.5.0 kernel.
>>>>
>>>> On Fri, Aug 02, 2024 at 05:37:11PM -0400, Kris Van Hees wrote:
>>>>> On Fri, Jun 28, 2024 at 06:16:34PM +0100, Alan Maguire wrote:
>>>>>> ...since tst.oncpu.d test passes now.
>>>>>>
>>>>>> Signed-off-by: Alan Maguire <alan.maguire at oracle.com>
>>>>>
>>>>> Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>
>>>>>
>>>>>> ---
>>>>>>  test/unittest/sched/tst.oncpu.d | 3 +--
>>>>>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/test/unittest/sched/tst.oncpu.d b/test/unittest/sched/tst.oncpu.d
>>>>>> index d2c2ce57..7a33bab4 100644
>>>>>> --- a/test/unittest/sched/tst.oncpu.d
>>>>>> +++ b/test/unittest/sched/tst.oncpu.d
>>>>>> @@ -1,11 +1,10 @@
>>>>>>  /*
>>>>>>   * Oracle Linux DTrace.
>>>>>> - * Copyright (c) 2006, 2020, Oracle and/or its affiliates. All rights reserved.
>>>>>> + * Copyright (c) 2006, 2024, Oracle and/or its affiliates. All rights reserved.
>>>>>>   * Licensed under the Universal Permissive License v 1.0 as shown at
>>>>>>   * http://oss.oracle.com/licenses/upl.
>>>>>>   */
>>>>>>  
>>>>>> -/* @@xfail: dtv2 */
>>>>>>  /* @@timeout: 15 */
>>>>>>  
>>>>>>  #pragma D option switchrate=100hz
>>>>>> -- 
>>>>>> 2.43.5
>>>>>>