[DTrace-devel] [PATCH v5 00/11] Add python, PCP support via libdtrace interfaces
Alan Maguire
alan.maguire at oracle.com
Fri Aug 21 16:10:48 UTC 2026
Having python bindings for libdtrace interfaces to compile, run
and collect information from DTrace programs is valuable because
it could help integration into metric collection frameworks like
PCP. This series adds python bindings for DTrace (patch 5)
and provides tests for them (patch 8). The support is
similar to what cmd/dtrace.c can do; compile a program,
enable probes, run it and collect data. Support is also
added to grab or create processes as is done by
dtrace -p, -c options. Aggregation snapshot walk is also
supported, with aggregation values filled out as raw values,
ints, lists of function names for stack keys and dicts for
quantized aggregations.
For detailed description/usage see the README in patch 4 and the
tests.
Patch 9 then provides a consumer of python support - a Performance
Co-Pilot (PCP) Performance Metric Data Agent (PMDA) which enables
flexible metric collection via DTrace scripts. See the README.md
in patch 9 for more details.
In order to support an environment where we have multiple handles
per process, some prep work is required. Patch 1 fixes some issues
with multiple handle use, while patch 2 improves sharing for
vmlinux BTF (and CTF generated from it if needed) which multiple
handles in a process image will benefit from. The upper bound
of per-process handles is now limited by the maximum number of
BPF programs attachable to a uprobe which is 64 (BPF_TRACE_MAX_PROGS)
which is encountered with attaches to the BEGIN probe.
Patch 3 fixes an issue that surfaced during testing - we try to
load BTF for builtin modules when we have modules.builtin.ranges -
to avoid this we mark DTrace modules as builtin and have them
share vmlinux BTF.
Patch 4 provides libdtrace APIs to compute aggregation keys/values
rather than having the python bindings reach too deeply into the
implementation.
Patch 5 is the cpython bindings themselves; the README.md describes
their usage. Patch 6 packages them and patches 7 and 8 facilitate
testing them.
Patch 9 then builds the PCP PMDA using the python bindings while
patch 10 packages them in a pcp-pmda-dtrace package. Finally
patch 11 provides some basic testing in the absence of a full
PCP setup.
Changes since v4:
- Fixed issues in libdtrace with module.builtin.ranges (patch 3)
- Instead of sharing math header and doing aggr computations in
the python code, provide libdtrace APIs to compute keys/values
(patch 4)
- Fixed issues around python/pcp package dependencies (Kris,
patches 6, 10)
- Fixed issues around pmdadtrace deleted/re-added stats (Kris,
patch 5)
- Fixed dynamic PMDA number use in pmdadtrace.python (Kris,
patch 9)
- Fixed session close issues for multi-session test (Kris,
patch 8)
- Fixed support to allow multiple defines to be passed in
(patch 5)
- Added tests for compile time/setopt options including defines
and added an aggregation clear test to improve coverage
(patch 8)
- Fixed up pmdadtrace to handle compile-time options properly
(patch 9)
Changes since v3:
- Fixed up issues identified by Kris around custom BTF path (patch 2)
- Added PCP patches 8/9
Changes since v2:
- Fixed some handle lifetime issues and related potential segfaults
in error paths (patch 4)
- Fixed up packaging to derive python package version from DTrace
version (patch 5)
Changes since v1:
- Fixed multi-handle issues (patches 1/2)
- Improved aggregation representation of stacks, added stddev and
*quantize representations (patch 4)
Alan Maguire (11):
libdtrace: Support multiple DTrace handles per process
libdtrace: share vmlinux BTF/CTF globally to support faster startup
libdtrace: avoid BTF probes for ranged built-in modules
libdtrace: Add aggregation key/value computation APIs
python: Add cpython bindings for libdtrace
dtrace.spec: add python bindings packaging
runtest.sh: Prepare Python bindings for in-tree testing
test: add tests for python bindings
dtrace: Add a PCP PMDA to expose DTrace data as metrics
dtrace.spec: Add PCP PMDA packaging pcp-pmda-dtrace
test: Add a test for the PCP ManagedDTraceScript interface
GNUmakefile | 4 +
Makecheck | 3 +
bindings/Build | 39 +
bindings/python/README.md | 205 ++
bindings/python/pyproject.toml | 3 +
bindings/python/setup.py | 67 +
bindings/python/src/pydtrace_module.c | 2057 +++++++++++++++++
configure | 7 +-
dtrace.spec | 88 +-
libdtrace/dt_aggregate.c | 517 +++++
libdtrace/dt_bpf.c | 6 +-
libdtrace/dt_btf.c | 100 +-
libdtrace/dt_btf.h | 1 +
libdtrace/dt_impl.h | 6 +-
libdtrace/dt_module.c | 20 +-
libdtrace/dt_open.c | 5 +-
libdtrace/dt_prov_dtrace.c | 6 +-
libdtrace/dtrace.h | 83 +
libdtrace/libdtrace.ver | 3 +
pcp/Build | 66 +
pcp/Install | 37 +
pcp/README.md | 343 +++
pcp/Remove | 26 +
pcp/autostart/.gitkeep | 0
pcp/dtrace.conf | 12 +
pcp/examples/io_times.d | 28 +
pcp/examples/io_times.json | 8 +
pcp/examples/irq_times.d | 51 +
pcp/examples/irq_times.json | 8 +
pcp/examples/packet_drop_reasons.d | 16 +
pcp/examples/packet_drop_reasons.json | 8 +
pcp/examples/profile_kernel.d | 33 +
pcp/examples/profile_kernel.json | 11 +
pcp/examples/syscall_counts.d | 8 +
pcp/examples/syscall_counts.json | 8 +
pcp/pmdadtrace.1 | 135 ++
pcp/pmdadtrace.python | 1664 +++++++++++++
runtest.sh | 4 +
test/unittest/python/test.x | 17 +
test/unittest/python/tst.aggr-actions.sh | 152 ++
test/unittest/python/tst.aggr-change.sh | 74 +
test/unittest/python/tst.aggr-clear.sh | 44 +
test/unittest/python/tst.aggr-keys.sh | 89 +
test/unittest/python/tst.aggr-stack.sh | 62 +
test/unittest/python/tst.aggr.sh | 58 +
test/unittest/python/tst.compile-opts.sh | 50 +
test/unittest/python/tst.exit.sh | 67 +
test/unittest/python/tst.multi-session-end.sh | 89 +
test/unittest/python/tst.multi-session.sh | 62 +
test/unittest/python/tst.pcp.sh | 69 +
test/unittest/python/tst.proc-create.sh | 71 +
test/unittest/python/tst.proc-exit.sh | 57 +
test/unittest/python/tst.proc-grab.sh | 84 +
test/unittest/python/tst.setopt.sh | 30 +
test/unittest/python/tst.work-error.sh | 36 +
55 files changed, 6778 insertions(+), 19 deletions(-)
create mode 100644 bindings/Build
create mode 100644 bindings/python/README.md
create mode 100644 bindings/python/pyproject.toml
create mode 100644 bindings/python/setup.py
create mode 100644 bindings/python/src/pydtrace_module.c
create mode 100644 pcp/Build
create mode 100755 pcp/Install
create mode 100644 pcp/README.md
create mode 100755 pcp/Remove
create mode 100644 pcp/autostart/.gitkeep
create mode 100644 pcp/dtrace.conf
create mode 100644 pcp/examples/io_times.d
create mode 100644 pcp/examples/io_times.json
create mode 100644 pcp/examples/irq_times.d
create mode 100644 pcp/examples/irq_times.json
create mode 100644 pcp/examples/packet_drop_reasons.d
create mode 100644 pcp/examples/packet_drop_reasons.json
create mode 100644 pcp/examples/profile_kernel.d
create mode 100644 pcp/examples/profile_kernel.json
create mode 100644 pcp/examples/syscall_counts.d
create mode 100644 pcp/examples/syscall_counts.json
create mode 100644 pcp/pmdadtrace.1
create mode 100755 pcp/pmdadtrace.python
create mode 100755 test/unittest/python/test.x
create mode 100755 test/unittest/python/tst.aggr-actions.sh
create mode 100755 test/unittest/python/tst.aggr-change.sh
create mode 100755 test/unittest/python/tst.aggr-clear.sh
create mode 100755 test/unittest/python/tst.aggr-keys.sh
create mode 100755 test/unittest/python/tst.aggr-stack.sh
create mode 100755 test/unittest/python/tst.aggr.sh
create mode 100755 test/unittest/python/tst.compile-opts.sh
create mode 100755 test/unittest/python/tst.exit.sh
create mode 100755 test/unittest/python/tst.multi-session-end.sh
create mode 100755 test/unittest/python/tst.multi-session.sh
create mode 100755 test/unittest/python/tst.pcp.sh
create mode 100755 test/unittest/python/tst.proc-create.sh
create mode 100755 test/unittest/python/tst.proc-exit.sh
create mode 100755 test/unittest/python/tst.proc-grab.sh
create mode 100755 test/unittest/python/tst.setopt.sh
create mode 100755 test/unittest/python/tst.work-error.sh
--
2.43.5
More information about the DTrace-devel
mailing list