[DTrace-devel] [PATCH v2 0/7] Add python bindings for libdtrace interfaces

[Alan Maguire]

Having python bindings for libdtrace interfaces to compile, run
and collect information from DTrace programs is valuable because
it could help integration into metric collection frameworks like
PCP.  This series adds python bindings for DTrace (patch 4)
and provides tests for them (patch 7).  The support is
similar to what cmd/dtrace.c can do; compile a program,
enable probes, run it and collect data.  Support is also
added to grab or create processes as is done by
dtrace -p, -c options.  Aggregation snapshot walk is also
supported, with aggregation values filled out as raw values,
ints, lists of function names for stack keys and dicts for
quantized aggregations.

For detailed description/usage see the README in patch 4 and the
tests.

In order to support an environment where we have multiple handles
per process, some prep work is required.  Patch 1 fixes some issues
with multiple handle use, while patch 2 improves sharing for
vmlinux BTF (and CTF generated from it if needed) which multiple
handles in a process image will benefit from.  The upper bound
of per-process handles is now limited by the maximum number of
BPF programs attachable to a uprobe which is 64 (BPF_TRACE_MAX_PROGS)
which is encountered with attaches to the BEGIN probe.

Patch 3 makes available some of the functions used in calculating
stddev() values which is useful for the python bindings since they
do not use the print callbacks to do this.

Patch 4 is the cpython bindings themselves; the README.md describes
their usage.  Patch 5 packages them and patches 6 and 7 facilitate
testing them.

Changes since v1:

- Fixed multi-handle issues (patches 1/2)
- Improved aggregation representation of stacks, added stddev and
  *quantize representations (patch 4)
- Expanded tests to cover multi-handle, aggregation value validation
  (patch 7)

Alan Maguire (7):
  libdtrace: Support multiple DTrace handles per process
  libdtrace: share vmlinux BTF/CTF globally to support faster startup
  libdtrace: Refactor math functions into dt_math.h
  python: Add cpython bindings for libdtrace
  dtrace.spec: add python bindings packaging
  runtest.sh: Export PYTHONPATH when running tests in-tree
  test: add tests for python bindings

 GNUmakefile                               |    2 +
 bindings/Build                            |   35 +
 bindings/python/README.md                 |  167 ++
 bindings/python/pyproject.toml            |    3 +
 bindings/python/setup.py                  |   52 +
 bindings/python/src/pydtrace_module.c     | 1887 +++++++++++++++++++++
 configure                                 |    4 +-
 dtrace.spec                               |   29 +-
 libdtrace/dt_aggregate.c                  |    1 +
 libdtrace/dt_bpf.c                        |    6 +-
 libdtrace/dt_btf.c                        |   57 +-
 libdtrace/dt_btf.h                        |    1 +
 libdtrace/dt_consume.c                    |  340 +---
 libdtrace/dt_impl.h                       |    3 +-
 libdtrace/dt_math.h                       |  358 ++++
 libdtrace/dt_open.c                       |    5 +-
 libdtrace/dt_printf.c                     |    1 +
 libdtrace/dt_prov_dtrace.c                |    6 +-
 runtest.sh                                |    2 +
 test/unittest/python/tst.aggr-actions.sh  |  133 ++
 test/unittest/python/tst.aggr-change.sh   |   75 +
 test/unittest/python/tst.aggr-stack.sh    |   61 +
 test/unittest/python/tst.aggr.sh          |   58 +
 test/unittest/python/tst.exit.sh          |   64 +
 test/unittest/python/tst.multi-session.sh |   58 +
 test/unittest/python/tst.proc-create.sh   |   56 +
 test/unittest/python/tst.proc-exit.sh     |   57 +
 test/unittest/python/tst.proc-grab.sh     |   81 +
 28 files changed, 3246 insertions(+), 356 deletions(-)
 create mode 100644 bindings/Build
 create mode 100644 bindings/python/README.md
 create mode 100644 bindings/python/pyproject.toml
 create mode 100644 bindings/python/setup.py
 create mode 100644 bindings/python/src/pydtrace_module.c
 create mode 100644 libdtrace/dt_math.h
 create mode 100755 test/unittest/python/tst.aggr-actions.sh
 create mode 100755 test/unittest/python/tst.aggr-change.sh
 create mode 100755 test/unittest/python/tst.aggr-stack.sh
 create mode 100755 test/unittest/python/tst.aggr.sh
 create mode 100755 test/unittest/python/tst.exit.sh
 create mode 100755 test/unittest/python/tst.multi-session.sh
 create mode 100755 test/unittest/python/tst.proc-create.sh
 create mode 100755 test/unittest/python/tst.proc-exit.sh
 create mode 100755 test/unittest/python/tst.proc-grab.sh

-- 
2.43.5