[DTrace-devel] [oracle/dtrace-utils] 351d46: Ensure proper probe compilation context is set for...

Thu Sep 9 04:05:34 PDT 2021

  Branch: refs/heads/nix/speculations
  Home:   https://github.com/oracle/dtrace-utils
  Commit: 351d461b77a760cb5cf4710c899e1809f8f1e648
      https://github.com/oracle/dtrace-utils/commit/351d461b77a760cb5cf4710c899e1809f8f1e648
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-07-06 (Tue, 06 Jul 2021)

  Changed paths:
    M libdtrace/dt_cc.c
    M libdtrace/dt_impl.h
    M libdtrace/dt_pid.c

  Log Message:
  -----------
  Ensure proper probe compilation context is set for trampolines

Trampolines are generated per-probe, well after the clauses have been
compiled.  The code generation for the trampolines was not setting the
proper probe context which could lead to obscure issues like not having
the proper probe argument information when the trampoline is generated.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Reviewed-by: Eugene Loh <eugene.loh at oracle.com>

  Commit: 2cea90a97f4e61c76a532638c9a254f07d7ebd6b
      https://github.com/oracle/dtrace-utils/commit/2cea90a97f4e61c76a532638c9a254f07d7ebd6b
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-07-06 (Tue, 06 Jul 2021)

  Changed paths:
    M libdtrace/dt_cg.c
    M libdtrace/dt_cg.h
    M libdtrace/dt_dctx.h
    M libdtrace/dt_prov_dtrace.c
    M libdtrace/dt_prov_fbt.c
    M libdtrace/dt_prov_pid.c
    M libdtrace/dt_prov_profile.c
    M libdtrace/dt_prov_sdt.c
    M libdtrace/dt_prov_syscall.c

  Log Message:
  -----------
  Provide and use standard functions to clear or copy CPU registers

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Reviewed-by: Eugene Loh <eugene.loh at oracle.com>

  Commit: 4eea2e3dcb6a0d9f5ee18751eea11b699ece8a5e
      https://github.com/oracle/dtrace-utils/commit/4eea2e3dcb6a0d9f5ee18751eea11b699ece8a5e
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-07-13 (Tue, 13 Jul 2021)

  Changed paths:
    M libdtrace/dt_prov_sdt.c
    R test/unittest/sdt/tst.arg-arity.d
    R test/unittest/sdt/tst.arg-arity.r
    R test/unittest/sdt/tst.args-sdt.d
    R test/unittest/sdt/tst.args-sdt.r
    A test/unittest/sdt/tst.args.d
    A test/unittest/sdt/tst.args.r

  Log Message:
  -----------
  Implement argument retrieval for SDT probes

The arguments for tracepoints are presented in the BPF context as a
sequence of 64-bit values.  They are preceded by a double word (64-bit
value) that is not accessible from BPF programs.

The tst.args.d test case verifies that we can retrieve argument data
for a tracepoint.  It is quite complex because we end up having to
decode the argument data because of the component datatypes.  This
adds to the value of the test because it verifies that the argument
data is presented in the correct way.

Both tst.arg-arity.d and tst.args-sdt.d are removed because they are
specific to the previous implementation of DTrace on Linux, and depend
on a deprecated kernel-level probe that existed for the sole purpose of
being able to use these tetss.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Reviewed-by: Eugene Loh <eugene.loh at oracle.com>

  Commit: 93d926a54e497d9148f6c6068049cf593794d117
      https://github.com/oracle/dtrace-utils/commit/93d926a54e497d9148f6c6068049cf593794d117
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-08-20 (Fri, 20 Aug 2021)

  Changed paths:
    M include/dtrace/options_defines.h
    M libdtrace/dt_bpf.c
    M libdtrace/dt_options.c
    A test/unittest/misc/tst.bpflog-cmdline.sh
    A test/unittest/misc/tst.bpflog-pragma.sh

  Log Message:
  -----------
  Introduce 'bpflog' runtime option to request BPF verifier log

The default behaviour is not to print the BPF verifier log unless a BPF
program fails to load.  It is sometimes desirable to obtain the BPF
verifier log for successful prgoram loads to assist in debugging other
issues with BPF programs.  Specifying -xbpflog on the command line or
setting this option using a pragma will force printing of the BPF
verifier log for all program loads.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Reviewed-by: Eugene Loh <eugene.loh at oracle.com>

  Commit: d1401ac23b485fb7d60ae4bb145603a70f61c9a8
      https://github.com/oracle/dtrace-utils/commit/d1401ac23b485fb7d60ae4bb145603a70f61c9a8
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-08-20 (Fri, 20 Aug 2021)

  Changed paths:
    A include/bpf-lib.h

  Log Message:
  -----------
  Provide set_upper_bound() and set_lower_bound() macro for C-to-BPF

The BPF verifier obtains (and needs) information about upper and lower
bounds for values based on conditional clauses.  The BPF GCC cross
compiler generates conditionals in a variety of forms.  The BPF verifier
sypports only a limited set of conditionals, so we need to be able to
enforce a specific form.  More specifically, the BPF verifier requires
the bound (a constant value) to be stored in the src register while the
value to be validated is in the dst register.

This patch provides macros that expand to inline assembler instructions
that enforce the expected register convention.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Reviewed-by: Eugene Loh <eugene.loh at oracle.com>

  Commit: dd59e2f9b01aaf0d6ee99c4bad122aad4cb0e0d3
      https://github.com/oracle/dtrace-utils/commit/dd59e2f9b01aaf0d6ee99c4bad122aad4cb0e0d3
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-08-20 (Fri, 20 Aug 2021)

  Changed paths:
    M libdtrace/dt_bpf.h
    M libdtrace/dt_cc.c
    M libdtrace/dt_dlibs.c

  Log Message:
  -----------
  Provide the maximum string length as BPF symbol STRSZ

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Reviewed-by: Eugene Loh <eugene.loh at oracle.com>

  Commit: 6dbeee8640a89907a93c8ad6349e304228244ce5
      https://github.com/oracle/dtrace-utils/commit/6dbeee8640a89907a93c8ad6349e304228244ce5
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-08-20 (Fri, 20 Aug 2021)

  Changed paths:
    M bpf/Build
    R bpf/memcpy.c
    M libdtrace/dt_cg.c
    M libdtrace/dt_dlibs.c

  Log Message:
  -----------
  Replace dt_memcpy() with bpf_probe_read()

The dt_memcpy() functionality can be provided by the bpf_probe_read()
helper.  Thanks to Eugene Loh for pointing this out.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Reviewed-by: Eugene Loh <eugene.loh at oracle.com>

  Commit: f4b2863ac3ffa15946fabccec4487aca9eff7665
      https://github.com/oracle/dtrace-utils/commit/f4b2863ac3ffa15946fabccec4487aca9eff7665
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-08-24 (Tue, 24 Aug 2021)

  Changed paths:
    M libdtrace/dt_parser.c

  Log Message:
  -----------
  Fix type alignment handling for enums

The type alignment handling code used to determine the alignment size
for a given datatype was treating enums as if they were integers, but
that is incorrect.  It should (just like integers and pointers) have
its alignment set to match its size, but it should avoid the explicit
void-checking that is done for integers.

Without this patch, libdtrace linked with newer versions of libctf
will see a test failure in unittest/sizeof/tst.SizeofDataTypes.d for
datatype 'new_enum'.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 7f80a0c0b14dba1c6e00b3bc77f48f7396059aeb
      https://github.com/oracle/dtrace-utils/commit/7f80a0c0b14dba1c6e00b3bc77f48f7396059aeb
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-08-26 (Thu, 26 Aug 2021)

  Changed paths:
    M libdtrace/dt_consume.c
    M test/unittest/error/tst.clause_scope-begin-ended.d

  Log Message:
  -----------
  Fix ERROR-in-BEGIN probe handling

The handling of ERROR probe invocations during the BEGIN probe execution
was flawed.  It failed to continue processing BEGIN clauses once an
ERROR invocation took place, and it didn't provide (as required) for
processing the CPU probe data buffer twice.

The continuation of processing has been corrected by ensuring that the
correct DTrace workstatus is reported from the probe callback functions.
The double buffer processing is now supported by passing a flag to
dt_consume_cpu() to indicate whether the tail of the ring buffer ought
to be updated or not.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Reviewed-by: Eugene Loh <eugene.loh at oracle.com>

  Commit: f7b32aeb5b8daf39dff576b7a89630d12df0ab3c
      https://github.com/oracle/dtrace-utils/commit/f7b32aeb5b8daf39dff576b7a89630d12df0ab3c
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-08-26 (Thu, 26 Aug 2021)

  Changed paths:
    M Makefunctions

  Log Message:
  -----------
  Add support for building .S files into .o objects

With the introduction of assembler source files (.S) for BPF code, to be
compiled into object files (.o), the obj-name macro in Makefunctions
needs to be extended to account for .S source files.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Reviewed-by: Eugene Loh <eugene.loh at oracle.com>

  Commit: 1e19a7afe7b2434714e9c8fe56d7a597ba997c2c
      https://github.com/oracle/dtrace-utils/commit/1e19a7afe7b2434714e9c8fe56d7a597ba997c2c
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-08-27 (Fri, 27 Aug 2021)

  Changed paths:
    M libdtrace/dt_bpf.c
    M libdtrace/dt_cg.c
    M libdtrace/dt_dctx.h
    M test/unittest/codegen/tst.stack_layout.r

  Log Message:
  -----------
  Introduce dctx->mem as a pointer to the generic scratch memory

Since there are multiple language features that need scratch memory,
it is more convenient for code generation to be able to refer to the
scratch memory by means of a base pointer.  The generic scratch
memory is allocated as part of the 'mem' BPF map.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 55902ec4088ddba492f92317741a4c785575b1f8
      https://github.com/oracle/dtrace-utils/commit/55902ec4088ddba492f92317741a4c785575b1f8
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-08-27 (Fri, 27 Aug 2021)

  Changed paths:
    M bpf/get_bvar.c
    M libdtrace/dt_bpf.h
    M libdtrace/dt_cc.c
    M libdtrace/dt_dlibs.c
    M test/unittest/stack/tst.stack_fbt.x

  Log Message:
  -----------
  Make stack trace use dctx->mem to store stack trace data

The stack trace implementation was using an offset into the dctx->buf
memory space to store the stack trace data.  Now that we have a generic
scratch memory area that can be referenced as dctx->mem, the stack
trace implementation has been adjusted to make use of that.

This patch also contains a tiny fix to the tst.stack_fbt.x script to
use /bin/bash because it depends on bash specific features.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: e2f1d1bfcab1771a4851e9a124339e50547531b0
      https://github.com/oracle/dtrace-utils/commit/e2f1d1bfcab1771a4851e9a124339e50547531b0
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-09-01 (Wed, 01 Sep 2021)

  Changed paths:
    M libdtrace/dt_consume.c

  Log Message:
  -----------
  Refactor dt_consume_one()

Split the bulk of dt_consume_one() into a new dt_consume_one_buffer(),
which can get buffers from places other than the perf ring buffer.
Much code got reindented.

Signed-off-by: Nick Alcock <nick.alcock at oracle.com>
Signed-off-by: Eugene Loh <eugene.loh at oracle.com>

  Commit: f9ba81e1e84c48a27309e90cfda215b5d400b68c
      https://github.com/oracle/dtrace-utils/commit/f9ba81e1e84c48a27309e90cfda215b5d400b68c
  Author: Nick Alcock <nick.alcock at oracle.com>
  Date:   2021-09-06 (Mon, 06 Sep 2021)

  Changed paths:
    M libdtrace/dt_htab.c

  Log Message:
  -----------
  htab: remove "based on a string" comment

The speculation code (upcoming) will have htabs of non-strings.

Signed-off-by: Nick Alcock <nick.alcock at oracle.com>

  Commit: 4a69590eafe30bab0e4d6a88ccb7d65c283762cd
      https://github.com/oracle/dtrace-utils/commit/4a69590eafe30bab0e4d6a88ccb7d65c283762cd
  Author: Nick Alcock <nick.alcock at oracle.com>
  Date:   2021-09-06 (Mon, 06 Sep 2021)

  Changed paths:
    M libdtrace/dt_open.c

  Log Message:
  -----------
  fix a tiny comment niggle

Signed-off-by: Nick Alcock <nick.alcock at oracle.com>

  Commit: a59ef607800c863a19981d26728049bbe3728090
      https://github.com/oracle/dtrace-utils/commit/a59ef607800c863a19981d26728049bbe3728090
  Author: Nick Alcock <nick.alcock at oracle.com>
  Date:   2021-09-06 (Mon, 06 Sep 2021)

  Changed paths:
    M bpf/Build
    A bpf/speculation.c
    A bpf/speculation_set_drainable.c
    A bpf/speculation_speculate.c
    M libdtrace/dt_bpf.c
    M libdtrace/dt_bpf.h
    M libdtrace/dt_bpf_maps.h
    M libdtrace/dt_cc.c
    M libdtrace/dt_cg.c
    M libdtrace/dt_consume.c
    M libdtrace/dt_dlibs.c
    M libdtrace/dt_errtags.h
    M libdtrace/dt_impl.h
    M libdtrace/dt_open.c
    M libdtrace/dt_peb.h
    M libdtrace/dtrace.h
    A test/unittest/speculation/err.CommitWithInvalid.d
    A test/unittest/speculation/err.CommitWithInvalid.r
    M test/unittest/speculation/err.D_ACT_SPEC.SpeculateWithCopyOut.d
    M test/unittest/speculation/err.D_ACT_SPEC.SpeculateWithCopyOutStr.d
    R test/unittest/speculation/err.D_COMM_COMM.CommitAftCommit.d
    R test/unittest/speculation/err.D_COMM_COMM.CommitAftCommit.r
    R test/unittest/speculation/err.D_COMM_COMM.DisjointCommit.d
    R test/unittest/speculation/err.D_COMM_COMM.DisjointCommit.r
    A test/unittest/speculation/err.DiscardWithInvalid.d
    A test/unittest/speculation/err.DiscardWithInvalid.r
    R test/unittest/speculation/err.SpecSizeVariations1.d
    R test/unittest/speculation/err.SpecSizeVariations1.r
    R test/unittest/speculation/err.SpecSizeVariations2.d
    R test/unittest/speculation/err.SpecSizeVariations2.r
    M test/unittest/speculation/tst.CommitAfterDiscard.d
    M test/unittest/speculation/tst.CommitAfterDiscard.r
    A test/unittest/speculation/tst.CommitCommitCommit.d
    A test/unittest/speculation/tst.CommitCommitCommit.r
    A test/unittest/speculation/tst.CommitDiscard4x.d
    A test/unittest/speculation/tst.CommitDiscard4x.r
    A test/unittest/speculation/tst.CommitWithInactive.d
    A test/unittest/speculation/tst.CommitWithInactive.r
    M test/unittest/speculation/tst.CommitWithZero.d
    M test/unittest/speculation/tst.CommitWithZero.r
    M test/unittest/speculation/tst.DataRecAftDiscard.d
    M test/unittest/speculation/tst.DiscardAftCommit.d
    M test/unittest/speculation/tst.DiscardAftCommit.r
    M test/unittest/speculation/tst.DiscardAftDataRec.d
    M test/unittest/speculation/tst.DiscardAftDiscard.d
    A test/unittest/speculation/tst.DiscardWithInactive.d
    A test/unittest/speculation/tst.DiscardWithInactive.r
    M test/unittest/speculation/tst.DiscardWithZero.d
    M test/unittest/speculation/tst.DiscardWithZero.r
    M test/unittest/speculation/tst.ExitAftDiscard.d
    M test/unittest/speculation/tst.NoSpecBuffer.d
    M test/unittest/speculation/tst.NoSpecBuffer.r
    A test/unittest/speculation/tst.SingleCPU.d
    A test/unittest/speculation/tst.SingleCPU.r
    R test/unittest/speculation/tst.SpecSizeVariations3.d
    R test/unittest/speculation/tst.SpecSizeVariations3.r
    M test/unittest/speculation/tst.SpecSizeVariations4.d
    M test/unittest/speculation/tst.SpecSizeVariations4.r
    M test/unittest/speculation/tst.SpecSizeVariations5.d
    M test/unittest/speculation/tst.SpecSizeVariations5.r
    M test/unittest/speculation/tst.SpeculationCommit.d
    A test/unittest/speculation/tst.SpeculationCommitNotQuiet.d
    A test/unittest/speculation/tst.SpeculationCommitNotQuiet.r
    A test/unittest/speculation/tst.SpeculationDefault.d
    A test/unittest/speculation/tst.SpeculationDefault.r
    A test/unittest/speculation/tst.SpeculationDefaultCommit.d
    A test/unittest/speculation/tst.SpeculationDefaultCommit.r
    A test/unittest/speculation/tst.SpeculationDefaultDiscard.d
    A test/unittest/speculation/tst.SpeculationDefaultDiscard.r
    M test/unittest/speculation/tst.SpeculationDiscard.d
    M test/unittest/speculation/tst.SpeculationDiscard.r
    A test/unittest/speculation/tst.SpeculationDiscardNotQuiet.d
    A test/unittest/speculation/tst.SpeculationDiscardNotQuiet.r
    M test/unittest/speculation/tst.SpeculationID.d
    M test/unittest/speculation/tst.SpeculationWithZero.d
    M test/unittest/speculation/tst.SpeculationWithZero.r
    M test/unittest/speculation/tst.TwoSpecBuffers.d
    M test/unittest/speculation/tst.TwoSpecBuffers.r
    M test/unittest/speculation/tst.negcommit.d
    M test/unittest/speculation/tst.negcommit.r
    M test/unittest/speculation/tst.zerosize.d

  Log Message:
  -----------
  consume, cg: implement speculations

This works by tracking live speculations in a specs map containing
dt_bpf_specs_t, each of which tracks the number of buffers written and
whether a commit/discard has been called on it.  (bpf/speculation.c,
which finds free speculations, is limited to 16 speculations and needs
adjusting when BPF loops work).  Successful speculate()s record the ID
of the active speculation in the clause header.  When a commit happens,
a COMMIT/DISCARD record is written.

At consume time, buffers with active speculate()s are hived off into a
list of dt_spec_bufs_t under a dt_spec_bufs_head_t (when peeking, this
is only done on the first peek, identified via the CONSUME_PEEK_START
peekflag); when a commit/discard hits for a given speculation, the efunc
invocation etc is suppressed if need be (discards can be followed by
data-recording actions, in which case the efunc is not suppressed), and
the specs map entry for this speculation is sucked into the
dt_spec_bufs_head and the head chained into the dt_spec_bufs_draining
list.  This list is traversed on every consume, and any new spec bufs
committed (as if just received from the ring buffer) or discarded
appropriately.

Signed-off-by: Nick Alcock <nick.alcock at oracle.com>

Compare: https://github.com/oracle/dtrace-utils/compare/351d461b77a7%5E...a59ef607800c