[DTrace-devel] [oracle/dtrace-utils] 993995: Refactor dt_consume_one()

Fri Oct 15 09:06:30 PDT 2021

  Branch: refs/heads/dev
  Home:   https://github.com/oracle/dtrace-utils
  Commit: 9939955a5741c5a729fac809f6ea9a52d29291da
      https://github.com/oracle/dtrace-utils/commit/9939955a5741c5a729fac809f6ea9a52d29291da
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-10-12 (Tue, 12 Oct 2021)

  Changed paths:
    M libdtrace/dt_consume.c

  Log Message:
  -----------
  Refactor dt_consume_one()

Split the bulk of dt_consume_one() into a new dt_consume_one_probe(),
which can get probe data from places other than the perf ring buffer.
Much code got reindented.

Signed-off-by: Nick Alcock <nick.alcock at oracle.com>
Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: a981b73eda5e63bcd18529b72ab65463c3fc47a5
      https://github.com/oracle/dtrace-utils/commit/a981b73eda5e63bcd18529b72ab65463c3fc47a5
  Author: Nick Alcock <nick.alcock at oracle.com>
  Date:   2021-10-12 (Tue, 12 Oct 2021)

  Changed paths:
    M libdtrace/dt_htab.c

  Log Message:
  -----------
  htab: remove "based on a string" comment

The speculation code (upcoming) will have htabs of non-strings.

Signed-off-by: Nick Alcock <nick.alcock at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 7fe6ce9a8a887c3a78e49a8eddefe96ea9e5b493
      https://github.com/oracle/dtrace-utils/commit/7fe6ce9a8a887c3a78e49a8eddefe96ea9e5b493
  Author: Nick Alcock <nick.alcock at oracle.com>
  Date:   2021-10-13 (Wed, 13 Oct 2021)

  Changed paths:
    M bpf/Build
    A bpf/speculation.c
    M libdtrace/dt_bpf.c
    M libdtrace/dt_bpf.h
    M libdtrace/dt_bpf_maps.h
    M libdtrace/dt_cc.c
    M libdtrace/dt_cg.c
    M libdtrace/dt_consume.c
    M libdtrace/dt_dlibs.c
    M libdtrace/dt_errtags.h
    M libdtrace/dt_impl.h
    M libdtrace/dt_open.c
    M libdtrace/dt_peb.h
    M libdtrace/dtrace.h
    A test/unittest/speculation/err.CommitWithInvalid.d
    A test/unittest/speculation/err.CommitWithInvalid.r
    M test/unittest/speculation/err.D_ACT_SPEC.SpeculateWithCopyOut.d
    M test/unittest/speculation/err.D_ACT_SPEC.SpeculateWithCopyOutStr.d
    R test/unittest/speculation/err.D_COMM_COMM.CommitAftCommit.d
    R test/unittest/speculation/err.D_COMM_COMM.CommitAftCommit.r
    R test/unittest/speculation/err.D_COMM_COMM.DisjointCommit.d
    R test/unittest/speculation/err.D_COMM_COMM.DisjointCommit.r
    A test/unittest/speculation/err.DiscardWithInvalid.d
    A test/unittest/speculation/err.DiscardWithInvalid.r
    R test/unittest/speculation/err.SpecSizeVariations1.d
    R test/unittest/speculation/err.SpecSizeVariations1.r
    R test/unittest/speculation/err.SpecSizeVariations2.d
    R test/unittest/speculation/err.SpecSizeVariations2.r
    M test/unittest/speculation/tst.CommitAfterDiscard.d
    M test/unittest/speculation/tst.CommitAfterDiscard.r
    A test/unittest/speculation/tst.CommitCommitCommit.d
    A test/unittest/speculation/tst.CommitCommitCommit.r
    A test/unittest/speculation/tst.CommitDiscard4x.d
    A test/unittest/speculation/tst.CommitDiscard4x.r
    A test/unittest/speculation/tst.CommitWithInactive.d
    A test/unittest/speculation/tst.CommitWithInactive.r
    M test/unittest/speculation/tst.CommitWithZero.d
    M test/unittest/speculation/tst.CommitWithZero.r
    M test/unittest/speculation/tst.DataRecAftDiscard.d
    M test/unittest/speculation/tst.DiscardAftCommit.d
    M test/unittest/speculation/tst.DiscardAftCommit.r
    M test/unittest/speculation/tst.DiscardAftDataRec.d
    M test/unittest/speculation/tst.DiscardAftDiscard.d
    A test/unittest/speculation/tst.DiscardWithInactive.d
    A test/unittest/speculation/tst.DiscardWithInactive.r
    M test/unittest/speculation/tst.DiscardWithZero.d
    M test/unittest/speculation/tst.DiscardWithZero.r
    M test/unittest/speculation/tst.ExitAftDiscard.d
    M test/unittest/speculation/tst.NoSpecBuffer.d
    M test/unittest/speculation/tst.NoSpecBuffer.r
    A test/unittest/speculation/tst.SingleCPU.d
    A test/unittest/speculation/tst.SingleCPU.r
    R test/unittest/speculation/tst.SpecSizeVariations3.d
    R test/unittest/speculation/tst.SpecSizeVariations3.r
    M test/unittest/speculation/tst.SpecSizeVariations4.d
    M test/unittest/speculation/tst.SpecSizeVariations4.r
    M test/unittest/speculation/tst.SpecSizeVariations5.d
    M test/unittest/speculation/tst.SpecSizeVariations5.r
    M test/unittest/speculation/tst.SpeculationCommit.d
    A test/unittest/speculation/tst.SpeculationCommitNotQuiet.d
    A test/unittest/speculation/tst.SpeculationCommitNotQuiet.r
    A test/unittest/speculation/tst.SpeculationDefault.d
    A test/unittest/speculation/tst.SpeculationDefault.r
    A test/unittest/speculation/tst.SpeculationDefaultCommit.d
    A test/unittest/speculation/tst.SpeculationDefaultCommit.r
    A test/unittest/speculation/tst.SpeculationDefaultDiscard.d
    A test/unittest/speculation/tst.SpeculationDefaultDiscard.r
    M test/unittest/speculation/tst.SpeculationDiscard.d
    M test/unittest/speculation/tst.SpeculationDiscard.r
    A test/unittest/speculation/tst.SpeculationDiscardNotQuiet.d
    A test/unittest/speculation/tst.SpeculationDiscardNotQuiet.r
    M test/unittest/speculation/tst.SpeculationID.d
    M test/unittest/speculation/tst.SpeculationWithZero.d
    M test/unittest/speculation/tst.SpeculationWithZero.r
    M test/unittest/speculation/tst.TwoSpecBuffers.d
    M test/unittest/speculation/tst.TwoSpecBuffers.r
    M test/unittest/speculation/tst.negcommit.d
    M test/unittest/speculation/tst.negcommit.r
    M test/unittest/speculation/tst.zerosize.d

  Log Message:
  -----------
  consume, cg: implement speculations

This works by tracking live speculations in a specs map containing
dt_bpf_specs_t, each of which tracks the number of buffers written and
whether a commit/discard has been called on it.  (dt_speculation(),
which finds free speculations, is limited to 16 speculations and needs
adjusting when BPF loops work).  Successful speculate()s record the ID
of the active speculation in the clause header.  When a commit happens,
a COMMIT/DISCARD record is written.

At consume time, buffers with active speculate()s are hived off into a
list of dt_spec_buf_data_t under a dt_spec_buf_t (when peeking, this is
only done on the first peek, identified via the CONSUME_PEEK_START
peekflag); when a commit/discard hits for a given speculation, the efunc
invocation etc is suppressed if need be (discards can be followed by
data-recording actions, in which case the efunc is not suppressed), and
the specs map entry for this speculation is sucked into the dt_spec_buf
and this chained into the dt_spec_bufs_draining list.  This list is
traversed on every consume, and any new spec bufs committed (as if just
received from the ring buffer) or discarded appropriately.

Signed-off-by: Nick Alcock <nick.alcock at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: ad372096de58f412d2d75e308001bddd4460fc85
      https://github.com/oracle/dtrace-utils/commit/ad372096de58f412d2d75e308001bddd4460fc85
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    M libdtrace/dt_bpf.c
    M libdtrace/dt_peb.c
    M libdtrace/dt_prov_profile.c
    M libdtrace/dt_provider_tp.c

  Log Message:
  -----------
  Ensure all perf_event_open() calls use the PERF_FLAG_FD_CLOEXEC flag

Also set the size of the attribute struct, and only use PERF_SAMPLE_RAW
for the output buffers.

Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 2542ec13c79acafe7221c56886790e92f37a1b5b
      https://github.com/oracle/dtrace-utils/commit/2542ec13c79acafe7221c56886790e92f37a1b5b
  Author: Kris Van Hees <kris.van.hees at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    A test/unittest/struct/tst.implicit_decl.d
    A test/unittest/struct/tst.implicit_decl.r

  Log Message:
  -----------
  test: Add test to Verify storage size for implicit struct declarations

Signed-off-by: Eugene Loh <eugene.loh at oracle.com
Signed-off-by: Kris Van Hees <kris.van.hees at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 9b337db309e35256164b86de0484c0229e0bff61
      https://github.com/oracle/dtrace-utils/commit/9b337db309e35256164b86de0484c0229e0bff61
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    M libdtrace/dt_cg.c

  Log Message:
  -----------
  Check not-NULL pointer before trace(string)

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 05f0212ca2a84e6f5db4f6d3890f72767f37d72d
      https://github.com/oracle/dtrace-utils/commit/05f0212ca2a84e6f5db4f6d3890f72767f37d72d
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    M bpf/Build
    A bpf/strcmp.S
    M libdtrace/dt_bpf.c
    M libdtrace/dt_cg.c
    M libdtrace/dt_dlibs.c
    M test/demo/builtin/probename.d
    M test/demo/builtin/probeprov.d
    M test/demo/fbt/xioctl.d
    M test/unittest/dif/strncmp.d
    M test/unittest/dtrace-util/tst.ListProbesFunc.sh
    M test/unittest/dtrace-util/tst.ListProbesName.sh
    A test/unittest/operators/tst.str_comparison-basic.d
    A test/unittest/operators/tst.str_comparison-longer.d
    A test/unittest/operators/tst.str_comparison-nested.d
    A test/unittest/operators/tst.str_comparison-nested.r
    A test/unittest/operators/tst.str_comparison-signed.d
    M test/unittest/types/tst.condexpr.d
    M test/unittest/types/tst.relstring.d

  Log Message:
  -----------
  Add support for string comparisons

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 8a9815ab47894f4d857fe62faca96f7e4a1b5c4b
      https://github.com/oracle/dtrace-utils/commit/8a9815ab47894f4d857fe62faca96f7e4a1b5c4b
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    M bpf/Build
    A bpf/strchr.S
    M libdtrace/dt_cg.c
    M libdtrace/dt_dlibs.c
    M test/unittest/dif/strchr.d

  Log Message:
  -----------
  Add support for strchr() subroutine

This implementation tries to minimize branching and looping in BPF code
to ease the load on the BPF verifier, which tries to walk each branch.
It does so by using bpf_probe_read() to copy the string into scratch
memory.  Then, the byte in question is xor'ed with every byte in scratch
memory.  This means the matching byte will now be a NULL byte while all
other bytes are non-null.  This means that bpf_probe_read_str() can now
give us the location of the byte in question.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 16c9de01cfe1e116ccb86407c3093e0aab666e41
      https://github.com/oracle/dtrace-utils/commit/16c9de01cfe1e116ccb86407c3093e0aab666e41
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    M bpf/Build
    A bpf/strrchr.S
    M libdtrace/dt_cg.c
    M libdtrace/dt_dlibs.c
    M test/unittest/funcs/tst.strchr.d

  Log Message:
  -----------
  Add support for strrchr() subroutine

The algorithm is different from the one used for strchr() due to
how the BPF verifier handles decrementing and incrementing loops
differently.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 5f8c4107a6a1b3afdc589714a30b5c47fda83b89
      https://github.com/oracle/dtrace-utils/commit/5f8c4107a6a1b3afdc589714a30b5c47fda83b89
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    M bpf/Build
    A bpf/index.S
    M libdtrace/dt_cg.c
    M libdtrace/dt_dlibs.c
    M test/unittest/dif/index2arg.d
    M test/unittest/dif/index3arg.d
    A test/unittest/funcs/tst.index2.d
    A test/unittest/funcs/tst.index2.r

  Log Message:
  -----------
  Add support for index() subroutine

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 67e980df60798b5f9e87cdfc6275fdad802d95ff
      https://github.com/oracle/dtrace-utils/commit/67e980df60798b5f9e87cdfc6275fdad802d95ff
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    M bpf/Build
    A bpf/rindex.S
    M libdtrace/dt_cg.c
    M libdtrace/dt_dlibs.c
    M test/stress/fbtsafety/tst.shortstr.d
    A test/unittest/funcs/tst.rindex.d
    A test/unittest/funcs/tst.rindex.r

  Log Message:
  -----------
  Add support for rindex() subroutine

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: e352545c94376abbf83838aa716dbc9c97f00111
      https://github.com/oracle/dtrace-utils/commit/e352545c94376abbf83838aa716dbc9c97f00111
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    M libdtrace/dt_cg.c
    M test/unittest/dif/strstr.d
    M test/unittest/funcs/tst.strstr.d

  Log Message:
  -----------
  Add support for strstr() subroutine

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: ccc2097066e72f4720b7bf98c9c02401274b9f1a
      https://github.com/oracle/dtrace-utils/commit/ccc2097066e72f4720b7bf98c9c02401274b9f1a
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    M bpf/Build
    A bpf/lltostr.S
    M libdtrace/dt_cg.c
    M libdtrace/dt_dlibs.c
    M test/unittest/dif/lltostr.d
    A test/unittest/funcs/tst.lltostr-short.d
    A test/unittest/funcs/tst.lltostr-short.r
    M test/unittest/funcs/tst.lltostr.d
    M test/unittest/funcs/tst.lltostr.r

  Log Message:
  -----------
  Add support for lltostr() subroutine

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 91182c2c3a5a8419c0551c39588096cbbad9d24f
      https://github.com/oracle/dtrace-utils/commit/91182c2c3a5a8419c0551c39588096cbbad9d24f
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2021-10-14 (Thu, 14 Oct 2021)

  Changed paths:
    M libdtrace/dt_module.c
    M test/unittest/aggs/tst.aggmod_full2.sh
    M test/unittest/consumer/tst.symbols.c

  Log Message:
  -----------
  Missing symbols for loadable modules when reading /proc/kallmodsyms

As DTrace-on-BPF development started, the patch
"dtrace: handle .init.scratch section in /proc/kallmodsyms" was applied,
but different versions were applied to legacy DTrace and DTrace on BPF:
81c6542 and f5c1f088, respectively.  The patch applied to DTrace on BPF was
wrong.  In particular, once dt_modsym_update() starts reading symbols for
loadable modules, it sets kernel_flag=-1.  If there is no .init.scratch
section, the code will forever think it is nonetheless in such a section
and ignore all symbols.  Hence, symbols for loadable modules are never read.

Fix dt_modsym_update(), specifically more in line with legacy DTrace.
Importantly,
  - handle .init.scratch first
  - make kernel_flag an unsigned int
  - signal the beginning of loadable modules by
      setting only one bit of kernel_flag, not all of them

Thanks to Rajan Shanmugavelu for reporting this problem.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Nick Alcock <nick.alcock at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

Compare: https://github.com/oracle/dtrace-utils/compare/9f27b8c7bb40...91182c2c3a5a