[DTrace-devel] [oracle/dtrace-utils] 174398: Improve dt_pid_create_usdt_probes() error handling

Tue Jan 7 04:08:17 UTC 2025

  Branch: refs/heads/devel
  Home:   https://github.com/oracle/dtrace-utils
  Commit: 1743984a98285e989251e97b23fa7c38f8e00eb2
      https://github.com/oracle/dtrace-utils/commit/1743984a98285e989251e97b23fa7c38f8e00eb2
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2025-01-06 (Mon, 06 Jan 2025)

  Changed paths:
    M libdtrace/dt_pid.c

  Log Message:
  -----------
  Improve dt_pid_create_usdt_probes() error handling

In one run of the test suite, roughly 90 consecutive tests failed.
The exact circumstances are difficult to reconstruct, but basically
they complained with messages like:

    dtrace: failed to compile script ...: failed to grab process 23132

The same pid was always indicated.  Then, the errors stopped.

Apparently, the problem was trying "dtrace -s foo.d" where the script
has a BEGIN (or END) probe.  Then we get to

    cmd/dtrace.c compile_file
    ->  libdtrace/dt_cc.c dt_program_compile
    ->  libdtrace/dt_cc.c dt_compile
    ->  libdtrace/dt_cc.c dt_compile_one_clause
    ->  libdtrace/dt_cc.c dt_setcontext
    ->  libdtrace/dt_pid.c dt_pid_create_usdt_probes

We look for processes that might have USDT probes.  Since the provider
description is blank, we check every process in .../run/dtrace.  It
turned out, that pid 23132 could not be locked.  This sent an error
back up the call stack.

We do not know why process 23132 could not be locked, but having dtrace
fail under these circumstances (using BEGIN or END probe) seems severe.

Change dt_pid_create_usdt_probes() error handling.  Even if there is a
problem with some USDT processes, report success anyhow if there was a
glob pid specification.  If, on the other hand, a pid was specifically
requested, then a problem with that pid results in an error.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 297c990bfc13e7033010f052dc6a9c37b76abc10
      https://github.com/oracle/dtrace-utils/commit/297c990bfc13e7033010f052dc6a9c37b76abc10
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2025-01-06 (Mon, 06 Jan 2025)

  Changed paths:
    M test/triggers/doio.sh
    M test/unittest/io/tst.local.sh
    M test/unittest/io/tst.wait.sh

  Log Message:
  -----------
  test: Retry umount if necessary

The io tests tst.local.sh and tst.wait.sh fail with some frequency,
with error messages like "umount: $iodir: target is busy."

Modify the tests' dtrace trigger, doio.sh, so that it will retry
umount a few times, if necessary.  Specifically, remove the "set -e"
and individually check for and report errors, in the case of umount
retrying a few times before giving up.

While we're at it, notice that the two tests use the same $iodir.
There is no need for the tests to be coupled in this way.  Change
those two tests so that each test has a fresh value of iodir.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 220c0bc201ab097ca3b972551a5e728e218391ca
      https://github.com/oracle/dtrace-utils/commit/220c0bc201ab097ca3b972551a5e728e218391ca
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2025-01-06 (Mon, 06 Jan 2025)

  Changed paths:
    M test/unittest/usdt/err.Z_no-w.sh
    M test/unittest/usdt/tst.defer-Z.sh
    M test/unittest/usdt/tst.defer.sh
    M test/unittest/usdt/tst.nusdtprobes.sh

  Log Message:
  -----------
  test: Have USDT "deferred" tests wait for dtrace to start

Systemwide USDT tracing includes having dtrace track USDT processes
that start after it does.  Associated tests start dtrace first, wait
a few seconds, and then start processes to be traced.

Waiting "a few seconds" can be insufficient, especially on slower
systems or as more work is being performed during dtrace startup.

Change tests to wait for dtrace to start by polling on the output file.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 7383d6dd3e2a31973b4f04d30e0cbd43174366d9
      https://github.com/oracle/dtrace-utils/commit/7383d6dd3e2a31973b4f04d30e0cbd43174366d9
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2025-01-06 (Mon, 06 Jan 2025)

  Changed paths:
    M test/unittest/dtrace-util/tst.ListProbesArgsUSDT.sh
    M test/unittest/dtrace-util/tst.ListProbesFunc.sh
    M test/unittest/dtrace-util/tst.ListProbesFuncUSDT.sh
    M test/unittest/dtrace-util/tst.ListProbesModuleUSDT.sh
    M test/unittest/dtrace-util/tst.ListProbesName.sh
    M test/unittest/dtrace-util/tst.ListProbesNameUSDT.sh
    M test/unittest/dtrace-util/tst.ListProbesProviderUSDT.sh

  Log Message:
  -----------
  test: Make tests more resilient to changes in start-up performance

The tests with no target might start dtrace many times.  The test
time-out values should be long enough to withstand many starts
even on slow test systems.  Use time-out values for dtrace
start-up times of as long as 10 seconds.

The tests with a target might start the target first and then
expect more than one dtrace to start up and find that target.
Make sure the target runs sufficiently long.  Do not worry about
making the target run too long, since the test will kill the
target anyhow once it is no longer needed.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 228f11db8c2f7a4b37ea58f64a815452fcf6f597
      https://github.com/oracle/dtrace-utils/commit/228f11db8c2f7a4b37ea58f64a815452fcf6f597
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2025-01-06 (Mon, 06 Jan 2025)

  Changed paths:
    M test/unittest/usdt/tst.multiple.sh

  Log Message:
  -----------
  test: Improve sync mechanism in tst.multiple.sh

In this test, a USDT process is launched.  Subsequently, dtrace is
started.  To give dtrace a chance to enable the USDT probes, the
USDT process waits a few seconds.  On some systems and when the
amount of work dtrace does upon startup is high, "a few seconds"
may not be enough, causing the test to miss expected probes.

Have the USDT process loop on an is-enabled probe;  this is a
more resilient sync mechanism between it and dtrace.

That said, dtrace does not enable all probes in one instant, and
it makes no promises in what order probes will be started.  So,
we still leave a (shorter) sleep() in the target process.

Arguably, the test could be simplified by launching the USDT
process with "dtrace -c".  In this patch, we choose instead to
retain the overall character of the existing test.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 352417d422d9e2f0fbe2eecf1440de2bb2b3d0d0
      https://github.com/oracle/dtrace-utils/commit/352417d422d9e2f0fbe2eecf1440de2bb2b3d0d0
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2025-01-06 (Mon, 06 Jan 2025)

  Changed paths:
    M test/triggers/usdt-tst-defer.c

  Log Message:
  -----------
  test: Lengthen the delay in USDT defer trigger.

The trigger usdt-tst-defer.c expects that, as soon as probe foo is
enabled, so too are the foo and bar is-enabled probes as well as
the bar probe.  Since they are not turned on absolutely concurrently,
the trigger actually has a microsleep after foo is enabled and
before further activity is resumed.  Nonetheless, there is evidence
in occasional test runs that some events are missed.

Since there are no guarantees how concurrently these probes are
turned on, lengthen the delay.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: 1eb9b20470d8b6607db0ff553a4b787c95c24b13
      https://github.com/oracle/dtrace-utils/commit/1eb9b20470d8b6607db0ff553a4b787c95c24b13
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2025-01-06 (Mon, 06 Jan 2025)

  Changed paths:
    M libdtrace/dt_pid.c

  Log Message:
  -----------
  Use prb instead of psp.pps_prb

Replace psp.pps_prb with prb, both because it is simpler and
more importantly psp.pps_prb has not yet been assigned!

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

  Commit: f450921c04b175bbb7201246b0c4bfce6a318fb6
      https://github.com/oracle/dtrace-utils/commit/f450921c04b175bbb7201246b0c4bfce6a318fb6
  Author: Eugene Loh <eugene.loh at oracle.com>
  Date:   2025-01-06 (Mon, 06 Jan 2025)

  Changed paths:
    M test/unittest/usdt/tst.dlclose4.sh

  Log Message:
  -----------
  test: Disown child pid for cleaner stderr

Extend commit 9546afc22 ("test: Disown child pid for cleaner stderr")
to include another test.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>

Compare: https://github.com/oracle/dtrace-utils/compare/d68b4e26e7d5...f450921c04b1

To unsubscribe from these emails, change your notification settings at https://github.com/oracle/dtrace-utils/settings/notifications