[DTrace-devel] [PATCH v3 06/21] dtprobed: add the DOF stash

[Kris Van Hees]

On Mon, Feb 12, 2024 at 10:00:37PM +0000, Nick Alcock wrote:
> On 12 Feb 2024, Kris Van Hees spake thusly:
> 
> > Initial comments (mostly on the comments, that are pretty crucial for people
> > to understand what is going on here).
> 
> ... here's the new top-of-file comment block, which hopefully make a bit
> more sense now:
> 
> /*
>  * Oracle Linux DTrace; DOF state storage management.
>  * Copyright (c) 2024, Oracle and/or its affiliates. All rights reserved.
>  * Licensed under the Universal Permissive License v 1.0 as shown at
>  * http://oss.oracle.com/licenses/upl.
>  *
>  * The DOF stash is the principal state preservation mechanism used by dtprobed.
>  * Keeping DOF on disk keeps memory management complexity low, makes it
>  * possible for dtprobed to track DOF across daemon restarts and upgrades,
>  * and gives the rest of dtrace a place to acquire the DOF contributed by
>  * processes on the system for USDT probe operations.
>  *
>  * The state storage structure is as follows:
>  *
>  * /run/dtrace/stash: Things private to dtprobed.

Perhaps use a trailing / for all entries here and below that are directories,
so it is immediately obvious what are files and what are directories?

>  *
>  *    .../dof/$dev-$ino: DOF contributed by particular mappings, in raw form
>  *    (as received from some probe-containing program).

I would think that this is a directory that contains:

	   ../raw				- raw DOF
	   ../parsed				- parsed DOF
	   ../parsed/$prv:$mod:$fun:$prb	- per-probe parsed DOF

because this data is not task (or PID) specific.  Granted, dtrace will use
a modified provider name ($prv$pid) when it creates probes for a specific task,
but at this level there is nothing task specific in this data.

>  *
>  *    .../dof-pid/$pid/$dev-$ino: contains everything relating to DOF
>  *    contributed by a particular USDT-containing ELF object within a given
>  *    process (henceforth "DOF source").  Each DTRACEHIOC_ADDDOF ioctl call
>  *    creates one of these.

I would make this a hardlink to .../dof/$dev-$ino.

Then I would add a file:

	.../dof-pid/$pid/$dev-$ino.dh

which contains the dof_helper_t data for that mapping in this particular task
(by pid).  Together with the data in .../dof-pid/$pid/$dev-$ino (which is
.../dof/$dev-$ino), you have enough to identify everything we need to know
about the USDT probes in a particular task.

>  *
>  *    .../dof-pid/$pid/$gen: symlink to a single $dev-$ino dir. "$gen" is an
>  *    incrementing generation counter starting at zero for every unique PID,
>  *    used to uniquely identify DOF pieces to remove.
>  *
>  *    .../dof-pid/$pid/next-gen: The next generation counter to use for a new
>  *    piece of DOF from this process.  Encoded as a sparse file whose size is
>  *    the next counter value to use.
>  *
>  *    .../dof-pid/$pid/$dev-$ino/raw: hardlink to the DOF for a given DOF
>  *    source.  Pruned of dead processes at startup and on occasion: entries also
>  *    deleted on receipt of DTRACEHIOC_REMOVE ioctls.  A hardlink is used in
>  *    order to bump the link count for the corresponding DOF in the dof/
>  *    directory: when this link count falls to 1, the DOF is considered dead and
>  *    the corresponding probe is removed.

Not needed - is found through .../dof-pid/$pid/$dev-ino.

>  *
>  *    .../dof-pid/$pid/$dev-$ino/dh: Raw form of the dof_helper_t received from
>  *    a given DTRACEHIOC_ADDDOF, serialized straight to disk with no changes.
>  *    Preserved for all DTRACEHIOC_ADDDOFs, even if the DOF was already
>  *    contributed by some other process.  Keeping this around allows the DOF to
>  *    be re-parsed as needed, even by a later re-execution of a different
>  *    version of the same daemon with a different dof_parsed_t structure, and to
>  *    be re-parsed with appropriate addresses for whichever mapping of this DOF
>  *    is under consideration.

See above (changed to be .../dof-pid/$pid/$dev-$ino.dh - see above).

>  *
>  *    .../dof-pid/$pid/$dev-$ino/parsed: Directory containing parsed DOF for a
>  *    given DOF source.  Deleted when empty.

Not needed - is found through .../dof-pid/$pid/$dev-ino.

>  *
>  *    .../dof-pid/$pid/$dev-$ino/parsed/$prv$pid:$mod:$fun:$prb: parsed DOF for
>  *    a given PID/mapping/probe triplet.  Generated from the helper for PIDs
>  *    that are the first appearance of a given piece of DOF.  Deleted at daemon
>  *    startup if the dof_parsed_t structure changed.  The file consists of a
>  *    uint64_t version number (DOF_PARSED_VERSION), a uint64_t count of
>  *    tracepoints, and then that many struct dof_parsed_t's of type
>  *    DIT_TRACEPOINT.

Not needed - is found through .../dof-pid/$pid/$dev-ino.

>  *
>  * /run/dtrace/probes: Per-probe info, written by dtprobed, read by DTrace.
>  *
>  *    .../$pid/$prv$pid/$mod/$fun/$prb: Hardlink from $prv$pid:$mod:$fun:$prb
>  *    above; parsed representation of one probe in a given process. Removed by
>  *    dtprobed when the process dies, or if all mappings containing the probe
>  *    are unmmapped.  Used by DTrace for tracing by PID.

Why not use .../$pid/$prv/$mod/$fun/$prb?  Since you are using globbing to
search for probe name matches, it doesn't matter where the pid is placed, and
there is no need to have it listed twice.  DTrace can still register the probe
internally as $prv$pid after its definition has been found.

>  *
>  *    (If a probe exists in multiple different mappings in a given process, they
>  *    must all be identical.)
>  *
>  * In future, symlinks in /run/dtrace/probes/$prv$pid may exist, used for
>  * globbing provider names across processes.
>  */