[DTrace-devel] [PATCH] doc: Add the DTrace Tutorial to the git repo and install package
Kris Van Hees
kris.van.hees at oracle.com
Thu Jan 15 23:24:27 UTC 2026
On Tue, Dec 16, 2025 at 01:11:23AM -0500, eugene.loh at oracle.com wrote:
> From: Eugene Loh <eugene.loh at oracle.com>
>
> Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
Reviewed-by: Kris Van Hees <kris.van.hees at oracle.com>
> ---
> Build | 4 +-
> doc/tutorial/1.IntroducingDTrace.md | 380 ++++
> .../2.TracingOperatingSystemBehavior.md | 1741 +++++++++++++++++
> .../3.TracingUserSpaceApplications.md | 832 ++++++++
> doc/tutorial/4.GoingFurtherWithDTrace.md | 14 +
> doc/tutorial/index.md | 15 +
> 6 files changed, 2985 insertions(+), 1 deletion(-)
> create mode 100644 doc/tutorial/1.IntroducingDTrace.md
> create mode 100644 doc/tutorial/2.TracingOperatingSystemBehavior.md
> create mode 100644 doc/tutorial/3.TracingUserSpaceApplications.md
> create mode 100644 doc/tutorial/4.GoingFurtherWithDTrace.md
> create mode 100644 doc/tutorial/index.md
>
> diff --git a/Build b/Build
> index 220c97845..434f8ad92 100644
> --- a/Build
> +++ b/Build
> @@ -3,14 +3,16 @@
>
> install::
> mkdir -p $(INSTDOCDIR)
> + mkdir -p $(INSTDOCDIR)/tutorial
> mkdir -p $(INSTDOCDIR)/userguide
> mkdir -p $(INSTDOCDIR)/userguide/explanation
> mkdir -p $(INSTDOCDIR)/userguide/how-to
> mkdir -p $(INSTDOCDIR)/userguide/reference
> mkdir -p $(INSTDOCDIR)/examples/
> mkdir -p $(INSTDOCDIR)/examples/language_features
> - $(call describe-install-target,$(INSTDOCDIR),README INCOMPATIBILITIES NEWS userguide examples)
> + $(call describe-install-target,$(INSTDOCDIR),README INCOMPATIBILITIES NEWS tutorial userguide examples)
> install -m 644 README INCOMPATIBILITIES NEWS $(INSTDOCDIR)
> + install -m 644 doc/tutorial/*.md $(INSTDOCDIR)/tutorial
> install -m 644 doc/userguide/index.md $(INSTDOCDIR)/userguide
> install -m 644 doc/userguide/explanation/* $(INSTDOCDIR)/userguide/explanation
> install -m 644 doc/userguide/how-to/* $(INSTDOCDIR)/userguide/how-to
> diff --git a/doc/tutorial/1.IntroducingDTrace.md b/doc/tutorial/1.IntroducingDTrace.md
> new file mode 100644
> index 000000000..c91cf88e4
> --- /dev/null
> +++ b/doc/tutorial/1.IntroducingDTrace.md
> @@ -0,0 +1,380 @@
> +
> +## Introducing DTrace
> +
> +This chapter introduces the dynamic tracing (DTrace) facility,
> +available on Linux.
> +You can use DTrace to examine the behavior of the operating system
> +and of user-space programs, particularly those that have been
> +instrumented with DTrace probes.
> +DTrace has been implemented on Linux using eBPF,
> +allowing DTrace to run as a user-space tool,
> +albeit as root for most uses.
> +
> +### About This Tutorial
> +
> +This tutorial includes a variety of DTrace scripts and describes
> +different ways in which you can use DTrace.
> +Several examples have additional exercises that offer further
> +practice in using DTrace.
> +You should already have a good understanding of Linux administration
> +and system programming, and broad experience using a programming language,
> +such as C or C++, and a scripting language, such as Python.
> +If you are not familiar with terms such as *system call*, *type*,
> +*cast*, *signal*, *struct*, or *pointer*,
> +you might have difficulty in understanding some of the examples
> +or completing some of the exercises in this tutorial.
> +However, each exercise provides a sample solution in case you do get stuck.
> +You are encouraged to experiment with the examples to develop your
> +skills at creating DTrace scripts.
> +
> +>Caution:
> +>
> +>To run the examples and perform the exercises in this tutorial,
> +you need to have `root` access to a system.
> +Only the `root` user or a user with `sudo` access to run commands
> +as `root` can use the `dtrace` utility.
> +As `root`, you have total power over a
> +system and so have total responsibility for that system.
> +Although DTrace is designed so that you can use it safely
> +without needing to worry about corrupting the operating system
> +or other processes,
> +there are ways to circumvent some of the default, built-in safety measures.
> +>
> +>To minimize risk, perform the examples and exercises in this
> +tutorial on a system other than a production system.
> +>
> +>Also, make sure that `/usr/sbin` is before `/usr/bin` in the root path,
> +or specify `/usr/sbin/dtrace` explicitly.
> +
> +The examples in this tutorial demonstrate the different ways that
> +you can perform dynamic tracing of your system: by entering a
> +simple D program as an argument to `dtrace` on
> +the command line, by using the `dtrace` command
> +to run a script that contains a D program, or by using an
> +executable D script that contains a *hashbang*
> +(`#!` or *shebang*) invocation of `dtrace`.
> +When you create your own D programs,
> +you can choose which method best suits your needs.
> +
> +### About DTrace
> +
> +DTrace is a comprehensive dynamic tracing facility that was first
> +developed for use on the Solaris operating system (now known as
> +Oracle Solaris) and subsequently ported to Oracle Linux.
> +You can use
> +DTrace to explore the operation of your system to better
> +understand how it works, to track down performance problems across
> +many layers of software, or to locate the causes of aberrant
> +behavior.
> +
> +Using DTrace, you can record data at previously instrumented
> +places of interest, which are referred to as
> +*probes*, in kernel and user-space programs.
> +A probe is a location to which DTrace can bind a request to perform
> +a set of actions, such as recording a stack trace, a timestamp, or
> +the argument to a function.
> +Probes function like programmable
> +sensors that record information.
> +When a probe is triggered, DTrace
> +gathers data that you have designated in a D script and reports
> +this data back to you.
> +
> +Using DTrace's D programming language, you can query the system
> +probes to provide immediate and concise answers to any number of
> +questions that you might formulate.
> +
> +A D program describes the actions that occur if one or more
> +specified probes is triggered.
> +A probe is uniquely specified by
> +the name of the DTrace provider that publishes the probe, the name
> +of the module, library, or user-space program in which the probe
> +is located, the name of the function in which the probe is
> +located, and the name of the probe itself, which usually describes
> +some operation or functionality that you can trace.
> +Because you do
> +not need to specify probes exactly, this allows DTrace to perform
> +the same action for a number of different probes.
> +Full and
> +explicit representation of a single probe in the D language takes
> +the form *provider*:*module*:*function*:*name*.
> +
> +When you use the `dtrace` command to run a D
> +program, you invoke the compiler for the D language.
> +The compiled code, eBPF instructions, is loaded into the kernel
> +and attached to the appropriate probes.
> +DTrace handles any runtime errors that might occur
> +during your D program's execution, including dividing by zero,
> +dereferencing invalid memory, and so on, and reports them to you.
> +
> +The *module* is the kernel module in which a probe appears or,
> +in the case of user code, the load object. Some probes, such
> +as clock-triggered profiling interrupts, have no module associated
> +with them.
> +
> +Unless you explicitly permit DTrace to perform potentially
> +destructive actions, you cannot construct an unsafe program that
> +would cause DTrace to inadvertently damage either the operating
> +system or any process that is running on your system. These safety
> +features enable you to use DTrace in a production environment
> +without worrying about crashing or corrupting your system. If you
> +make a programming mistake, DTrace reports the error and
> +deactivates your program's probes. You can then correct your
> +program and try again.
> +
> +For more information about using DTrace, see the
> +[DTrace User Guide](../userguide/index.md).
> +
> +### About DTrace Providers
> +
> +Here are the providers in the Oracle Linux implementation of DTrace:
> +
> +```
> + kernel user space
> + +-----------------------------+
> + | dtrace | I want to control my D program.
> + +----------------+------------+
> + | | syscall | I want to see how I call the kernel.
> + +----------------+------------+
> + | cpc profile | I want to look at resource usage.
> + +----------------+------------+
> + | ip tcp udp | |
> + | io lockstat | | I know the semantics of the code.
> + | sdt rawtp | usdt | (I know what probes are in the code.)
> + | proc sched | |
> + +----------------+------------+
> + | fbt rawfbt | pid | I know the functions of the code.
> + +----------------+------------+
> +```
> +
> +That is, these providers provide:
> +
> +- `dtrace`: probes that relate to DTrace itself,
> + such as `BEGIN`, `ERROR`, and `END`.
> + You can use these probes to initialize DTrace's state before
> + tracing begins, process its state after tracing has completed,
> + and handle unexpected execution errors in other probes.
> +- `syscall`: probes that fire at the entry to and return from every system call.
> + Because system calls are the primary interface between user-level
> + applications and the operating system kernel, these probes can
> + offer you an insight into the interaction between applications and
> + the system.
> +- `cpc`: probes that fire when hardware resource counters (or
> + software emulation thereof) trip, giving you an idea of
> + where statistically hardware events are triggered.
> +- `profile`: probes that fire at fixed and specified time intervals.
> + You can use these probes to sample some aspect of a system's state.
> +- `ip`: probes that fire for important steps in the IP protocol,
> + for both IPv4 and IPv6.
> +- `tcp`: probes that fire for important steps in the TCP protocol,
> + for both IPv4 and IPv6.
> +- `udp`: probes that fire for important steps in the the UDP protocol,
> + for both IPv4 and IPv6.
> +- `io`: probes that relate to data input and output.
> + The `io` provider enables quick exploration of behavior
> + observed through I/O monitoring.
> +- `lockstat`: probes that fire for important steps in lock handling,
> + including for mutexes, read-write locks, and spin locks.
> +- `sdt`: probes that fire when statically defined tracepoints in the Linux kernel
> + are executed.
> +- `rawtp`: probes that fire when statically defined tracepoints in the Linux kernel
> + are executed, but report raw arguments.
> +- `proc`: probes that fire for process creation and termination, LWP creation
> + and termination, execution of new programs, and signal handling.
> +- `sched`: probes that fire for important steps in CPU scheduling.
> + Because CPUs are the one resource that all threads must consume,
> + the `sched` provider is very useful for understanding systemic behavior.
> +- `usdt`: probes that fire when user-space statically defined trace points
> + are encountered.
> +- `fbt`: probes that fire for function boundary tracing (FBT) -- that is,
> + when kernel functions are entered or return.
> +- `rawfbt`: probes that are like `fbt`, but are implemented with `kprobes`,
> + include synthetic functions (such as compiler-generated optimized
> + variants of functions with `.` suffixes.), and have unconverted and
> + untyped probe arguments.
> +- `pid`: probes that fire for the specified process id (`pid`) upon
> + entry into or return from a function.
> +
> +See [DTrace Providers](../userguide/reference/dtrace_providers.md) in
> +the [DTrace User Guide](../userguide/index.md)
> +for more information about providers and their probes.
> +
> +### Preparing to Install and Configure DTrace
> +
> +DTrace is available for Linux from
> +[github](https://github.com/oracle/dtrace-utils/tree/stable).
> +Or, there are
> +[DTrace packages built for Oracle Linux](https://www.oracle.com/linux/downloads/linux-dtrace.html).
> +
> +#### Example: Displaying Probes for a Provider
> +
> +The following example shows how you would display the probes
> +for a provider, such as `proc`, by using the `dtrace` command.
> +
> +```
> +# dtrace -l -P proc
> + ID PROVIDER MODULE FUNCTION NAME
> + 53 proc vmlinux start
> + 52 proc vmlinux signal-send
> + 51 proc vmlinux signal-handle
> + 50 proc vmlinux signal-discard
> + 49 proc vmlinux signal-clear
> + 48 proc vmlinux lwp-start
> + 47 proc vmlinux lwp-exit
> + 46 proc vmlinux lwp-create
> + 45 proc vmlinux exit
> + 44 proc vmlinux exec-success
> + 43 proc vmlinux exec-failure
> + 42 proc vmlinux exec
> + 41 proc vmlinux create
> +```
> +
> +These probes enable you to monitor how the system creates
> +processes, executes programs, and handles signals.
> +
> +The output shows the numeric identifier of the probe, the name
> +of the probe provider, the name of the probe module, the name
> +of the function that contains the probe (none for this provider),
> +and the name of the probe itself.
> +
> +The full name of a probe is *provider*:*module*:*function*:*name*,
> +for example, `proc:vmlinux::create`.
> +
> +When probes are listed, a missing field indicates there is no
> +corresponding value.
> +When you specify a probe, omitting a field is equivalent to using
> +the wildcard `*`.
> +
> +#### Exercise: Enabling and Listing DTrace Probes
> +
> +Try listing the probes of the `syscall` provider.
> +Notice that both `entry` and `return` probes are provided for each system call.
> +
> +#### Solution to Exercise: Enabling and Listing DTrace Probes
> +
> +```
> +# dtrace -l -P syscall
> + ID PROVIDER MODULE FUNCTION NAME
> + 5055 syscall vmlinux socket entry
> + 5054 syscall vmlinux socket return
> + 5053 syscall vmlinux socketpair entry
> + 5052 syscall vmlinux socketpair return
> + 5051 syscall vmlinux bind entry
> + 5050 syscall vmlinux bind return
> + 5049 syscall vmlinux listen entry
> + 5048 syscall vmlinux listen return
> + ...
> + 4375 syscall vmlinux ioperm entry
> + 4374 syscall vmlinux ioperm return
> + 4373 syscall vmlinux iopl entry
> + 4372 syscall vmlinux iopl return
> + 4371 syscall vmlinux rt_sigreturn entry
> + 4370 syscall vmlinux rt_sigreturn return
> + 4369 syscall vmlinux arch_prctl entry
> + 4368 syscall vmlinux arch_prctl return
> +```
> +
> +The probe ID numbers can differ from run to run.
> +
> +### Running a Simple DTrace Program
> +
> +The following example shows how you could use a simple D program
> +that is in a file called `hello.d`.
> +
> +#### Example: Simple D Program That Uses the BEGIN Probe (hello.d)
> +
> +```
> +/* hello.d -- A simple D program that uses the BEGIN probe */
> +
> +BEGIN
> +{
> + /* This is a C-style comment */
> + trace("hello, world");
> + exit(0);
> +}
> +```
> +
> +A D program consists of a series of clauses, where each clause
> +describes one or more probes to enable, and an optional set of actions
> +to perform when the probe fires. The actions are listed as a series
> +of statements enclosed in braces `{}` following the probe name.
> +Each statement ends with a semicolon (`;`).
> +
> +In this example, the function `trace` directs DTrace to record the
> +specified argument, the string ???hello, world???, when the `BEGIN` probe fires,
> +and then print it out.
> +The function `exit()` tells DTrace to cease tracing and exit the `dtrace`
> +command.
> +
> +The full name of the `BEGIN` probe is `dtrace:::BEGIN`.
> +The `dtrace` provider provides three probes: `dtrace:::BEGIN`,
> +`dtrace:::END`, and `dtrace:::ERROR`.
> +Because these probe names are unique to the `dtrace` provider,
> +their names can be shortened to `BEGIN`, `END`, and `ERROR`.
> +
> +When you have saved your program, you can run it by using the
> +`dtrace` command with the `-s` option,
> +which specifies the name of the file that contains the D program:
> +
> +```
> +# dtrace -s hello.d
> +dtrace: script 'hello.d' matched 1 probe
> +CPU ID FUNCTION:NAME
> + 0 1 :BEGIN hello, world
> +```
> +
> +DTrace interprets and runs the script. You will notice
> +that in addition to the string `"hello,world"`,
> +the default behavior of DTrace is to display information about
> +the CPU on which the script was running when a probe fired,
> +the ID of the probe, the name of the function that contains
> +the probe, and the name of the probe itself.
> +The function name is displayed as blank for `BEGIN`,
> +as DTrace provides this probe.
> +
> +You can suppress the probe information in a number of different
> +ways, for example, by specifying the `-q` quiet option:
> +
> +```
> +# dtrace -q -s hello.d
> +hello, world
> +```
> +
> +#### Exercise: Using the END Probe
> +
> +Copy the `hello.d` program to the file `goodbye.d`.
> +Edit this file so that it traces the string "goodbye, world"
> +and uses the `END` probe instead of `BEGIN`.
> +When you run this new script, you need to type `Ctrl-C` to cause the
> +probe to fire and exit `dtrace`.
> +
> +#### Solution to Exercise and Example: Using the END Probe
> +
> +The following is an example of a simple D program that
> +demonstrates the use of the `END` probe:
> +
> +```
> +/* goodbye.d -- Simple D program that demonstrates the END probe */
> +
> +END
> +{
> + trace("goodbye, world");
> +}
> +```
> +
> +The default output is:
> +
> +```
> +# dtrace -s goodbye.d
> +dtrace: script 'goodbye.d' matched 1 probe
> +^C
> +CPU ID FUNCTION:NAME
> + 3 2 :END goodbye, world
> +```
> +
> +Or, with the `-q` quiet option:
> +
> +```
> +# dtrace -q -s ./goodbye.d
> +^C
> +goodbye, world
> +```
> diff --git a/doc/tutorial/2.TracingOperatingSystemBehavior.md b/doc/tutorial/2.TracingOperatingSystemBehavior.md
> new file mode 100644
> index 000000000..b3445ab6d
> --- /dev/null
> +++ b/doc/tutorial/2.TracingOperatingSystemBehavior.md
> @@ -0,0 +1,1741 @@
> +
> +## Tracing Operating System Behavior
> +
> +This chapter provides examples of D programs that you can use to
> +investigate what is happening in the operating system.
> +
> +### Tracing Process Creation
> +
> +The `proc` probes enable you to trace process creation and termination,
> +execution of new program images, and signal processing on a system. See
> +[Proc Provider](../userguide/reference/dtrace_providers_proc.md)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
> +for a description of the `proc` probes and their arguments.
> +
> +#### Example: Monitoring the System as Programs Are Executed (execcalls.d)
> +
> +The following example shows the D program, `execcalls.d`, which uses
> +`proc` probes to monitor the system as it executes process images:
> +
> +```
> +/* execcalls.d -- Monitor the system as it executes programs */
> +
> +proc:::exec
> +{
> + trace(stringof(args[0]));
> +}
> +```
> +
> +The `args[0]` argument to the `exec` probe is set to the path name of the
> +program that is being executed.
> +You use the `stringof()` function to convert the type from
> +`char *` to the D type `string`.
> +
> +Type the `dtrace -s execcalls.d` command to run the D program in one window.
> +Then start different programs from another window,
> +while observing the output from `dtrace` in the first window.
> +To stop tracing after a few seconds have elapsed,
> +type `Ctrl-C` in the window that is running `dtrace`.
> +
> +```
> +# dtrace -s execcalls.d
> +dtrace: script 'execcalls.d' matched 1 probe
> +CPU ID FUNCTION:NAME
> + 3 42 :exec /usr/sbin/sshd
> + 1 42 :exec /bin/bash
> + 0 42 :exec /usr/bin/id
> + 2 42 :exec /usr/bin/hostnamectl
> + 1 42 :exec /usr/lib/systemd/systemd-hostnamed
> + 3 42 :exec /usr/bin/register-python-argcomplete
> + 3 42 :exec /usr/libexec/grepconf.sh
> + 0 42 :exec /usr/bin/grep
> + 2 42 :exec /usr/bin/dircolors
> + 3 42 :exec /usr/bin/grep
> + 3 42 :exec /usr/libexec/grepconf.sh
> +^C
> +```
> +
> +The activity here shows a login to the same system (from another
> +terminal) while the script is running.
> +
> +The probe `proc:::exec` fires whenever the system executes a new program and
> +the associated action uses `trace()` to display the path name of the program.
> +
> +#### Exercise: Suppressing Verbose Output From DTrace
> +
> +Run the `execcalls.d` program again, but this time add the
> +`-q` option to suppress all output except output from `trace()`.
> +Notice how DTrace displays only what you traced with `trace()`.
> +
> +#### Solution to Exercise: Suppressing Verbose Output From DTrace
> +
> +```
> +# dtrace -q -s execcalls.d
> +/usr/sbin/sshd/bin/bash/usr/bin/id/usr/bin/hostnamectl/usr/lib/systemd/systemd-h
> +ostnamed/usr/bin/register-python-argcomplete/usr/libexec/grepconf.sh/usr/bin/gre
> +p/usr/bin/dircolors/usr/bin/grep/usr/libexec/grepconf.sh^C
> +```
> +
> +### Tracing System Calls
> +
> +System calls are the interface between user programs and the
> +kernel, which perform operations on the programs' behalf.
> +
> +The next example shows the next D program,
> +`syscalls.d`, which uses
> +`syscall` probes to record
> +`open()` system call activity on a system.
> +
> +#### Example: Recording open() System Calls on a System (syscalls.d)
> +
> +```
> +/* syscalls.d -- Record open() system calls on a system */
> +
> +syscall::open:entry
> +{
> + printf("%-16s %-16s\n",execname,copyinstr(arg0));
> +}
> +syscall::openat*:entry
> +{
> + printf("%-16s %-16s\n",execname,copyinstr(arg1));
> +}
> +```
> +
> +In this example, the `printf()` function is used to display the
> +name of the executable that is calling `open()` and the path name
> +of the file that it is attempting to open.
> +
> +>Note:
> +>
> +>Use the `copyinstr()` function to convert the first argument (`arg0`)
> +in the `open()` call to a string.
> +Whenever a probe accesses a pointer to data in the address space of a user process,
> +you must use one of the `copyin()`, `copyinstr()`, or `copyinto()` functions
> +to copy the data from user space to a DTrace buffer in kernel space.
> +In this example, it is appropriate to use `copyinstr()`,
> +as the pointer refers to a character array.
> +If the string is not null-terminated, you also need to specify the
> +length of the string to `copyinstr()`, for example,
> +`copyinstr(arg1, arg2 + 1)`, for a system call such as `write()`,
> +to copy arg2 bytes and then add a NUL terminating character after those bytes,
> +which are possibly already NUL-terminated.
> +See
> +[Pointers and Address Spaces](../userguide/reference/dtrace-ref-PointersandScalarArrays.md#pointers_and_address_spaces)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
> +
> +```
> +# dtrace -q -s syscalls.d
> +irqbalance /proc/interrupts
> +irqbalance /proc/stat
> +irqbalance /proc/interrupts
> +irqbalance /proc/stat
> +irqbalance /proc/interrupts
> +irqbalance /proc/stat
> +NetworkManager /proc/sys/net/ipv6/conf/enX0/mtu
> +irqbalance /proc/interrupts
> +irqbalance /proc/stat
> +systemd /proc/1200459/cgroup
> +NetworkManager /proc/sys/net/ipv6/conf/enX0/mtu
> +^C
> +```
> +
> +#### Exercise: Using the printf() Function to Format Output
> +
> +Amend the arguments to the `printf()` function so that `dtrace`
> +also prints the process ID and user ID for the process.
> +Use a conversion specifier such as `%-4d`.
> +
> +See
> +[printf](../userguide/reference/function_printf.md)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
> +for a description of the function.
> +
> +The process ID and user ID are available as the variables
> +`pid` and `uid`. Use the
> +`BEGIN` probe to create a header for the
> +output.
> +
> +#### Solution to Exercise: Using the printf() Function to Format Output
> +
> +```
> +/* syscalls1.d -- Modified version of syscalls.d that displays more information */
> +
> +BEGIN
> +{
> + printf("%-7s %-4s %-16s %-16s\n","PID","UID","EXECNAME","FILENAME");
> + mypid = pid;
> +}
> +
> +syscall::open:entry
> +/pid != mypid/
> +{
> + printf("%-7d %-4d %-16s %-16s\n",pid,uid, execname,copyinstr(arg0));
> +}
> +syscall::openat*:entry
> +/pid != mypid/
> +{
> + printf("%-7d %-4d %-16s %-16s\n",pid,uid, execname,copyinstr(arg1));
> +}
> +```
> +
> +Note how this solution uses similar formatting strings to output
> +the header and the data.
> +Also, we add *predicates* to exclude probes that fire on the `dtrace`
> +process itself.
> +
> +```
> +# dtrace -q -s syscalls1.d
> +PID UID EXECNAME FILENAME
> +2852652 0 nm-daemon-helpe /etc/ld.so.cache
> +2852652 0 nm-daemon-helpe /lib64/libgcc_s.so.1
> +2852652 0 nm-daemon-helpe /lib64/libc.so.6
> +2852652 0 nm-daemon-helpe /etc/nsswitch.conf
> +2852652 0 nm-daemon-helpe /etc/host.conf
> +2852652 0 nm-daemon-helpe /etc/resolv.conf
> +2852652 0 nm-daemon-helpe /etc/hosts
> +2852653 0 nm-daemon-helpe /etc/ld.so.cache
> +2852653 0 nm-daemon-helpe /lib64/libgcc_s.so.1
> +2852653 0 nm-daemon-helpe /lib64/libc.so.6
> +2852653 0 nm-daemon-helpe /etc/nsswitch.conf
> +2852653 0 nm-daemon-helpe /etc/host.conf
> +2852653 0 nm-daemon-helpe /etc/resolv.conf
> +2852653 0 nm-daemon-helpe /etc/hosts
> +1200456 0 irqbalance /proc/interrupts
> +1200456 0 irqbalance /proc/stat
> +1 0 systemd /proc/1200459/cgroup
> +1 0 systemd /proc/1200460/cgroup
> +^C
> +```
> +
> +### Performing an Action at Specified Intervals
> +
> +The `profile` provider includes `tick` probes that you can use
> +to sample some aspect of a system's state at regular intervals.
> +
> +#### Example: Using `tick.d`
> +
> +The following is an example of the `tick.d` program.
> +
> +```
> +/* tick.d -- Perform an action at regular intervals */
> +
> +BEGIN
> +{
> + i = 0;
> +}
> +
> +profile:::tick-1sec
> +{
> + printf("i = %d\n",++i);
> +}
> +
> +END
> +{
> + trace(i);
> +}
> +```
> +
> +In this example, the program initializes (and thereby implicitly
> +declares) the variable `i` when the D program starts,
> +increments the variable and prints its value once every second,
> +and displays the final value of `i` when the program exits.
> +
> +When you run this program, it produces output that is similar to
> +the following, until you type `Ctrl-C`:
> +
> +```
> +# dtrace -s tick.d
> +dtrace: script 'tick.d' matched 3 probes
> +CPU ID FUNCTION:NAME
> + 1 5315 :tick-1sec i = 1
> +
> + 1 5315 :tick-1sec i = 2
> +
> + 1 5315 :tick-1sec i = 3
> +
> + 1 5315 :tick-1sec i = 4
> +
> + 1 5315 :tick-1sec i = 5
> +
> + 1 5315 :tick-1sec i = 6
> +
> +^C
> + 1 5315 :tick-1sec i = 7
> +
> + 0 2 :END 7
> +```
> +
> +To suppress all of the output except the output from
> +`printf()` and `trace()`, specify the `-q` option:
> +
> +```
> +# dtrace -q -s tick.d
> +i = 1
> +i = 2
> +i = 3
> +i = 4
> +^C
> +i = 5
> +5
> +```
> +
> +#### Exercise: Using tick Probes
> +
> +List the available `profile` provider probes.
> +Experiment with using a different `tick` probe.
> +Replace the `trace()` call in
> +`END` with a `printf()` call.
> +
> +See
> +[Profile Provider](../userguide/reference/dtrace_providers_profile.md)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
> +for a description of the probes.
> +
> +#### Solution to Exercise and Example: Using tick Probes
> +
> +```
> +# dtrace -l -P profile
> + ID PROVIDER MODULE FUNCTION NAME
> + 5 profile profile-97
> + 6 profile profile-199
> + 7 profile profile-499
> + 8 profile profile-997
> + 9 profile profile-1999
> + 10 profile profile-4001
> + 11 profile profile-4999
> + 12 profile tick-1
> + 13 profile tick-10
> + 14 profile tick-100
> + 15 profile tick-500
> + 16 profile tick-1000
> + 17 profile tick-5000
> +5315 profile tick-1sec
> +5316 profile tick-10sec
> +```
> +
> +#### Example: Modified Version of tick.d
> +
> +```
> +/* tick1.d -- Modified version of tick.d */
> +
> +BEGIN
> +{
> + i = 0;
> +}
> +
> +/* tick-500ms fires every 500 milliseconds */
> +profile:::tick-500ms
> +{
> + printf("i = %d\n",++i);
> +}
> +
> +END
> +{
> + printf("\nFinal value of i = %d\n",i);
> +}
> +```
> +
> +This example uses the `tick-500ms` probe,
> +which fires twice per second.
> +
> +```
> +# dtrace -s tick1.d
> +dtrace: script 'tick1.d' matched 3 probes
> +CPU ID FUNCTION:NAME
> + 2 642 :tick-500ms i = 1
> +
> + 2 642 :tick-500ms i = 2
> +
> + 2 642 :tick-500ms i = 3
> +
> + 2 642 :tick-500ms i = 4
> +
> +^C
> + 2 642 :tick-500ms i = 5
> +
> + 3 2 :END
> +Final value of i = 5
> +```
> +
> +### Using Predicates to Select Actions
> +
> +Predicates are logic statements that choose whether DTrace invokes
> +the actions that are associated with a probe. You can use
> +predicates to focus tracing analysis on specific contexts under
> +which a probe fires.
> +
> +#### Example: Using daterun.d
> +
> +The following example shows an executable DTrace script, `daterun.d`, which
> +displays the file descriptor, output string, and string length specified to
> +the `write()` system call whenever the `date` command is run on the system.
> +
> +```
> +#!/usr/sbin/dtrace -qs
> +
> +/* daterun.d -- Display arguments to write() when date runs */
> +
> +syscall::write:entry
> +/execname == "date"/
> +{
> + printf("%s(%d, %s, %d)\n", probefunc, arg0, copyinstr(arg1, arg2 + 1), arg2);
> +}
> +```
> +
> +In the example, the predicate is `/execname == "date"/`, which
> +specifies that if the probe `syscall::write:entry` is triggered,
> +DTrace runs the associated action only if the name of the executable is `date`.
> +The `copyinstr()` string will have room for `arg2+1` bytes,
> +including a NUL-terminated byte in case the first `arg2` bytes need it.
> +
> +Make the script executable by changing its mode:
> +
> +```
> +# chmod +x daterun.d
> +```
> +
> +If you run the script from one window,
> +while typing the `date` command in another window,
> +output similar to the following is displayed in the first window:
> +
> +```
> +# ./daterun.d
> +write(1, Tue Dec 9 11:14:43 GMT 2025
> +, 29)
> +```
> +
> +#### Example: Listing Available syscall Provider Probes
> +
> +The following example shows how you would list available
> +`syscall` provider probes.
> +
> +```
> +# dtrace -l -P syscall | less
> + ID PROVIDER MODULE FUNCTION NAME
> + 18 syscall vmlinux read entry
> + 19 syscall vmlinux read return
> + 20 syscall vmlinux write entry
> + 21 syscall vmlinux write return
> + 22 syscall vmlinux open entry
> + 23 syscall vmlinux open return
> + 24 syscall vmlinux close entry
> + 25 syscall vmlinux close return
> + 26 syscall vmlinux newstat entry
> + 27 syscall vmlinux newstat return
> +...
> + 648 syscall vmlinux pkey_alloc entry
> + 649 syscall vmlinux pkey_alloc return
> + 650 syscall vmlinux pkey_free entry
> + 651 syscall vmlinux pkey_free return
> + 652 syscall vmlinux statx entry
> + 653 syscall vmlinux statx return
> + 654 syscall vmlinux waitfd entry
> + 655 syscall vmlinux waitfd return
> +```
> +
> +#### Exercise: Using syscall Probes
> +
> +Experiment by adapting the `daterun.d` script for another program.
> +Make the new script produce output when the system is running `w`.
> +
> +#### Solution to Exercise: Using syscall Probes
> +
> +```
> +#!/usr/sbin/dtrace -qs
> +
> +/* wrun.d -- Modified version of daterun.d for the w command */
> +
> +syscall::write:entry
> +/execname == "w"/
> +{
> + printf("%s(%d, %s, %d)\n", probefunc, arg0, copyinstr(arg1, arg2 + 1), arg2);
> +}
> +```
> +
> +We can run the script as follows:
> +
> +```
> +# chmod +x wrun.d
> +# ./wrun.d
> +write(1, 12:14:55 up 3:21, 3 users, load average: 0.14, 0.15, 0.18
> +, 62)
> +write(1, USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
> +, 61)
> +write(1, guest pts/0 :0.0 08:57 7.00s 0.17s 0.03s w
> +, 58)
> +...
> +^C
> +```
> +
> +### Timing Events on a System
> +
> +Determining the time that a system takes to perform different
> +activities is a fundamental technique for analyzing its operation
> +and determining where bottlenecks might be occurring.
> +
> +#### Example: Monitoring read() System Call Duration (readtrace.d)
> +
> +The following is an example of the D program, `readtrace.d`.
> +
> +```
> +/* readtrace.d -- Display time spent in read() calls */
> +
> +syscall::read:entry
> +{
> + self->t = timestamp; /* Initialize a thread-local variable */
> +}
> +
> +syscall::read:return
> +/self->t != 0/
> +{
> + printf("%s (pid=%d) spent %d microseconds in read()\n",
> + execname, pid, ((timestamp - self->t)/1000)); /* Divide by 1000 for microseconds */
> +
> + self->t = 0; /* Reset the variable */
> +}
> +```
> +
> +In the example, the `readtrace.d` program displays the command name,
> +process ID, and call duration in microseconds whenever a process
> +invokes the `read()` system call.
> +The variable `self->t` is *thread-local*,
> +meaning that it exists only within the scope of execution of a
> +thread on the system.
> +
> +The program records the value of `timestamp` in `self->t` when
> +the process calls `read()`, and subtracts this value from the
> +value of `timestamp` when the call returns.
> +The units of `timestamp` are nanoseconds,
> +so you divide by 1000 to obtain a value in microseconds.
> +
> +The following is output from running this program:
> +
> +```
> +# dtrace -q -s readtrace.d
> +NetworkManager (pid=878) spent 10 microseconds in read()
> +NetworkManager (pid=878) spent 9 microseconds in read()
> +NetworkManager (pid=878) spent 2 microseconds in read()
> +in:imjournal (pid=815) spent 63 microseconds in read()
> +gdbus (pid=878) spent 7 microseconds in read()
> +gdbus (pid=878) spent 66 microseconds in read()
> +gdbus (pid=878) spent 63 microseconds in read()
> +irqbalance (pid=816) spent 56 microseconds in read()
> +irqbalance (pid=816) spent 113 microseconds in read()
> +irqbalance (pid=816) spent 104 microseconds in read()
> +irqbalance (pid=816) spent 91 microseconds in read()
> +irqbalance (pid=816) spent 61 microseconds in read()
> +irqbalance (pid=816) spent 63 microseconds in read()
> +irqbalance (pid=816) spent 61 microseconds in read()
> +irqbalance (pid=816) spent 61 microseconds in read()
> +irqbalance (pid=816) spent 61 microseconds in read()
> +irqbalance (pid=816) spent 61 microseconds in read()
> +irqbalance (pid=816) spent 61 microseconds in read()
> +irqbalance (pid=816) spent 61 microseconds in read()
> +sshd (pid=10230) spent 8 microseconds in read()
> +in:imjournal (pid=815) spent 6 microseconds in read()
> +sshd (pid=10230) spent 7 microseconds in read()
> +in:imjournal (pid=815) spent 5 microseconds in read()
> +sshd (pid=10230) spent 7 microseconds in read()
> +in:imjournal (pid=815) spent 6 microseconds in read()
> +sshd (pid=10230) spent 7 microseconds in read()
> +in:imjournal (pid=815) spent 5 microseconds in read()
> +^C
> +```
> +
> +#### Exercise: Timing System Calls
> +
> +Add a predicate to the `entry` probe in `readtrace.d` so that `dtrace`
> +displays results for a disk space usage report that is selected
> +by the name of its executable (`df`).
> +
> +#### Solution to Exercise: Timing System Calls
> +
> +The following example shows a modified version of the
> +`readtrace.d` program that includes a predicate.
> +
> +```
> +/* readtrace1.d -- Modified version of readtrace.d that includes a predicate */
> +
> +syscall::read:entry
> +/execname == "df"/
> +{
> + self->t = timestamp;
> +}
> +
> +syscall::read:return
> +/self->t != 0/
> +{
> + printf("%s (pid=%d) spent %d microseconds in read()\n",
> + execname, pid, ((timestamp - self->t)/1000));
> +
> + self->t = 0; /* Reset the variable */
> +}
> +```
> +
> +The predicate `/execname == "df"/` tests whether the
> +`df` program is running when the probe fires.
> +
> +```
> +# dtrace -q -s readtrace1.d
> +df (pid=1666) spent 6 microseconds in read()
> +df (pid=1666) spent 8 microseconds in read()
> +df (pid=1666) spent 1 microseconds in read()
> +df (pid=1666) spent 50 microseconds in read()
> +df (pid=1666) spent 38 microseconds in read()
> +df (pid=1666) spent 10 microseconds in read()
> +df (pid=1666) spent 1 microseconds in read()
> +^C
> +```
> +
> +#### Exercise: Timing All System Calls for cp (calltrace.d)
> +
> +Using the `probefunc` variable and the `syscall:::entry` and
> +`syscall:::return` probes, create a D program, `calltrace.d`,
> +which times all system calls for the executable `cp`.
> +
> +#### Solution to Exercise: Timing All System Calls for cp (calltrace.d)
> +
> +```
> +/* calltrace.d -- Time all system calls for cp */
> +
> +syscall:::entry
> +/execname == "cp"/
> +{
> + self->t = timestamp; /* Initialize a thread-local variable */
> +}
> +
> +syscall:::return
> +/self->t != 0/
> +{
> + printf("%s (pid=%d) spent %d microseconds in %s()\n",
> + execname, pid, ((timestamp - self->t)/1000), probefunc);
> +
> + self->t = 0; /* Reset the variable */
> +}
> +```
> +
> +Dropping the function name `read` from the probe specifications
> +matches all instances of `entry` and `return` probes for `syscall`.
> +The following is a check for system calls resulting from running
> +the `cp` executable:
> +
> +```
> +# dtrace -q -s calltrace.d
> +cp (pid=2801) spent 4 microseconds in brk()
> +cp (pid=2801) spent 5 microseconds in mmap()
> +cp (pid=2801) spent 15 microseconds in access()
> +cp (pid=2801) spent 7 microseconds in open()
> +cp (pid=2801) spent 2 microseconds in newfstat()
> +cp (pid=2801) spent 3 microseconds in mmap()
> +cp (pid=2801) spent 1 microseconds in close()
> +cp (pid=2801) spent 8 microseconds in open()
> +cp (pid=2801) spent 3 microseconds in read()
> +cp (pid=2801) spent 1 microseconds in newfstat()
> +cp (pid=2801) spent 4 microseconds in mmap()
> +cp (pid=2801) spent 12 microseconds in mprotect()
> + ...
> +cp (pid=2801) spent 183 microseconds in open()
> +cp (pid=2801) spent 1 microseconds in newfstat()
> +cp (pid=2801) spent 1 microseconds in fadvise64()
> +cp (pid=2801) spent 17251 microseconds in read()
> +cp (pid=2801) spent 80 microseconds in write()
> +cp (pid=2801) spent 58 microseconds in read()
> +cp (pid=2801) spent 57 microseconds in close()
> +cp (pid=2801) spent 85 microseconds in close()
> +cp (pid=2801) spent 57 microseconds in lseek()
> +cp (pid=2801) spent 56 microseconds in close()
> +cp (pid=2801) spent 56 microseconds in close()
> +cp (pid=2801) spent 56 microseconds in close()
> +^C
> +```
> +
> +### Tracing Parent and Child Processes
> +
> +When a process forks, it creates a child process that is effectively
> +a copy of its parent process, but with a different process ID.
> +For information about other differences, see the `fork(2)` manual page.
> +The child process can either run independently from its parent process
> +to perform some separate task.
> +Or, a child process can execute a new program image that replaces
> +the child's program image while retaining the same process ID.
> +
> +#### Example: Using proc Probes to Report Activity on a System (activity.d)
> +
> +The D program `activity.d` in the following example uses `proc`
> +probes to report `fork()` and `exec()` activity on a system.
> +
> +```
> +#pragma D option quiet
> +
> +/* activity.d -- Record fork() and exec() activity */
> +
> +proc:::create
> +{
> + /* Extract PID of child process from the psinfo_t pointed to by args[0] */
> + childpid = args[0]->pr_pid;
> +
> + time[childpid] = timestamp;
> + p_pid[childpid] = pid; /* Current process ID (parent PID of new child) */
> + p_name[childpid] = execname; /* Parent command name */
> + p_exec[childpid] = ""; /* Child has not yet been exec'ed */
> +}
> +
> +proc:::exec
> +/p_pid[pid] != 0/
> +{
> + p_exec[pid] = args[0]; /* Child process path name */
> +}
> +
> +proc:::exit
> +/p_pid[pid] != 0 && p_exec[pid] != ""/
> +{
> + printf("%s (%d) executed %s (%d) for %d microseconds\n",
> + p_name[pid], p_pid[pid], p_exec[pid], pid, (timestamp - time[pid])/1000);
> +}
> +
> +proc:::exit
> +/p_pid[pid] != 0 && p_exec[pid] == ""/
> +{
> + printf("%s (%d) forked itself (as %d) for %d microseconds\n",
> + p_name[pid], p_pid[pid], pid, (timestamp - time[pid])/1000);
> +}
> +```
> +
> +In the example, the statement `#pragma D option quiet` has the same
> +effect as specifying the `-q` option on the command line.
> +
> +The process ID of the child process (`childpid`), following a `fork()`,
> +is determined by examining the `pr_pid` member of the `psinfo_t` data
> +structure that is pointed to by the `args[0]` probe argument.
> +For more information about the arguments to `proc` probes, see
> +[Proc Provider](../userguide/reference/dtrace_providers_proc.md)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
> +
> +The program uses the value of the child process ID to initialize
> +globally unique associative array entries, such as `p_pid[childpid]`.
> +
> +>Note:
> +>
> +>An *associative array* is similar to a normal array, in that it associates
> +keys with values, but the keys can be of any type; they need not be integers.
> +
> +When you run the program, you should see output similar to the following as you
> +use the `ssh` command to access the same system from another terminal window.
> +You might want to try running different programs from this new terminal
> +window to generate additional output:
> +
> +```
> +# dtrace -s activity.d
> +sshd (3966) forked itself (as 3967) for 3667020 microseconds
> +bash (3971) forked itself (as 3972) for 1718 microseconds
> +bash (3973) executed /usr/bin/hostname (3974) for 1169 microseconds
> +grepconf.sh (3975) forked itself (as 3976) for 1333 microseconds
> +bash (3977) forked itself (as 3978) for 967 microseconds
> +bash (3977) executed /usr/bin/tput (3979) for 1355 microseconds
> +bash (3980) executed /usr/bin/dircolors (3981) for 1212 microseconds
> +sshd (3966) executed /usr/sbin/unix_chkpwd (3968) for 31444 microseconds
> +sshd (3966) executed /usr/sbin/unix_chkpwd (3969) for 1653 microseconds
> +bash (3970) forked itself (as 3971) for 2411 microseconds
> +bash (3970) forked itself (as 3973) for 1830 microseconds
> +bash (3970) executed /usr/libexec/grepconf.sh (3975) for 3696 microseconds
> +bash (3970) forked itself (as 3977) for 3273 microseconds
> +bash (3970) forked itself (as 3980) for 1928 microseconds
> +bash (3970) executed /usr/bin/grep (3982) for 1570 microseconds
> +^C
> +```
> +
> +#### Exercise: Using a Predicate to Control the Execution of an Action
> +
> +Modify `activity.d` so that `dtrace` displays results for parent processes
> +that are selected by their executable name, for example, `bash`,
> +or by a program name that you specify as an argument to the `dtrace` command.
> +
> +#### Solution to Exercise: Using a Predicate to Control the Execution of an Action
> +
> +The only change that is required to specify the name of an executable
> +is to add a predicate to the `proc::_:create` probe, for example:
> +
> +```
> +/execname == "bash"/
> +```
> +
> +A more generic version of the program sets the predicate check
> +value from a passed-in command-line argument instead, for example:
> +
> +```
> +/execname == $1/
> +```
> +
> +#### Example: Recording fork() and exec() Activity for a Specified Program (activity1.d)
> +
> +The following example uses a predicate that is passed in from the command line.
> +
> +```
> +#pragma D option quiet
> +
> +/* activity1.d -- Record fork() and exec() activity for a specified program */
> +
> +proc:::create
> +/execname == $1/
> +{
> + /* Extract PID of child process from the psinfo_t pointed to by args[0] */
> + childpid = args[0]->pr_pid;
> +
> + time[childpid] = timestamp;
> + p_pid[childpid] = pid; /* Current process ID (parent PID of new child) */
> + p_name[childpid] = execname; /* Parent command name */
> + p_exec[childpid] = ""; /* Child has not yet been exec'ed */
> +}
> +
> +proc:::exec
> +/p_pid[pid] != 0/
> +{
> + p_exec[pid] = args[0]; /* Child process path name */
> +}
> +
> +proc:::exit
> +/p_pid[pid] != 0 && p_exec[pid] != ""/
> +{
> + printf("%s (%d) executed %s (%d) for %d microseconds\n",
> + p_name[pid], p_pid[pid], p_exec[pid], pid, (timestamp - time[pid])/1000);
> +}
> +
> +proc:::exit
> +/p_pid[pid] != 0 && p_exec[pid] == ""/
> +{
> + printf("%s (%d) forked itself (as %d) for %d microseconds\n",
> + p_name[pid], p_pid[pid], pid, (timestamp - time[pid])/1000);
> +}
> +```
> +
> +As shown in the following example, you can now specify the name
> +of the program to be traced as an argument to the `dtrace` command.
> +Note that you need to escape the argument to protect the double quotes
> +from the shell:
> +
> +```
> +# dtrace -s activity.d '"bash"'
> +bash (10367) executed /bin/ps (10368) for 10926 microseconds
> +bash (10360) executed /usr/bin/tty (10361) for 3046 microseconds
> +bash (10359) forked itself (as 10363) for 32005 microseconds
> +bash (10366) executed /bin/basename (10369) for 1285 microseconds
> +bash (10359) forked itself (as 10370) for 12373 microseconds
> +bash (10360) executed /usr/bin/tput (10362) for 34409 microseconds
> +bash (10363) executed /usr/bin/dircolors (10364) for 29527 microseconds
> +bash (10359) executed /bin/grep (10365) for 21024 microseconds
> +bash (10366) forked itself (as 10367) for 11749 microseconds
> +bash (10359) forked itself (as 10360) for 41918 microseconds
> +bash (10359) forked itself (as 10366) for 14197 microseconds
> +bash (10370) executed /usr/bin/id (10371) for 11729 microseconds
> +^C
> +```
> +
> +### Simple Data Aggregations
> +
> +DTrace provides several functions for aggregating the data that individual
> +probes gather.
> +These functions include `avg()`, `count()`, `max()`, `min()`, `stddev()`,
> +and `sum()`, which return the average (mean), count (number), maximum value,
> +minimum value, standard deviation, and summation of the data being gathered,
> +respectively.
> +See
> +[Aggregations](../userguide/reference/aggregation.md)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
> +for descriptions of aggregation functions.
> +
> +DTrace indexes the results of an aggregation by using a tuple
> +expression similar to what is used for an associative array:
> +
> +```
> + at name[list_of_keys] = aggregating_function(args);
> +```
> +
> +The name of the aggregation is prefixed with an `@` character.
> +The keys describe the data that the aggregating function is collecting.
> +If you do not specify a name for the aggregation, DTrace uses `@` as an
> +anonymous aggregation name, which is usually sufficient for simple D programs.
> +
> +#### Example: Counting the Number of write() System Calls Invoked by Processes
> +
> +In the following example, the command counts the number of `write()`
> +system calls that are invoked by processes, until you type `Ctrl-C`.
> +
> +```
> +# dtrace -n 'syscall::write:entry { @["write() calls"] = count(); }'
> +dtrace: description 'syscall:::' matched 1 probe
> +^C
> +
> + write() calls 6932
> +```
> +
> +>Note:
> +>
> +>Rather than create a separate D script for this simple example,
> +the probe and the action is specified on the `dtrace` command line.
> +
> +DTrace prints the result of the aggregation automatically.
> +Alternatively, you can use the `printa()`
> +function to format the result of the aggregation.
> +
> +#### Example: Counting the Number of read() and write() System Calls
> +
> +The following example counts the number of both
> +`read()` and `write()` system calls.
> +
> +```
> +# dtrace -n 'syscall::write:entry,syscall::read:entry { @[strjoin(probefunc,"() calls")] = count(); }'
> +dtrace: description 'syscall::write:entry,syscall::read:entry' matched 2 probes
> +^C
> +
> + write() calls 150
> + read() calls 1555
> +```
> +
> +#### Exercise: Counting System Calls Over a Fixed Period
> +
> +Write a D program named `countcalls.d` that uses a `tick` probe and `exit()`
> +to stop collecting data after 100 seconds and display the number of `open()`,
> +`read()` and `write()` calls.
> +
> +#### Solution to Exercise and Example: Counting Write, Read, and Open System Calls Over 100 Seconds (countcalls.d)
> +
> +```
> +/* countcalls.d -- Count write, read, and open system calls over 100 seconds */
> +
> +profile:::tick-100sec
> +{
> + exit(0);
> +}
> +
> +syscall::write:entry, syscall::read:entry, syscall::open:entry
> +{
> + @[strjoin(probefunc,"() calls")] = count();
> +}
> +```
> +
> +The action that is associated with the `tick-100s` probe means that
> +`dtrace` exits after 100 seconds. By default, exit will print the
> +results of the aggregation.
> +
> +```
> +# dtrace -s countcalls.d
> +dtrace: script 'countcalls.d' matched 4 probes
> +CPU ID FUNCTION:NAME
> + 3 643 :tick-100sec
> +
> + write() calls 1062
> + open() calls 1672
> + read() calls 29672
> +```
> +
> +#### Example: Counting System Calls Invoked by a Process (countsyscalls.d)
> +
> +The D program `countsyscalls.d` shown in the following example counts
> +the number of times a process that is specified by its process ID
> +invokes different system calls.
> +
> +```
> +#!/usr/sbin/dtrace -qs
> +
> +/* countsyscalls.d -- Count system calls invoked by a process */
> +
> +syscall:::entry
> +/pid == $1/
> +{
> + @num[probefunc] = count();
> +}
> +```
> +
> +After making the `syscalls.d` file executable,
> +you can run it from the command line,
> +specifying a process ID as its argument.
> +
> +The following example shows how you would monitor the use of the
> +`emacs` program that was previously invoked.
> +After the script is invoked, within `emacs` a couple files are opened,
> +modified, and then saved before exiting the D script.
> +
> +Make the script executable:
> +
> +```
> +# chmod +x countsyscalls.d
> +```
> +
> +From another command line, type:
> +
> +```
> +# emacs foobar.txt
> +```
> +
> +Now, start the script and use the opened `emacs` window:
> +
> +```
> +# ./countsyscalls.d $(pgrep -u root emacs)
> + ^C
> +
> + chmod 1
> + exit_group 1
> + futex 1
> + getpgrp 1
> + lseek 1
> + lsetxattr 1
> + rename 1
> + fsync 2
> + lgetxattr 2
> + alarm 3
> + rt_sigaction 3
> + unlink 3
> + mmap 4
> + munmap 4
> + symlink 4
> + fcntl 6
> + newfstat 6
> + getgid 7
> + getuid 7
> + geteuid 8
> + openat 8
> + access 9
> + getegid 14
> + open 14
> + getdents 15
> + close 17
> + readlink 19
> + newlstat 33
> + newstat 155
> + read 216
> + timer_settime 231
> + write 314
> + pselect6 376
> + rt_sigreturn 393
> + ioctl 995
> + rt_sigprocmask 1261
> + clock_gettime 3495
> +```
> +
> +In the preceding example, the `pgrep` command is used to determine the
> +process ID of the `emacs` program that the `root` user is running.
> +
> +#### Exercise: Tracing Processes That Are Run by a User
> +
> +Create a program `countprogs.d` that counts and displays the number of
> +times a user (specified by their user name) runs different programs.
> +You can use the `id -u` command to obtain the ID that corresponds to a user name.
> +
> +#### Solution to Exercise and Example: Counting Programs Invoked by a Specified User (countprogs.d)
> +
> +```
> +#!/usr/sbin/dtrace -qs
> +
> +/* countprogs.d -- Count programs invoked by a specified user */
> +
> +proc:::exec
> +/uid == $1/
> +{
> + @num[execname] = count();
> +}
> +```
> +
> +The predicate `/uid == $1/` compares the effective UID for each program
> +that is run against the argument specified on the command line.
> +You can use the `id -u` command to find out the ID of the guest user account, for example:
> +
> +```
> +# chmod +x countprogs.d
> +# ./countprogs.d $(id -u guest)
> +^C
> +
> +less 1
> +lesspipe.sh 1
> +sh 1
> +bash 9
> +```
> +
> +You can use the same command for the `root` user, which is typically user `0`.
> +For testing purposes, you might want to have the user account under a test
> +login by using another window and then run some nominal programs.
> +
> +#### Example: Counting the Number of Times a Program Reads From Different Files in 10 Seconds (fdscount.d)
> +
> +The following D program counts the number of times
> +a program reads from files in a ten-second period
> +and displays at most the top five results.
> +
> +```
> +/* fdscount.d -- Count read system calls for 10 seconds */
> +
> +syscall::read:entry
> +/execname == ENAME/
> +{
> + @[fds[arg0].fi_name] = count();
> +}
> +
> +profile:::tick-10sec {
> + exit(0);
> +}
> +
> +END
> +{
> + trunc(@, 5);
> +}
> +```
> +
> +We can run the script as follows:
> +
> +```
> +# dtrace -C -D ENAME='"df"' -qs fdscount.d
> +
> + libc.so.6 1
> + locale.alias 2
> + mountinfo 5
> +```
> +
> +You specify a C preprocessor directive to `dtrace` that sets
> +the value of the `ENAME` variable, such as to `df`.
> +You must use additional single quotes to escape the string quotes.
> +
> +Use the `fds[]` built-in array to determine which file
> +corresponds to the file descriptor argument `arg0` to `read()`.
> +The `fi_name` member of the `fileinfo_t` structure `fds[arg0]`
> +contains the basename of the file.
> +See
> +[`fds`](../userguide/reference/dtrace_builtin_variable_reference.md#dt_ref_var_fds)
> +and
> +[`fileinfo_t`](../userguide/reference/dtrace_providers_io.md#dt_ref_iofile_prov)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
> +for more information.
> +
> +The [`trunc()`](../userguide/reference/function_trunc.md) function
> +in the `END` action removes all but the 5 most important keys.
> +Here, there are fewer than 5 keys to start with.
> +
> +#### Exercise: Counting Context Switches on a System
> +
> +Create an executable D program named `cswpercpu.d` that displays
> +a timestamp and prints the number of context switches per CPU and
> +the total for all CPUs once per second, together with the CPU
> +number or `"total"`.
> +
> +- Using the `BEGIN` probe, print a header for the display
> + with columns labelled `Timestamp`, `CPU`, and `Ncsw`.
> +
> +- Using the `sched:::on-cpu` probe to detect the end of a context switch,
> + use `count()` to increment the aggregation variable `@n`, once with the
> + key value set to the CPU number and once with the key value set to `"total"`.
> + Since `"total"` is a string and aggregation keys must have consistent types,
> + the CPU keys are also converted to strings, using `lltostr()`.
> +
> + See
> + [Sched Provider](../userguide/reference/dtrace_providers_sched.md)
> + in the
> + [Oracle Linux: DTrace Reference Guide](../userguide/index.md)
> + for a description of the `sched:::on-cpu`
> + probe.
> +
> +- Using the `profile:::tick-1sec` probe, use `printf()` to
> + print the data and time, use `printa()` to print the key
> + (the CPU number string or `"total"`) and the aggregation value.
> + The date and time are available as the value of `walltimestamp` variable,
> + which you can print using the `%Y` conversion format
> +
> +- Use `clear()` to reset the aggregation variable `@n`.
> +
> +#### Solution to Exercise and Example: Counting Context Switches on a System
> +
> +The following example shows the executable D program `cswpercpu.d`.
> +The program displays a timestamp and prints the number of context switches,
> +per-CPU, and the total for all CPUs, once per second, together with the CPU
> +number or `"total"`:
> +
> +```
> +#!/usr/sbin/dtrace -qs
> +
> +/* cswpercpu.d -- Print number of context switches per CPU once per second */
> +
> +#pragma D option quiet
> +
> +dtrace:::BEGIN
> +{
> + /* Print the header */
> + printf("%-25s %5s %15s", "Timestamp", "CPU", "Ncsw");
> +}
> +
> +sched:::on-cpu
> +{
> + /* Convert the cpu number to a string */
> + cpustr = lltostr(cpu);
> + /* Increment the counters */
> + @n[cpustr] = count();
> + @n["total"] = count();
> +}
> +
> +profile:::tick-1sec
> +{
> + /* Print the date and time before the first result */
> + printf("\n%-25Y ", walltimestamp);
> +
> + /* Print the aggregated counts for each CPU and the total for all CPUs */
> + printa("%5s %@15d\n ", @n);
> +
> + /* Reset the aggregation */
> + clear(@n);
> +}
> +```
> +
> +```
> +# chmod +x cswpercpu.d
> +# ./cswpercpu.d
> +Timestamp CPU Ncsw
> +2013 Nov 6 20:47:26 1 148
> + 0 155
> + 3 200
> + 2 272
> + total 775
> +
> +2013 Nov 6 20:47:27 1 348
> + 0 364
> + 3 364
> + 2 417
> + total 1493
> +
> +2013 Nov 6 20:47:28 3 47
> + 1 100
> + 0 121
> + 2 178
> + total 446
> + ^C
> +```
> +
> +You might want to experiment with aggregating the total time that
> +is spent context switching and the average time per context switch.
> +For example, you can experiment by initializing a thread-local variable
> +to the value of `timestamp` in the action to a `sched:::off-cpu` probe,
> +and subtracting this value from the value of `timestamp`
> +in the action to `sched:::on-cpu`.
> +Use the `sum()` and `avg()` aggregation functions, respectively.
> +
> +### Working With More Complex Data Aggregations
> +
> +Use the `lquantize()` and `quantize()` functions to display
> +linear and power-of-two frequency distributions of data.
> +See
> +[Aggregations](../userguide/reference/aggregation.md)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
> +for a description of aggregation functions.
> +
> +#### Example: Displaying the Distribution of Read Sizes Resulting From a Command
> +
> +As shown in the following example, you can display the
> +distribution of the sizes specified to `arg2` of `read()` calls
> +that were invoked by all instances of `find` that are running.
> +After running the script, start a search with `find`
> +in another window, such as `find .` or `find /.`.
> +
> +```
> +# dtrace -n 'syscall::read:entry /execname=="find"/{@dist["find"]=quantize(arg2);}'
> +dtrace: description 'syscall::read:entry ' matched 1 probe
> +^C
> +
> + find
> + value ------------- Distribution ------------- count
> + 256 | 0
> + 512 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 6
> + 1024 | 0
> + 2048 | 0
> + 4096 |@@@@@@@@@@ 2
> + 8192 | 0
> +```
> +
> +When the program is as simple as in this example,
> +it is often convenient to run it from the command line.
> +
> +#### Example: Displaying the Distribution of I/O Throughput for Block Devices (diskact.d)
> +
> +In the following example, the `diskact.d` script uses `io`
> +provider probes to display the distribution of I/O throughput
> +for the block devices on the system.
> +
> +```
> +#pragma D option quiet
> +
> +/* diskact.d -- Display the distribution of I/O throughput for block devices */
> +
> +io:::start
> +{
> + start[args[0]->b_edev, args[0]->b_blkno] = timestamp;
> +}
> +
> +io:::done
> +/start[args[0]->b_edev, args[0]->b_blkno]/
> +{
> + /*
> + You want to get an idea of throughput to this device in KB/sec
> + but you have values that are measured in bytes and nanoseconds.
> + You want to calculate the following:
> +
> + bytes / 1024
> + ------------------------
> + nanoseconds / 1000000000
> +
> + As DTrace uses integer arithmetic and the denominator is usually
> + between 0 and 1 for most I/O, the calculation as shown will lose
> + precision. So, restate the fraction as:
> +
> + bytes 1000000000 bytes * 976562
> + ----------- * ------------- = --------------
> + nanoseconds 1024 nanoseconds
> +
> + This is easy to calculate using integer arithmetic.
> + */
> + this->elapsed = timestamp - start[args[0]->b_edev, args[0]->b_blkno];
> + @[args[1]->dev_statname, args[1]->dev_pathname] =
> + quantize((args[0]->b_bcount * 976562) / this->elapsed);
> + start[args[0]->b_edev, args[0]->b_blkno] = 0;
> +}
> +
> +END
> +{
> + printa(" %s (%s)\n%@d\n", @);
> +}
> +```
> +
> +The `#pragma D option quiet` statement is used
> +to suppress unwanted output and the `printa()`
> +function is used to display the results of the aggregation.
> +
> +See
> +[IO Provider](../userguide/reference/dtrace_providers_io.md)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
> +for a description of the arguments to the
> +`io:::start` and `io:::done` probes.
> +
> +See
> +[printa](../userguide/reference/function_printa.md)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
> +for a description of the function.
> +
> +After running the program for approximately a minute, type
> +`Ctrl-C` to display the results:
> +
> +```
> +# dtrace -s diskact.d
> + ^C
> +
> +xvda2 (<unknown>)
> +
> + value ------------- Distribution ------------- count
> + -1 | 0
> + 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 3
> + 1 | 0
> +
> +xvdc (<unknown>)
> +
> + value ------------- Distribution ------------- count
> + -1 | 0
> + 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 3
> + 1 | 0
> +
> +xvdc1 (<unknown>)
> +
> + value ------------- Distribution ------------- count
> + -1 | 0
> + 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 3
> + 1 | 0
> +
> + dm-0 (<unknown>)
> +
> + value ------------- Distribution ------------- count
> + 256 | 0
> + 512 |@@ 1
> + 1024 |@@ 1
> + 2048 |@@@@@@ 3
> + 4096 |@@@@@@@@@@ 5
> + 8192 |@@@@@@@@@@@@@@@@@ 9
> + 16384 |@@@@ 2
> + 32768 | 0
> +```
> +
> +#### Exercise: Displaying Read and Write I/O Throughput Separately
> +
> +Create a version of `diskact.d` that aggregates the results
> +separately for reading from, and writing to, block devices.
> +Use a `tick` probe to collect data for 10 seconds.
> +
> +- In the actions for `io:::start` and `io:::done`,
> + assign the value of `args[0]->b_flags & B_READ ? "READ" : "WRITE"`
> + to the clause-local variable `this->iodir`.
> +
> +- In the actions for `io:::start` and `io:::done`,
> + add `this->iodir` as a key to the `start[]` associative array.
> +
> +- In the action for `io:::done`,
> + add `this->iodir` as a key to the anonymous aggregation variable `@[]`.
> +
> +- Modify the format string for `printa()` to display the value of the `iodir` key.
> +
> +#### Solution to Exercise: Displaying Read and Write I/O Throughput Separately
> +
> +The following example shows a modified version of the `diskact.d` script,
> +which displays separate results for read and write I/O:
> +
> +```
> +#pragma D option quiet
> +
> +/* rwdiskact.d -- Modified version of diskact.d that displays
> + separate results for read and write I/O */
> +
> +profile:::tick-10sec
> +{
> + exit(0);
> +}
> +
> +io:::start
> +{
> + this->iodir = args[0]->b_flags & B_READ ? "READ" : "WRITE";
> + start[args[0]->b_edev, args[0]->b_blkno, this->iodir] = timestamp;
> +}
> +
> +io:::done
> +/
> + (this->iodir = args[0]->b_flags & B_READ ? "READ" : "WRITE") != NULL
> + &&
> + start[args[0]->b_edev, args[0]->b_blkno, this->iodir] != 0
> +/
> +{
> + this->elapsed = timestamp - start[args[0]->b_edev, args[0]->b_blkno, this->iodir];
> + @[args[1]->dev_statname, args[1]->dev_pathname, this->iodir] =
> + quantize((args[0]->b_bcount * 976562) / this->elapsed);
> + start[args[0]->b_edev, args[0]->b_blkno, this->iodir] = 0;
> +}
> +
> +END
> +{
> + printa(" %s (%s) %s \n%@d\n", @);
> +}
> +```
> +
> +In the example, adding the `this->iodir` variable to the
> +tuple in the aggregation variable enables DTrace to display
> +separate aggregations for read and write I/O operations.
> +
> +```
> +# dtrace -s rwdiskact.d
> +
> +^C
> + xvda2 (<unknown>) WRITE
> +
> + value ------------- Distribution ------------- count
> + -1 | 0
> + 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
> + 1 | 0
> +
> + xvdc (<unknown>) WRITE
> +
> + value ------------- Distribution ------------- count
> + -1 | 0
> + 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
> + 1 | 0
> +
> + xvdc1 (<unknown>) WRITE
> +
> + value ------------- Distribution ------------- count
> + -1 | 0
> + 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
> + 1 | 0
> +
> + nfs (<nfs>) READ
> +
> + value ------------- Distribution ------------- count
> + -1 | 0
> + 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 5
> + 1 | 0
> +
> + dm-0 (<unknown>) WRITE
> +
> + value ------------- Distribution ------------- count
> + 4096 | 0
> + 8192 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
> + 16384 | 0
> +```
> +
> +#### Example: Displaying Cumulative Read and Write Activity Across a File System Device (fsact)
> +
> +The following example is a `bash` shell script that uses
> +an embedded D program to display cumulative read and write
> +block counts for a local file system according to their
> +location on the file system's underlying block device.
> +The `lquantize()` aggregation function is used to
> +display the results linearly as tenths of the total
> +number of blocks on the device.
> +
> +```
> +#!/bin/bash
> +
> +# fsact -- Display cumulative read and write activity across a file system device
> +#
> +# Usage: fsact [<filesystem>]
> +
> +# If no file system is specified, assume /
> +[ $# -eq 1 ] && FSNAME=$1 || FSNAME="/"
> +[ ! -e $FSNAME ] && echo "$FSNAME not found" && exit 1
> +
> +# Determine the mountpoint, major and minor numbers, and file system size
> +MNTPNT=$(df $FSNAME | gawk '{ getline; print $1; exit }')
> +MAJOR=$(printf "%d\n" 0x$(stat -Lc "%t" $MNTPNT))
> +MINOR=$(printf "%d\n" 0x$(stat -Lc "%T" $MNTPNT))
> +FSSIZE=$(stat -fc "%b" $FSNAME)
> +
> +# Run the embedded D program
> +dtrace -qs /dev/stdin << EOF
> +io:::done
> +/args[1]->dev_major == $MAJOR && args[1]->dev_minor == $MINOR/
> +{
> + this->iodir = args[0]->b_flags & B_READ ? "READ" : "WRITE";
> + /* Normalize the block number as an integer in the range 0 to 10 */
> + blkno = (args[0]->b_blkno)*10/$FSSIZE;
> + /* Aggregate blkno linearly over the range 0 to 10 in steps of 1 */
> + @a[this->iodir] = lquantize(blkno,0,10,1)
> +}
> +
> +tick-10sec
> +{
> + printf("%Y\n",walltimestamp);
> + /* Display the results of the aggregation */
> + printa("%s\n%@d\n", at a);
> + /* To reset the aggregation every tick, uncomment the following line */
> + /* clear(@a); */
> +}
> +EOF
> +```
> +
> +You embed the D program in a shell script so that you can
> +set up the parameters that are needed, which are the major
> +and minor numbers of the underlying device and the total
> +size of the file system in file system blocks.
> +You then access these parameters directly in the D code.
> +
> +>Note:
> +>
> +>An alternate way of passing values into the D program is to
> +use C preprocessor directives, for example:
> +
> +```
> +dtrace -C -D MAJ=$MAJOR -D MIN=$MINOR -D FSZ=$FSSIZE -qs /dev/stdin << EOF
> +```
> +
> +You can then refer to the variables in the D program by their
> +macro names instead of their shell names:
> +
> +```
> +/args[1]->dev_major == MAJ && args[1]->dev_minor == MIN/
> +
> +blkno = (args[0]->b_blkno)*10/FSZ;
> +```
> +
> +The following example shows output from running the `fsact`
> +command after making the script executable,
> +then running `cp -R` on a directory and `rm -rf` on the copied directory:
> +
> +```
> +# chmod +x fsact
> +# ./fsact
> +2018 Feb 16 16:59:46
> +READ
> +
> + value ------------- Distribution ------------- count
> + < 0 | 0
> + 0 |@@@@@@@ 8
> + 1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 32
> + 2 | 0
> + 3 | 0
> + 4 | 0
> + 5 | 0
> + 6 | 0
> + 7 | 0
> + 8 | 0
> + 9 | 0
> + >= 10 |@@@@@@@ 8
> +
> +WRITE
> +
> + value ------------- Distribution ------------- count
> + 9 | 0
> + >= 10 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 42 0
> +
> +^C
> +```
> +
> +### Displaying System Call Errors
> +
> +The following information pertains to using the D program
> +`errno.d` to display system call errors.
> +
> +#### Example: Displaying System Call Errors (`errno.d`)
> +
> +The following is an example of the D program, `errno.d`.
> +In this example, the program displays the value of `errno`
> +and the file name if an error occurs when using the `open()`
> +system call to open a file.
> +
> +```
> +#!/usr/sbin/dtrace -qs
> +
> +/* errno.d -- Display errno and the file name for failed open() calls */
> +
> +syscall::open:entry
> +{
> + self->filename = copyinstr(arg0);
> +}
> +
> +syscall::openat*:entry
> +{
> + self->filename = copyinstr(arg1);
> +}
> +
> +syscall::open*:return
> +/arg0 < 0 && self->filename != NULL/
> +{
> + printf("errno = %-2d file = %s\n", errno, self->filename);
> +}
> +
> +syscall::open*:return
> +{
> + self->filename = NULL;
> +}
> +```
> +
> +If an error occurs in the `open()` system call, the `return` probe
> +sets the `arg0` argument to `-1` and the value of the built-in
> +`errno` variable indicates the nature of the error.
> +A predicate is used to test the value of `arg0`.
> +Alternatively, you could test whether the value of `errno` is greater than zero.
> +
> +The dynamic variable `self->filename` should be freed (set to 0,
> +or in this case `NULL`) as a habit to reduce memory leaks.
> +
> +When you have saved this script to a file and made the file executable,
> +you can then run it to display information about any failures of the
> +`open()` system call that occur on the system.
> +After you have started the script, in a separate terminal window,
> +you can run commands that result in an error, such as running the
> +`ls` command to list a file that does not exist.
> +Or, as in the following example, from another terminal the `cat`
> +command has been issued on a directory, which results in an error:
> +
> +```
> +# ./errno.d
> +
> +errno = 2 file = /usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo
> +errno = 2 file = /usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo
> +errno = 2 file = /usr/share/locale/en_US/LC_MESSAGES/libc.mo
> +errno = 2 file = /usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo
> +errno = 2 file = /usr/share/locale/en.utf8/LC_MESSAGES/libc.mo
> +errno = 2 file = /usr/share/locale/en/LC_MESSAGES/libc.mo
> +^C
> +```
> +
> +#### Exercise: Displaying More Information About System Call Errors
> +
> +Adapt `errno.d` to display the name of the
> +error instead of its number for any failed system call.
> +
> +- The numeric values of errors such as `EACCES` and `EEXIST`
> + are defined in `/usr/include/asm-generic/errno-base.h` and
> + `/usr/include/asm-generic/errno.h`.
> + DTrace defines inline names (which are effectively constants)
> + for the numeric error values in `/usr/lib64/dtrace/*/errno.d`.
> + Use an associative array named `error[]` to store the mapping
> + between the inline names and the error names that are defined
> + in `/usr/include/asm-generic/errno-base.h`.
> +
> +- Use `printf()` to display the user ID, the process ID, the
> + program name, the error name, and the name of the system call.
> +
> +- Use the `BEGIN` probe to print column headings.
> +
> +- Use the value of `errno` rather than `arg0` to test whether an
> + error from the range of mapped names has occurred in a system call.
> +
> +#### Solution to Exercise: Displaying More Information About System Call Errors
> +
> +The following is an example that shows a modified version of `errno.d`,
> +which displays error names.
> +
> +##### Example: Modified Version of errno.d Displaying Error Names (displayerrno.d)
> +
> +```
> +#!/usr/sbin/dtrace -qs
> +
> +/* displayerrno.d -- Modified version of errno.d that displays error names */
> +
> +BEGIN
> +{
> + printf("%-4s %-6s %-10s %-10s %s\n", "UID", "PID", "Prog", "Error", "Func");
> +
> + /* Assign error names to the associative array error[] */
> + error[EPERM] = "EPERM"; /* Operation not permitted */
> + error[ENOENT] = "ENOENT"; /* No such file or directory */
> + error[ESRCH] = "ESRCH"; /* No such process */
> + error[EINTR] = "EINTR"; /* Interrupted system call */
> + error[EIO] = "EIO"; /* I/O error */
> + error[ENXIO] = "ENXIO"; /* No such device or address */
> + error[E2BIG] = "E2BIG"; /* Argument list too long */
> + error[ENOEXEC] = "ENOEXEC"; /* Exec format error */
> + error[EBADF] = "EBADF"; /* Bad file number */
> + error[ECHILD] = "ECHILD"; /* No child processes */
> + error[EAGAIN] = "EAGAIN"; /* Try again or operation would block */
> + error[ENOMEM] = "ENOMEM"; /* Out of memory */
> + error[EACCES] = "EACCES"; /* Permission denied */
> + error[EFAULT] = "EFAULT"; /* Bad address */
> + error[ENOTBLK] = "ENOTBLK"; /* Block device required */
> + error[EBUSY] = "EBUSY"; /* Device or resource busy */
> + error[EEXIST] = "EEXIST"; /* File exists */
> + error[EXDEV] = "EXDEV"; /* Cross-device link */
> + error[ENODEV] = "ENODEV"; /* No such device */
> + error[ENOTDIR] = "ENOTDIR"; /* Not a directory */
> + error[EISDIR] = "EISDIR"; /* Is a directory */
> + error[EINVAL] = "EINVAL"; /* Invalid argument */
> + error[ENFILE] = "ENFILE"; /* File table overflow */
> + error[EMFILE] = "EMFILE"; /* Too many open files */
> + error[ENOTTY] = "ENOTTY"; /* Not a typewriter */
> + error[ETXTBSY] = "ETXTBSY"; /* Text file busy */
> + error[EFBIG] = "EFBIG"; /* File too large */
> + error[ENOSPC] = "ENOSPC"; /* No space left on device */
> + error[ESPIPE] = "ESPIPE"; /* Illegal seek */
> + error[EROFS] = "EROFS"; /* Read-only file system */
> + error[EMLINK] = "EMLINK"; /* Too many links */
> + error[EPIPE] = "EPIPE"; /* Broken pipe */
> + error[EDOM] = "EDOM"; /* Math argument out of domain of func */
> + error[ERANGE] = "ERANGE"; /* Math result not representable */
> +}
> +
> +/* Specify any syscall return probe and test that the value of errno is in range */
> +
> +syscall:::return
> +/errno > 0 && errno <= ERANGE/
> +{
> + printf("%-4d %-6d %-10s %-10s %s()\n", uid, pid, execname, error[errno], probefunc);
> +}
> +```
> +
> +```
> +# chmod +x displayerrno.d
> +# ./displayerrno.d
> +UID PID Prog Error Func
> +500 3575 test EACCES open()
> +500 3575 test EINTR clock_gettime()
> +^C
> +```
> +
> +You could modify this program so that it displays verbose
> +information about the nature of the error, in addition to the
> +name of the error.
> diff --git a/doc/tutorial/3.TracingUserSpaceApplications.md b/doc/tutorial/3.TracingUserSpaceApplications.md
> new file mode 100644
> index 000000000..1d9bf69f2
> --- /dev/null
> +++ b/doc/tutorial/3.TracingUserSpaceApplications.md
> @@ -0,0 +1,832 @@
> +
> +## Tracing User-Space Applications
> +
> +This chapter provides information about how to trace a user-space
> +application and includes examples of D programs that you can use
> +to investigate what is happening in an example user-space program.
> +
> +### Sample Application
> +
> +This section provides a sample application to be used in
> +subsequent exercises and examples in this chapter. The example,
> +which illustrates a simple program, favors brevity and probing
> +opportunity rather than completeness or efficiency.
> +
> +>Note:
> +>
> +>The following simple program is provided for example purposes
> +*only* and is not intended to efficiently solve a practical
> +problem nor exhibit preferred coding methods.
> +
> +The sample program finds the lowest factor of a number, which you input.
> +The program is comprised of the following four files:
> +`makefile`, `primelib.h`, `primelib.c`, and `primain.c`,
> +which are stored in the same working directory.
> +
> +#### Description and Format of the makefile File
> +
> +The following example shows the contents of the `makefile` file.
> +
> +>Note:
> +>
> +>A `makefile` must use tabs for indentation so
> +that the `make` command can function properly.
> +Also, be sure that tabs are retained if the
> +file is copied and then used.
> +
> +```
> +default: prime
> +
> +# compile the library primelib first
> +primelib.o: primelib.c
> + gcc -c primelib.c
> +
> +# compile the main program
> +primain.o: primain.c
> + gcc -c primain.c
> +
> +# link and create executable file "prime"
> +prime: primelib.o primain.o
> + gcc primain.o primelib.o -o prime -lm
> +
> +clean:
> + -rm -f *.o
> + -rm -f prime
> +```
> +
> +#### Description of the primelib.h Source File
> +
> +The following example shows the contents of the `primelib.h` file.
> +
> +```
> +int findMaxCheck( int inValue );
> +int seekFactorA( int input, int maxtry );
> +int seekFactorB( int input );
> +```
> +
> +#### Description of the primelib.c Source File
> +
> +The following example shows the contents of the `primelib.c` file.
> +
> +```
> +#include <stdio.h>
> +#include <math.h>
> +
> +/*
> + * utility functions which are called from the main source code
> + */
> +
> +// Find and return our highest value to check -- which is the square root
> +int findMaxCheck( int inValue ) {
> + float sqRoot;
> + sqRoot = sqrt( inValue );
> + printf("Square root of %d is %lf\n", inValue, sqRoot);
> + return floor( sqRoot );
> +}
> +
> +int debugFlag = 0;
> +
> +// Search for a factor to the input value, proving prime on return of zero
> +int seekFactorA( int input, int maxtry ) {
> + int divisor, factor = 0;
> + for( divisor=2; divisor<=maxtry; ++divisor ) {
> + if( 0 == input%divisor ) {
> + factor = divisor;
> + break;
> + }
> + else if ( debugFlag != 0 )
> + printf( "modulo %d yields: %d\n", divisor, input%divisor );
> + }
> + return factor;
> +}
> +
> +// Search for a factor to the input value, proving prime on return of zero
> +// This is a different method than "A", using one argument
> +int seekFactorB( int input ) {
> + int divisor, factor = 0;
> + if( 0 == input%2 ) return 2;
> + for( divisor=3; divisor<=input/2; divisor+=2 ) {
> + if( 0 == input%divisor ) {
> + factor = divisor;
> + break;
> + }
> + }
> + return factor;
> +}
> +```
> +
> +#### Description of the primain.c Source File
> +
> +The following example shows the contents of the `primain.c` file.
> +
> +```
> +#include <stdio.h>
> +#include "primelib.h"
> +
> +/*
> + * Nominal C program churning to provide a code base we might want to
> + * instrument with D
> + */
> +
> +// Search for a divisor -- thereby proving composite value of the input.
> +int main() {
> + int targVal, divisor, factorA=0, factorB=0;
> +
> + printf( "Enter a positive target integer to test for prime status: " );
> + scanf( "%d", &targVal );
> +
> + // Check that the user input is valid
> + if( targVal < 2 ) {
> + printf( "Invalid input value -- exiting now\n" );
> + return -2;
> + }
> +
> + // Search for a divisor using method and function A
> + int lastCheck;
> + lastCheck = findMaxCheck( targVal );
> + printf( "%d highest value to check as divisor\n", lastCheck );
> + factorA = seekFactorA( targVal, lastCheck );
> +
> + // Search for a divisor using method and function B
> + factorB = seekFactorB( targVal );
> +
> + // Warn if the methods give different results
> + if (factorA != factorB)
> + printf( "%d does not equal %d! How can this be?\n", factorA, factorB );
> +
> + // Print results
> + if( !factorA )
> + printf( "%d is a prime number\n", targVal );
> + else
> + printf( "%d is not prime because there is a factor %d\n",
> + targVal, factorA );
> + return 0;
> +}
> +```
> +
> +#### Compiling the Program and Running the prime Executable
> +
> +With the four files previously described located in the
> +same working directory, compile the program by using the
> +`make` command as follows:
> +
> +```
> +# make
> +gcc -c primelib.c
> +gcc -c primain.c
> +gcc primain.o primelib.o -o prime -lm
> +```
> +
> +Running the `make` command creates an executable named `prime`,
> +which can be run to find the lowest prime value of the input,
> +as shown in the following two examples:
> +
> +```
> +# ./prime
> +Enter a positive target integer to test for prime status: 5099
> +Square root of 5099 is 71.407280
> +71 highest value to check as divisor
> +5099 is a prime number
> +```
> +```
> +# ./prime
> +Enter a positive target integer to test for prime status: 95099
> +Square root of 95099 is 308.381256
> +308 highest value to check as divisor
> +95099 is not prime because there is a factor 61
> +```
> +
> +### Using the `pid` Provider
> +
> +So far, we have built our application without
> +taking DTrace at all into account.
> +Nevertheless, we can still study execution of this application,
> +using the `pid` provider to trace entry into and return from user functions.
> +
> +For example, from one window, start the application
> +but do not yet provide it with any input,
> +thereby leaving the application paused and waiting:
> +
> +```
> +$ ./prime
> +Enter a positive target integer to test for prime status:
> +```
> +
> +In another window, we can list the probes supplied by the `pid` provider.
> +Actually, there is a different such provider for each process,
> +meaning that we must append the process ID to specify the process to trace.
> +For example, we can run `pgrep prime` to get the process ID.
> +In our case, it is 2889188.
> +Then, we can list the `pid` probes associated with this process in either of two ways:
> +
> +```
> +# dtrace -ln pid2889188:a.out::
> +# dtrace -p 2889188 -ln pid'$target':a.out::
> +```
> +
> +In the first case, we append the process ID
> +explicitly to form the provider name `pid2889188`.
> +In the second case, the provider name refers to the process ID symbolically,
> +using the macro `$target`,
> +and the traced process 2889188 is specified using a
> +`dtrace` command-line option `-p`.
> +This second form is useful when the probe specification is contained in
> +a D script that we do not want to modify each time we use it.
> +
> +We specify a module of the application executable.
> +It could be the load object of the executable, here referred to as `a.out`,
> +or it could be a shared library.
> +The probe function and name are omitted, equivalent to the wildcard `*`.
> +
> +In our case, this probe specification results in 187 probes.
> +We show a few of them here:
> +
> +```
> + ID PROVIDER MODULE FUNCTION NAME
> + 5433 pid2889188 prime seekFactorB 56
> + 5431 pid2889188 prime seekFactorB 55
> + 5429 pid2889188 prime seekFactorB 52
> + [...]
> + 5375 pid2889188 prime seekFactorB 4
> + 5373 pid2889188 prime seekFactorB 1
> + 5371 pid2889188 prime seekFactorB 0
> + 5370 pid2889188 prime seekFactorB entry
> + 5368 pid2889188 prime seekFactorB return
> + 5366 pid2889188 prime seekFactorA 68
> + 5364 pid2889188 prime seekFactorA 67
> + 5362 pid2889188 prime seekFactorA 64
> + [...]
> + 5302 pid2889188 prime seekFactorA 4
> + 5300 pid2889188 prime seekFactorA 1
> + 5298 pid2889188 prime seekFactorA 0
> + 5297 pid2889188 prime seekFactorA entry
> + 5295 pid2889188 prime seekFactorA return
> + 5293 pid2889188 prime findMaxCheck 70
> + 5291 pid2889188 prime findMaxCheck 6f
> + 5289 pid2889188 prime findMaxCheck 6b
> + [...]
> + 5243 pid2889188 prime findMaxCheck 4
> + 5241 pid2889188 prime findMaxCheck 1
> + 5239 pid2889188 prime findMaxCheck 0
> + 5238 pid2889188 prime findMaxCheck entry
> + 5236 pid2889188 prime findMaxCheck return
> + 5234 pid2889188 prime main ef
> + 5232 pid2889188 prime main ee
> + 5230 pid2889188 prime main e9
> + [...]
> + 5110 pid2889188 prime main 4
> + 5108 pid2889188 prime main 1
> + 5106 pid2889188 prime main 0
> + 5105 pid2889188 prime main entry
> + 5103 pid2889188 prime main return
> + [...]
> +```
> +
> +The module, which we specified by the nickname `a.out`,
> +is listed by its executable name `prime`.
> +The functions of this module appear: `seekFactorB`, `seekFactorA`, and so on.
> +For each function, there is a probe for entry, return, and each instruction offset.
> +
> +This listing shows us what probes we might specify to trace this
> +user program without having added any DTrace-specific instrumentation.
> +For more information on how to use the `pid` provider to trace
> +user function entry and return or on specific instruction offsets,
> +see
> +[Pid Provider](../userguide/reference/dtrace_providers_pid.md)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
> +
> +### Adding USDT Probes to an Application
> +
> +In this section, we practice adding USDT probes to an application.
> +For background information and other details, see
> +[Adding USDT Probes to Application Code](../userguide/reference/dtrace-ref-StaticallyDefinedTracingofUserApplications.md#dt_ref_usdt_probe_add_prov)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
> +
> +To get started, you will need to create a `.d` file, as described in
> +[Defining USDT Providers and Probes](../userguide/reference/dtrace-ref-StaticallyDefinedTracingofUserApplications.md#dt_ref_usdtprobes_prov)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
> +
> +>Note:
> +>
> +>This `.d` file is not a script that is run in the same
> +way that is shown in previous examples in this tutorial,
> +but is rather the `.d` source file that you use when
> +compiling and linking your application.
> +
> +In this `.d` file, you define the probes you will
> +place in the `primain.c` source file.
> +These probes mark the sequence of operations that are
> +used after the user entry is completed and checked:
> +
> +| Description | Probe |
> +| :----------- | :----------- |
> +| User entry complete and checked | `userentry( int )` |
> +| Return from `findMaxCheck()` | `maxcheckval( int, int )` |
> +| Return from `seekFactorA()` | `factorreturnA( int, int )` |
> +| Return from `seekFactorB()` | `factorreturnB( int, int )` |
> +| Immediately prior to the program exiting | `final()` |
> +
> +#### Exercise: Creating a dprime.d File
> +
> +Declare the above probes in a file `dprime.d` and store the
> +file in the same working directory as the other source files.
> +
> +#### Solution to Exercise: Creating a dprime.d File
> +
> +```
> +provider primeget
> +{
> + probe query__userentry( int );
> + probe query__maxcheckval( int, int );
> + probe query__factorreturnA( int, int );
> + probe query__factorreturnB( int, int );
> + probe query__final();
> +};
> +```
> +
> +#### Example: Creating a .h File From a dprime.d File
> +
> +The next step is to create a `.h` header file to use in your
> +C source program from the `dprime.d` file, as shown here:
> +
> +```
> +# dtrace -h -s dprime.d
> +```
> +
> +The `dprime.h` file that is created contains a reference to each
> +of the probe points that are defined in the `dprime.d` file.
> +
> +Next, in the application source file, `primain.c`, we add
> +`#include "dprime.h"` and the appropriate probe macros at the proper
> +locations.
> +
> +In the resulting `primain.c` file, the probe macros are easy
> +to recognize, as they appear in uppercase letters:
> +
> +```
> +#include <stdio.h>
> +#include "primelib.h"
> +#include "dprime.h"
> +
> +/*
> + * Nominal C program churning to provide a code base we might want to
> + * instrument with D
> +*/
> +
> +// Search for a divisor -- thereby proving composite value of the input.
> +int main() {
> + int targVal, divisor, factorA=0, factorB=0;
> +
> + printf( "Enter a positive target integer to test for prime status: " );
> + scanf( "%d", &targVal );
> +
> + // Check that the user input is valid
> + if( targVal < 2 ) {
> + printf( "Invalid input value -- exiting now\n" );
> + return -2;
> + }
> + if (PRIMEGET_QUERY_USERENTRY_ENABLED())
> + PRIMEGET_QUERY_USERENTRY(targVal);
> +
> + // Search for a divisor using method and function A
> + int lastCheck;
> + lastCheck = findMaxCheck( targVal );
> + printf( "%d highest value to check as divisor\n", lastCheck );
> + if (PRIMEGET_QUERY_MAXCHECKVAL_ENABLED())
> + PRIMEGET_QUERY_MAXCHECKVAL(lastCheck, targVal);
> +
> + factorA = seekFactorA( targVal, lastCheck );
> + if (PRIMEGET_QUERY_FACTORRETURNA_ENABLED())
> + PRIMEGET_QUERY_FACTORRETURNA(factorA, targVal);
> +
> + // Search for a divisor using method and function B
> + factorB = seekFactorB( targVal );
> + if (PRIMEGET_QUERY_FACTORRETURNB_ENABLED())
> + PRIMEGET_QUERY_FACTORRETURNB(factorB, targVal);
> +
> + // Warn if the methods give different results
> + if (factorA != factorB)
> + printf( "%d does not equal %d! How can this be?\n", factorA, factorB );
> +
> + // Print results
> + if( !factorA )
> + printf( "%d is a prime number\n", targVal );
> + else
> + printf( "%d is not prime because there is a factor %d\n",
> + targVal, factorA );
> + if (PRIMEGET_QUERY_FINAL_ENABLED())
> + PRIMEGET_QUERY_FINAL();
> +
> + return 0;
> +}
> +```
> +
> +Any `*_ENABLED()` probe will translate into a truth value if
> +the associated probe is enabled (some consumer is using it),
> +and a false value if the associated probe is not enabled.
> +
> +Next, you will need to modify the `makefile` file.
> +For step-by-step instructions, See
> +[Building Applications With USDT Probes](../userguide/reference/dtrace-ref-StaticallyDefinedTracingofUserApplications.md#dt_ref_usdt_build_prov)
> +in the
> +[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
> +
> +#### Exercise: Directing makefile to Re-Create the dprime.h File
> +
> +Add a target that instructs `dtrace` to re-create the `dprime.h` file
> +in the event that changes are subsequently made to the `dprime.d` file.
> +This step ensures that you do not have to manually run the
> +`dtrace -h -s dprime.d` command if any changes are made.
> +
> +This exercise also has you direct `dtrace` to create a `prime.o` file.
> +
> +#### Solution to Exercise: Directing makefile to Re-Create the dprime.h File
> +
> +```
> +default: prime
> +
> +# re-create new dprime.h if dprime.d file has been changed
> +dprime.h: dprime.d
> + dtrace -h -s dprime.d
> +
> +# compile the library primelib first
> +primelib.o: primelib.c
> + gcc -c primelib.c
> +
> +# compile the main program
> +primain.o: primain.c dprime.h
> + gcc -c -I/usr/lib64/dtrace/include primain.c
> +
> +# have dtrace post-process the object files
> +prime.o: dprime.d primelib.o primain.o
> + dtrace -G -s dprime.d primelib.o primain.o -o prime.o
> +
> +# link and create executable file "prime"
> +prime: prime.o
> + gcc -Wl,--export-dynamic,--strip-all -o prime prime.o primelib.o primain.o -lm
> +
> +clean:
> + -rm -f *.o
> + -rm -f prime
> + -rm -f dprime.h
> +```
> +
> +Notice that the `primain.c` file now includes the generated
> +`dprime.h` file, which includes `<sys/usdt.h>`.
> +Therefore, when we compile `gcc -c primain.c`,
> +we include a `-I` option to find `<sys/usdt.h>`.
> +The path to use on your system is given by:
> +
> +```
> +pkg-config --cflags dtrace_sdt
> +```
> +
> +which is `/usr/lib64/dtrace/include` in the example above.
> +In the DTrace source code directory, the file is in `uts/common`.
> +
> +In the link stage, the `-Wl,--export-dynamic` link options to `gcc`
> +are required for symbol lookup in a stripped executable at runtime,
> +for example, when you use the D function `ustack()`.
> +
> +#### Example: Testing the Program
> +
> +After creating a fresh build, test that the executable is still
> +working as expected:
> +
> +```
> +# make clean
> +rm -f *.o
> +rm -f prime
> +rm -f dprime.h
> +# make
> +gcc -c primelib.c
> +dtrace -h -s dprime.d
> +gcc -c primain.c
> +dtrace -G -s dprime.d primelib.o primain.o -o prime.o
> +gcc -Wl,--export-dynamic,--strip-all -o prime prime.o primelib.o primain.o dprime.h -lm
> +```
> +
> +```
> +# ./prime
> +Enter a positive target integer to test for prime status: 6799
> +Square root of 6799 is 82.456047
> +82 highest value to check as divisor
> +6799 is not prime because there is a factor 13
> +```
> +
> +### Using USDT Probes
> +
> +This section provides some practice in the nominal use of the USDT
> +probes that were created in
> +[Adding USDT Probes to an Application](#adding-usdt-probes-to-an-application).
> +
> +Initially, the probes are not visible because the application is
> +not running with the probes, as shown in the following output:
> +
> +```
> +# dtrace -l -P 'prime*'
> + ID PROVIDER MODULE FUNCTION NAME
> +dtrace: failed to match prime*:::: No probe matches description
> +```
> +
> +Start the application, but do not enter any value yet:
> +
> +```
> +# ./prime
> +Enter a positive target integer to test for prime status:
> +```
> +
> +From another command line, issue a probe listing:
> +
> +```
> +# dtrace -l -P 'prime*'
> + ID PROVIDER MODULE FUNCTION NAME
> + 2475 primeget26556 prime main query-factorreturnA
> + 2476 primeget26556 prime main query-factorreturnB
> + 2477 primeget26556 prime main query-final
> + 2478 primeget26556 prime main query-maxcheckval
> + 2479 primeget26556 prime main query-userentry
> +```
> +
> +Note that the provider name is a combination of the defined `provider primeget`,
> +from the `dprime.d` file, and the PID of the running application `prime`.
> +The output of the following command displays the PID of prime:
> +
> +```
> +# ps aux | grep prime
> +root 26556 0.0 0.0 7404 1692 pts/0 S+ 21:50 0:00 ./prime
> +```
> +
> +At this point, we can kill the application or provide an input
> +value so that the application runs to completion.
> +
> +#### Example: Using simpleTimeProbe.d to Show the Elapsed Time Between Two Probes
> +
> +The following example shows how you would create a simple script
> +that measures the time elapsed between the first probe and the
> +second probe (`query-userentry` to `query-maxcheckval`).
> +
> +```
> +/* simpleTimeProbe.d */
> +
> +/* Show how much time elapses between two probes */
> +
> +primeget*:::query-userentry
> +{
> + self->t = timestamp; /* Initialize a thread-local variable with the time */
> +}
> +
> +primeget*:::query-maxcheckval
> +/self->t != 0/
> +{
> + this->timeNow = timestamp;
> + /* Divide by 1000 for microseconds */
> + printf("%s (pid=%d) spent %d microseconds between userentry & maxcheckval\n",
> + execname, pid, ((this->timeNow - self->t)/1000));
> +
> + self->t = 0; /* Reset the variable */
> +}
> +```
> +
> +Since `timeNow` is used only within this clause,
> +we use `this->` to denote its clause-local scope.
> +
> +Once again, start the execution of the target application:
> +
> +```
> +# ./prime
> +Enter a positive target integer to test for prime status:
> +```
> +
> +Then, run the DTrace script from another window:
> +
> +```
> +# dtrace -q -s simpleTimeProbe.d
> +```
> +
> +As the application is running, the output of the script is also
> +running in parallel:
> +
> +```
> +# ./prime
> +Enter a positive target integer to test for prime status: 7921
> +Square root of 7921 is 89.000000
> +89 highest value to check as divisor
> +7921 is not prime because there is a factor 89
> +# ./prime
> +Enter a positive target integer to test for prime status: 995099
> +Square root of 995099 is 997.546509
> +997 highest value to check as divisor
> +995099 is not prime because there is a factor 7
> +# ./prime
> +Enter a positive target integer to test for prime status: 7921
> +Square root of 7921 is 89.000000
> +89 highest value to check as divisor
> +7921 is not prime because there is a factor 89
> +```
> +
> +On the command line where the script is being run, you should
> +see output similar to the following:
> +
> +```
> +# dtrace -q -s simpleTimeProbe.d
> +prime (pid=2328) spent 45 microseconds between userentry & maxcheckval
> +prime (pid=2330) spent 41 microseconds between userentry & maxcheckval
> +prime (pid=2331) spent 89 microseconds between userentry & maxcheckval
> +^C
> +```
> +
> +Another way of running the D script is with the `-Z` option.
> +Previously, we started the application first so that some
> +`primeget*` probes would be found.
> +With the `-Z` option, `dtrace` will start up even if
> +zero matching probes are found.
> +Then, it will wait patiently until such probes are discovered
> +and those probes fire. That is, in one window, run:
> +
> +```
> +# dtrace -q -Z -s simpleTimeProbe.d
> +```
> +
> +Note the addition of the `-Z` option.
> +In other window, run the application:
> +
> +```
> +# ./prime
> +Enter a positive target integer to test for prime status: 7921
> +Square root of 7921 is 89.000000
> +89 highest value to check as divisor
> +7921 is not prime because there is a factor 89
> +#
> +```
> +
> +The first window should show the `dtrace` output
> +and you can terminate the script.
> +
> +```
> +# dtrace -q -Z -s simpleTimeProbe.d
> +prime (pid=4334) spent 85 microseconds between userentry & maxcheckval
> +^C
> +```
> +
> +And yet one more way of launching both the D script and the
> +application, from the same window, is to use the `-c` option
> +to `dtrace` to launch a command to trace:
> +
> +```
> +# dtrace -q -c ./prime -s simpleTimeProbe.d
> +Enter a positive target integer to test for prime status: 579
> +Square root of 579 is 24.062418
> +24 highest value to check as divisor
> +579 is not prime because there is a factor 3
> +prime (pid=2884516) spent 98 microseconds between userentry & maxcheckval
> +```
> +
> +Here, `dtrace` is launched, it starts the command `./prime`,
> +and when `dtrace` or the application terminates, so does the other.
> +
> +#### Example: Using timeTweenprobes.d to Show the Elapsed Time Between Each Probe
> +
> +You can broaden the script to monitor all of the following
> +probes in the application:
> +
> +- `query-userentry`
> +- `query-maxcheckval`
> +- `query-factorreturnA`
> +- `query-factorreturnB`
> +- `query-final`
> +
> +```
> +/* timeTweenProbes.d */
> +
> +/* show how much time elapses between each probe */
> +
> +BEGIN
> +{
> + iterationCount = 0;
> +}
> +
> +primeget*:::query-userentry
> +{
> + printf("%s (pid=%d) running\n", execname, pid);
> + self->t = timestamp; /* Initialize a thread-local variable with time */
> +}
> +
> +primeget*:::query-maxcheckval
> +/self->t != 0/
> +{
> + this->timeNow = timestamp;
> + printf(" maxcheckval spent %d microseconds since userentry\n",
> + ((this->timeNow - self->t)/1000)); /* Divide by 1000 for microseconds */
> + self->t = this->timeNow; /* set the time to recent sample */
> +}
> +
> +primeget*:::query-factorreturnA
> +/self->t != 0/
> +{
> + this->timeNow = timestamp;
> + printf(" factorreturnA spent %d microseconds since maxcheckval\n",
> + ((this->timeNow - self->t)/1000)); /* Divide by 1000 for microseconds */
> + self->t = this->timeNow; /* set the time to recent sample */
> +}
> +
> +primeget*:::query-factorreturnB
> +/self->t != 0/
> +{
> + this->timeNow = timestamp;
> + printf(" factorreturnB spent %d microseconds since factorreturnA\n",
> + ((this->timeNow - self->t)/1000)); /* Divide by 1000 for microseconds */
> + self->t = this->timeNow; /* set the time to recent sample */
> +}
> +
> +primeget*:::query-final
> +/self->t != 0/
> +{
> + printf(" prime spent %d microseconds from factorreturnB until ending\n",
> + ((timestamp - self->t)/1000));
> + self->t = 0; /* Reset the variable */
> + iterationCount++;
> +}
> +
> +END
> +{
> + trace(iterationCount);
> +}
> +```
> +
> +Again, start the execution of the target application first,
> +without yet providing input.
> +Then from another window, start the D script.
> +Back in the first window, you can run the application multiple times.
> +Finally, in the second window, you can terminate the D script.
> +The first window might look something like this:
> +
> +```
> +# ./prime
> +Enter a positive target integer to test for prime status: 995099
> +Square root of 995099 is 997.546509
> +997 highest value to check as divisor
> +995099 is not prime because there is a factor 7
> +# ./prime
> +Enter a positive target integer to test for prime status: 7921
> +Square root of 7921 is 89.000000
> +89 highest value to check as divisor
> +7921 is not prime because there is a factor 89
> +# ./prime
> +Enter a positive target integer to test for prime status: 95099
> +Square root of 95099 is 308.381256
> +308 highest value to check as divisor
> +95099 is not prime because there is a factor 61
> +# ./prime
> +Enter a positive target integer to test for prime status: 95099
> +Square root of 95099 is 308.381256
> +308 highest value to check as divisor
> +95099 is not prime because there is a factor 61
> +# ./prime
> +Enter a positive target integer to test for prime status: 5099
> +Square root of 5099 is 71.407280
> +71 highest value to check as divisor
> +5099 is a prime number
> +```
> +
> +The corresponding output from the script is similar to the
> +following:
> +
> +```
> +# dtrace -q -s ./timeTweenProbes.d
> +prime (pid=2437) running
> + maxcheckval spent 96 microseconds since userentry
> + factorreturnA spent 9 microseconds since maxcheckval
> + factorreturnB spent 6 microseconds since factorreturnA
> + prime spent 9 microseconds from factorreturnB until ending
> +prime (pid=2439) running
> + maxcheckval spent 45 microseconds since userentry
> + factorreturnA spent 10 microseconds since maxcheckval
> + factorreturnB spent 7 microseconds since factorreturnA
> + prime spent 9 microseconds from factorreturnB until ending
> +prime (pid=2440) running
> + maxcheckval spent 43 microseconds since userentry
> + factorreturnA spent 11 microseconds since maxcheckval
> + factorreturnB spent 8 microseconds since factorreturnA
> + prime spent 10 microseconds from factorreturnB until ending
> +prime (pid=2441) running
> + maxcheckval spent 53 microseconds since userentry
> + factorreturnA spent 10 microseconds since maxcheckval
> + factorreturnB spent 7 microseconds since factorreturnA
> + prime spent 10 microseconds from factorreturnB until ending
> +prime (pid=2442) running
> + maxcheckval spent 40 microseconds since userentry
> + factorreturnA spent 9 microseconds since maxcheckval
> + factorreturnB spent 48 microseconds since factorreturnA
> + prime spent 10 microseconds from factorreturnB until ending
> +
> +^C
> +5
> +```
> +
> +As is observed in the previous example, there is now a set of
> +DTrace features that can be used with the probes that were
> +created.
> diff --git a/doc/tutorial/4.GoingFurtherWithDTrace.md b/doc/tutorial/4.GoingFurtherWithDTrace.md
> new file mode 100644
> index 000000000..e267db907
> --- /dev/null
> +++ b/doc/tutorial/4.GoingFurtherWithDTrace.md
> @@ -0,0 +1,14 @@
> +
> +## Going Further With DTrace
> +
> +For more information about using DTrace on Linux, see the
> +[DTrace User Guide](../userguide/index.md).
> +
> +There are [DTrace trainings videos](https://oracle-samples.github.io/oltrain/tracks/ol/dtrace).
> +
> +The latest DTrace development work and source code for Linux is
> +available at <https://github.com/oracle/dtrace-utils/>.
> +
> +Or, there are [DTrace packages built for Oracle Linux](https://www.oracle.com/linux/downloads/linux-dtrace.html).
> +There are
> +[Oracle Linux: DTrace Release Notes](https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-relnotes/)
> diff --git a/doc/tutorial/index.md b/doc/tutorial/index.md
> new file mode 100644
> index 000000000..60d5dd8b4
> --- /dev/null
> +++ b/doc/tutorial/index.md
> @@ -0,0 +1,15 @@
> +# DTrace Tutorial
> +
> +- [Introducing DTrace](1.IntroducingDTrace.md)
> +- [Tracing Operating System Behavior](2.TracingOperatingSystemBehavior.md)
> +- [Tracing User Space Applications](3.TracingUserSpaceApplications.md)
> +- [Going Further With DTrace](4.GoingFurtherWithDTrace.md)
> +
> +<!--
> +These are from:
> +https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-Preface.html
> +https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-IntroducingDTrace.html
> +https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-TracingOperatingSystemBehavior.html
> +https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-TracingUserSpaceApplications.html
> +https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-GoingFurtherWithDTrace.html
> +-->
> --
> 2.47.3
>
More information about the DTrace-devel
mailing list