[DTrace-devel] [PATCH] Improve diagnosibility of disassembly tests
Kris Van Hees
kris.van.hees at oracle.com
Fri Apr 29 01:31:14 UTC 2022
On Thu, Apr 28, 2022 at 08:20:26PM -0400, eugene.loh--- via DTrace-devel wrote:
> From: Eugene Loh <eugene.loh at oracle.com>
>
> If these tests fail, they provide essentially no diagnostic output.
Diagnosing a test dailure usually is done as a combination of looking at
test output in the log and actually running the testcase manually, and
investigating more detailed output. Most tests are written that way since
it would take quite a bit of extra code to properly emit diagnotstics for
all the possible failures cases for tests. And especially when the failure
is due to something entirely unrelated, there is just no real way to offer
proper diagnostics (since you do not know in advance what you may need).
> Further, if these tests pass, they emit a little bit of output into
> the log file. This is largely harmless, but simply not useful.
I think that "useful" is a matter of opinion, and given that I wrote these
tests I can certainly state that the output *is* useful. The tests produces
output that is checked against the expected output, as we do with most tests.
It also assists in ascertaining that the test is actually doing what it is
supposed to do rather than perhaps reporting a PASS for the wrong reasons.
> Change the tests so that fails report what went wrong and passes
> emit no gratuitous output.
No thank you. I'd be open to code being added to check the return code of
the dtrace invocation to report whether execution of the command failed in
some way. One way would be to have the awk filter have a clause for /^dtrace:/
since that will cause a mismatch in expected output, and you get the error
message nicely reported in the runtest.log. For tests that do not pass the
-e option, we just need to update the expected output to include the reporting
of how many probes were matched.
> Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
> ---
> test/unittest/disasm/tst.ann-gvar-assoc.r | 2 -
> test/unittest/disasm/tst.ann-gvar-assoc.sh | 26 +++++++++-
> test/unittest/disasm/tst.ann-js-neg.r | 2 -
> test/unittest/disasm/tst.ann-js-neg.sh | 38 ++++++++++----
> test/unittest/disasm/tst.ann-reg-spill.r | 7 ---
> test/unittest/disasm/tst.ann-reg-spill.sh | 35 +++++++++++--
> .../unittest/disasm/tst.ann-strconst-strtab.r | 2 -
> .../disasm/tst.ann-strconst-strtab.sh | 48 ++++++++++++------
> test/unittest/disasm/tst.ann-strconst.r | 1 -
> test/unittest/disasm/tst.ann-strconst.sh | 39 +++++++++++----
> test/unittest/disasm/tst.ann-tvar-assoc.r | 2 -
> test/unittest/disasm/tst.ann-tvar-assoc.sh | 26 +++++++++-
> test/unittest/disasm/tst.ann-tvar.r | 1 -
> test/unittest/disasm/tst.ann-tvar.sh | 25 +++++++++-
> test/unittest/disasm/tst.ann-var.r | 23 ---------
> test/unittest/disasm/tst.ann-var.sh | 50 ++++++++++++++++++-
> 16 files changed, 243 insertions(+), 84 deletions(-)
> delete mode 100644 test/unittest/disasm/tst.ann-gvar-assoc.r
> delete mode 100644 test/unittest/disasm/tst.ann-js-neg.r
> delete mode 100644 test/unittest/disasm/tst.ann-reg-spill.r
> delete mode 100644 test/unittest/disasm/tst.ann-strconst-strtab.r
> delete mode 100644 test/unittest/disasm/tst.ann-strconst.r
> delete mode 100644 test/unittest/disasm/tst.ann-tvar-assoc.r
> delete mode 100644 test/unittest/disasm/tst.ann-tvar.r
> delete mode 100644 test/unittest/disasm/tst.ann-var.r
>
> diff --git a/test/unittest/disasm/tst.ann-gvar-assoc.r b/test/unittest/disasm/tst.ann-gvar-assoc.r
> deleted file mode 100644
> index f7563a51..00000000
> --- a/test/unittest/disasm/tst.ann-gvar-assoc.r
> +++ /dev/null
> @@ -1,2 +0,0 @@
> -85 0 1 0000 ffffffff call dt_get_assoc ! st[]
> -85 0 1 0000 ffffffff call dt_get_assoc ! ld[]
> diff --git a/test/unittest/disasm/tst.ann-gvar-assoc.sh b/test/unittest/disasm/tst.ann-gvar-assoc.sh
> index 6690967d..2fa7e9c1 100755
> --- a/test/unittest/disasm/tst.ann-gvar-assoc.sh
> +++ b/test/unittest/disasm/tst.ann-gvar-assoc.sh
> @@ -8,6 +8,8 @@
>
> dtrace=$1
>
> +label=$tmpdir/$(basename $_test).$$.$RANDOM
> +
> $dtrace $dt_flags -Sen '
> int st[int], ld[int];
>
> @@ -17,6 +19,26 @@ BEGIN
> trace(ld[5]);
> exit(0);
> }
> -' 2>&1 | awk '/ call dt_get_assoc/ { sub(/^[^:]+: /, ""); print; }'
> +' > $label.out 2>&1
> +
> +if [ $? -ne 0 ]; then
> + echo DTrace run failed
> + cat $label.out
> + exit 1
> +fi
> +
> +cat << EOF > $label.cmp
> +85 0 1 0000 ffffffff call dt_get_assoc ! st[]
> +85 0 1 0000 ffffffff call dt_get_assoc ! ld[]
> +EOF
> +
> +awk '/ call dt_get_assoc/ { sub(/^[^:]+: /, ""); print; }' $label.out \
> +| diff -q - $label.cmp
> +
> +if [ $? -ne 0 ]; then
> + echo comparison failed
> + cat $label.out
> + exit 1
> +fi
>
> -exit $?
> +exit 0
> diff --git a/test/unittest/disasm/tst.ann-js-neg.r b/test/unittest/disasm/tst.ann-js-neg.r
> deleted file mode 100644
> index d70a02ff..00000000
> --- a/test/unittest/disasm/tst.ann-js-neg.r
> +++ /dev/null
> @@ -1,2 +0,0 @@
> -75 0 0 XXXX 00000000 jsge %r0, 0, 3
> -d5 0 0 XXXX fffff800 jsle %r0, -2048, 2
> diff --git a/test/unittest/disasm/tst.ann-js-neg.sh b/test/unittest/disasm/tst.ann-js-neg.sh
> index 0914d3fe..a7d6994f 100755
> --- a/test/unittest/disasm/tst.ann-js-neg.sh
> +++ b/test/unittest/disasm/tst.ann-js-neg.sh
> @@ -8,17 +8,37 @@
>
> dtrace=$1
>
> +label=$tmpdir/$(basename $_test).$$.$RANDOM
> +
> $dtrace $dt_flags -xdisasm=8 -Sn '
> BEGIN, syscall::write:return
> {
> exit(0);
> }
> -' 2>&1 | \
> - awk '/js[a-z]+/ {
> - sub(/^[^:]+: /, "");
> - sub(/ +!.*$/, "");
> - sub(/ [0-9a-f]{4} /, " XXXX ");
> - print;
> - }'
> -
> -exit $?
> +' > $label.out 2>&1
> +
> +if [ $? -ne 0 ]; then
> + echo DTrace run failed
> + cat $label.out
> + exit 1
> +fi
> +
> +cat << EOF > $label.cmp
> +75 0 0 XXXX 00000000 jsge %r0, 0, 3
> +d5 0 0 XXXX fffff800 jsle %r0, -2048, 2
> +EOF
> +
> +awk '/js[a-z]+/ {
> + sub(/^[^:]+: /, "");
> + sub(/ +!.*$/, "");
> + sub(/ [0-9a-f]{4} /, " XXXX ");
> + print;
> + }' $label.out | diff -q - $label.cmp
> +
> +if [ $? -ne 0 ]; then
> + echo comparison failed
> + cat $label.out
> + exit 1
> +fi
> +
> +exit 0
> diff --git a/test/unittest/disasm/tst.ann-reg-spill.r b/test/unittest/disasm/tst.ann-reg-spill.r
> deleted file mode 100644
> index 2b71ca92..00000000
> --- a/test/unittest/disasm/tst.ann-reg-spill.r
> +++ /dev/null
> @@ -1,7 +0,0 @@
> -7b a 8 ffa8 00000000 stdw [%fp-88], %r8 ! spill %r8
> -79 8 a ffa8 00000000 lddw %r8, [%fp-88] ! restore %r8
> -7b a 8 ffa8 00000000 stdw [%fp-88], %r8 ! spill %r8
> -7b a 7 ffb0 00000000 stdw [%fp-80], %r7 ! spill %r7
> -79 7 a ffb0 00000000 lddw %r7, [%fp-80] ! restore %r7
> -79 8 a ffa8 00000000 lddw %r8, [%fp-88] ! restore %r8
> -123456798
> diff --git a/test/unittest/disasm/tst.ann-reg-spill.sh b/test/unittest/disasm/tst.ann-reg-spill.sh
> index 16786947..8c144d4b 100755
> --- a/test/unittest/disasm/tst.ann-reg-spill.sh
> +++ b/test/unittest/disasm/tst.ann-reg-spill.sh
> @@ -8,6 +8,8 @@
>
> dtrace=$1
>
> +label=$tmpdir/$(basename $_test).$$.$RANDOM
> +
> $dtrace $dt_flags -qSn '
> BEGIN
> {
> @@ -22,6 +24,33 @@ BEGIN
> i = 9;
> trace(++a + (++b + (++c + (++d + (++e + (++f + (++g + (++h + ++i))))))));
> exit(0);
> -}' 2>&1 | awk '/! (spill|restore)/ { sub(/^[^:]+: /, ""); print; next; }
> - { s = $0; }
> - END { print s; }'
> +}
> +' > $label.out 2>&1
> +
> +if [ $? -ne 0 ]; then
> + echo DTrace run failed
> + cat $label.out
> + exit 1
> +fi
> +
> +cat << EOF > $label.cmp
> +7b a 8 ffa8 00000000 stdw [%fp-88], %r8 ! spill %r8
> +79 8 a ffa8 00000000 lddw %r8, [%fp-88] ! restore %r8
> +7b a 8 ffa8 00000000 stdw [%fp-88], %r8 ! spill %r8
> +7b a 7 ffb0 00000000 stdw [%fp-80], %r7 ! spill %r7
> +79 7 a ffb0 00000000 lddw %r7, [%fp-80] ! restore %r7
> +79 8 a ffa8 00000000 lddw %r8, [%fp-88] ! restore %r8
> +123456798
> +EOF
> +
> +awk '/! (spill|restore)/ { sub(/^[^:]+: /, ""); print; next; }
> + { s = $0; }
> + END { print s; }' $label.out | diff -q - $label.cmp
> +
> +if [ $? -ne 0 ]; then
> + echo comparison failed
> + cat $label.out
> + exit 1
> +fi
> +
> +exit 0
> diff --git a/test/unittest/disasm/tst.ann-strconst-strtab.r b/test/unittest/disasm/tst.ann-strconst-strtab.r
> deleted file mode 100644
> index 2c10e380..00000000
> --- a/test/unittest/disasm/tst.ann-strconst-strtab.r
> +++ /dev/null
> @@ -1,2 +0,0 @@
> -linked 07 X 0 0000 XXXXXXXX add %rX, D ! "a"
> -final 07 X 0 0000 XXXXXXXX add %rX, D ! "a"
> diff --git a/test/unittest/disasm/tst.ann-strconst-strtab.sh b/test/unittest/disasm/tst.ann-strconst-strtab.sh
> index 76a65604..84df6b84 100755
> --- a/test/unittest/disasm/tst.ann-strconst-strtab.sh
> +++ b/test/unittest/disasm/tst.ann-strconst-strtab.sh
> @@ -15,23 +15,43 @@
>
> dtrace=$1
>
> +label=$tmpdir/$(basename $_test).$$.$RANDOM
> +
> $dtrace $dt_flags -xdisasm=12 -Sn '
> BEGIN
> {
> a["a"] = 42;
> exit(0);
> }
> -' 2>&1 | \
> - awk '/^Disassembly of/ {
> - kind = $3;
> - next;
> - }
> - / ! "/ {
> - sub(/^[^:]+:/, kind);
> - sub(/ 07 [0-9] /, " 07 X ");
> - sub(/[0-9a-f]{8} add/, "XXXXXXXX add");
> - sub(/%r[0-9], [0-9]+ +/, "%rX, D ");
> - print;
> - }'
> -
> -exit $?
> +' > $label.out 2>&1
> +
> +if [ $? -ne 0 ]; then
> + echo DTrace run failed
> + cat $label.out
> + exit 1
> +fi
> +
> +cat << EOF > $label.cmp
> +linked 07 X 0 0000 XXXXXXXX add %rX, D ! "a"
> +final 07 X 0 0000 XXXXXXXX add %rX, D ! "a"
> +EOF
> +
> +awk '/^Disassembly of/ {
> + kind = $3;
> + next;
> + }
> + / ! "/ {
> + sub(/^[^:]+:/, kind);
> + sub(/ 07 [0-9] /, " 07 X ");
> + sub(/[0-9a-f]{8} add/, "XXXXXXXX add");
> + sub(/%r[0-9], [0-9]+ +/, "%rX, D ");
> + print;
> + }' $label.out | diff -q - $label.cmp
> +
> +if [ $? -ne 0 ]; then
> + echo comparison failed
> + cat $label.out
> + exit 1
> +fi
> +
> +exit 0
> diff --git a/test/unittest/disasm/tst.ann-strconst.r b/test/unittest/disasm/tst.ann-strconst.r
> deleted file mode 100644
> index 49273e33..00000000
> --- a/test/unittest/disasm/tst.ann-strconst.r
> +++ /dev/null
> @@ -1 +0,0 @@
> -07 X 0 0000 XXXXXXXX add %rX, D ! "strconst"
> diff --git a/test/unittest/disasm/tst.ann-strconst.sh b/test/unittest/disasm/tst.ann-strconst.sh
> index 81e1268c..d43d42ed 100755
> --- a/test/unittest/disasm/tst.ann-strconst.sh
> +++ b/test/unittest/disasm/tst.ann-strconst.sh
> @@ -8,19 +8,38 @@
>
> dtrace=$1
>
> +label=$tmpdir/$(basename $_test).$$.$RANDOM
> +
> $dtrace $dt_flags -Sen '
> BEGIN
> {
> var = "strconst";
> exit(0);
> }
> -' 2>&1 | \
> - awk '/ ! "strconst"/ {
> - sub(/^[^:]+: /, "");
> - sub(/^07 [0-9] /, "07 X ");
> - sub(/[0-9a-f]{8} add/, "XXXXXXXX add");
> - sub(/%r[0-9], [0-9]+ +/, "%rX, D ");
> - print;
> - }'
> -
> -exit $?
> +' > $label.out 2>&1
> +
> +if [ $? -ne 0 ]; then
> + echo DTrace run failed
> + cat $label.out
> + exit 1
> +fi
> +
> +cat << EOF > $label.cmp
> +07 X 0 0000 XXXXXXXX add %rX, D ! "strconst"
> +EOF
> +
> +awk '/ ! "strconst"/ {
> + sub(/^[^:]+: /, "");
> + sub(/^07 [0-9] /, "07 X ");
> + sub(/[0-9a-f]{8} add/, "XXXXXXXX add");
> + sub(/%r[0-9], [0-9]+ +/, "%rX, D ");
> + print;
> + }' $label.out | diff -q - $label.cmp
> +
> +if [ $? -ne 0 ]; then
> + echo comparison failed
> + cat $label.out
> + exit 1
> +fi
> +
> +exit 0
> diff --git a/test/unittest/disasm/tst.ann-tvar-assoc.r b/test/unittest/disasm/tst.ann-tvar-assoc.r
> deleted file mode 100644
> index 573c93e2..00000000
> --- a/test/unittest/disasm/tst.ann-tvar-assoc.r
> +++ /dev/null
> @@ -1,2 +0,0 @@
> -85 0 1 0000 ffffffff call dt_get_assoc ! self->st[]
> -85 0 1 0000 ffffffff call dt_get_assoc ! self->ld[]
> diff --git a/test/unittest/disasm/tst.ann-tvar-assoc.sh b/test/unittest/disasm/tst.ann-tvar-assoc.sh
> index 354d8fbe..4bbf6817 100755
> --- a/test/unittest/disasm/tst.ann-tvar-assoc.sh
> +++ b/test/unittest/disasm/tst.ann-tvar-assoc.sh
> @@ -8,6 +8,8 @@
>
> dtrace=$1
>
> +label=$tmpdir/$(basename $_test).$$.$RANDOM
> +
> $dtrace $dt_flags -Sen '
> self int st[int], ld[int];
>
> @@ -17,6 +19,26 @@ BEGIN
> trace(self->ld[5]);
> exit(0);
> }
> -' 2>&1 | awk '/ call dt_get_assoc/ { sub(/^[^:]+: /, ""); print; }'
> +' > $label.out 2>&1
> +
> +if [ $? -ne 0 ]; then
> + echo DTrace run failed
> + cat $label.out
> + exit 1
> +fi
> +
> +cat << EOF > $label.cmp
> +85 0 1 0000 ffffffff call dt_get_assoc ! self->st[]
> +85 0 1 0000 ffffffff call dt_get_assoc ! self->ld[]
> +EOF
> +
> +awk '/ call dt_get_assoc/ { sub(/^[^:]+: /, ""); print; }' $label.out \
> +| diff -q - $label.cmp
> +
> +if [ $? -ne 0 ]; then
> + echo comparison failed
> + cat $label.out
> + exit 1
> +fi
>
> -exit $?
> +exit 0
> diff --git a/test/unittest/disasm/tst.ann-tvar.r b/test/unittest/disasm/tst.ann-tvar.r
> deleted file mode 100644
> index 5194531b..00000000
> --- a/test/unittest/disasm/tst.ann-tvar.r
> +++ /dev/null
> @@ -1 +0,0 @@
> -85 0 1 0000 ffffffff call dt_get_tvar ! self->var
> diff --git a/test/unittest/disasm/tst.ann-tvar.sh b/test/unittest/disasm/tst.ann-tvar.sh
> index 66501840..cb193934 100755
> --- a/test/unittest/disasm/tst.ann-tvar.sh
> +++ b/test/unittest/disasm/tst.ann-tvar.sh
> @@ -8,12 +8,33 @@
>
> dtrace=$1
>
> +label=$tmpdir/$(basename $_test).$$.$RANDOM
> +
> $dtrace $dt_flags -Sen '
> BEGIN
> {
> self->var = 42;
> exit(0);
> }
> -' 2>&1 | awk '/ call dt_get_tvar/ { sub(/^[^:]+: /, ""); print; }'
> +' > $label.out 2>&1
> +
> +if [ $? -ne 0 ]; then
> + echo DTrace run failed
> + cat $label.out
> + exit 1
> +fi
> +
> +cat << EOF > $label.cmp
> +85 0 1 0000 ffffffff call dt_get_tvar ! self->var
> +EOF
> +
> +awk '/ call dt_get_tvar/ { sub(/^[^:]+: /, ""); print; }' $label.out \
> +| diff -q - $label.cmp
> +
> +if [ $? -ne 0 ]; then
> + echo comparison failed
> + cat $label.out
> + exit 1
> +fi
>
> -exit $?
> +exit 0
> diff --git a/test/unittest/disasm/tst.ann-var.r b/test/unittest/disasm/tst.ann-var.r
> deleted file mode 100644
> index d7fdef90..00000000
> --- a/test/unittest/disasm/tst.ann-var.r
> +++ /dev/null
> @@ -1,23 +0,0 @@
> -CheckVariable_Z
> -CheckVariable_Y
> -CheckVariable_X
> -CheckVariable_A
> -CheckVariable_A
> -CheckVariable_A
> -CheckVariable_A
> -CheckVariable_B
> -CheckVariable_A
> -CheckVariable_C
> -CheckVariable_Z
> -this->CheckVariable_z
> -this->CheckVariable_y
> -this->CheckVariable_x
> -CheckVariable_C
> -this->CheckVariable_c
> -this->CheckVariable_b
> -this->CheckVariable_a
> -this->CheckVariable_b
> -CheckVariable_C
> -this->CheckVariable_x
> -CheckVariable_Y
> -this->CheckVariable_z
> diff --git a/test/unittest/disasm/tst.ann-var.sh b/test/unittest/disasm/tst.ann-var.sh
> index 68a827b6..0bef4d37 100755
> --- a/test/unittest/disasm/tst.ann-var.sh
> +++ b/test/unittest/disasm/tst.ann-var.sh
> @@ -1,7 +1,7 @@
> #!/bin/bash
> #
> # Oracle Linux DTrace.
> -# Copyright (c) 2021, Oracle and/or its affiliates. All rights reserved.
> +# Copyright (c) 2021, 2022, Oracle and/or its affiliates. All rights reserved.
> # Licensed under the Universal Permissive License v 1.0 as shown at
> # http://oss.oracle.com/licenses/upl.
>
> @@ -9,6 +9,8 @@
>
> dtrace=$1
>
> +label=$tmpdir/$(basename $_test).$$.$RANDOM
> +
> # generate disassembly and print the emitted variable names
> $dtrace $dt_flags -Sen '
> struct foo {
> @@ -50,4 +52,48 @@ BEGIN {
> trace(this->CheckVariable_x);
> trace( CheckVariable_Y);
> trace(this->CheckVariable_z);
> -}' |& awk '/ ! (|this->)CheckVariable_/ { print $NF }'
> +}
> +' > $label.out 2>&1
> +
> +if [ $? -ne 0 ]; then
> + echo DTrace run failed
> + cat $label.out
> + exit 1
> +fi
> +
> +cat << EOF > $label.cmp
> +CheckVariable_Z
> +CheckVariable_Y
> +CheckVariable_X
> +CheckVariable_A
> +CheckVariable_A
> +CheckVariable_A
> +CheckVariable_A
> +CheckVariable_B
> +CheckVariable_A
> +CheckVariable_C
> +CheckVariable_Z
> +this->CheckVariable_z
> +this->CheckVariable_y
> +this->CheckVariable_x
> +CheckVariable_C
> +this->CheckVariable_c
> +this->CheckVariable_b
> +this->CheckVariable_a
> +this->CheckVariable_b
> +CheckVariable_C
> +this->CheckVariable_x
> +CheckVariable_Y
> +this->CheckVariable_z
> +EOF
> +
> +awk '/ ! (|this->)CheckVariable_/ { print $NF }' $label.out \
> +| diff -q - $label.cmp
> +
> +if [ $? -ne 0 ]; then
> + echo comparison failed
> + cat $label.out
> + exit 1
> +fi
> +
> +exit 0
> --
> 2.18.4
>
>
> _______________________________________________
> DTrace-devel mailing list
> DTrace-devel at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/dtrace-devel
More information about the DTrace-devel
mailing list