[DTrace-devel] [PATCH 13/61] test: Report unstable tests that fail as xfail

[Kris Van Hees]

I would like Nick to also givwe his OK on this since he is the brain behind
runtest.sh.  It looks like a sensible change to me.

Small comment below...

On Fri, Jul 08, 2022 at 10:44:57AM -0400, eugene.loh--- via DTrace-devel wrote:
> From: Eugene Loh <eugene.loh at oracle.com>
> 
> Generally, tests are deterministically expected to pass or xfail.
> 
> Some tests fail intermittently, however, and are tagged "unstable".
> Either pass or fail is acceptable, but one has to check manually
> to see if the test is unstable.
> 
> If a test fails and it is unstable, just report it as xfail and
> unstable.

This commit msg is good.  The one you have one your branch is more verbose,
and actually, more confusing.  Let's use this one, and not the newer one on
your branch.
> 
> Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
> ---
>  runtest.sh | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/runtest.sh b/runtest.sh
> index af1f67d3..4c0bb381 100755
> --- a/runtest.sh
> +++ b/runtest.sh
> @@ -1393,6 +1393,18 @@ for dt in $dtrace; do
>              pass "$xfail" "$xfailmsg" "$capturing"
>  
>          else
> +
> +            # If it fails, is not xfail, and is unstable, just call it xfail.
> +            if [[ -z $xfail ]] && [[ "$(extract_options tags $_test)" == *unstable* ]]; then
> +                xfail=t
> +
> +                if [[ -z $xfailmsg ]]; then
> +                    xfailmsg="unstable"
> +                else
> +                    xfailmsg="unstable; $xfailmsg"
> +                fi
> +            fi
> +
>              fail "$xfail" "$xfailmsg" "$failmsg${capturing:+, $capturing}"
>          fi
>  
> -- 
> 2.18.4
> 
> 
> _______________________________________________
> DTrace-devel mailing list
> DTrace-devel at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/dtrace-devel