[DTrace-devel] [PATCH v2 13/61] test: Report unexpected results for unstable tests as xfail

[Nick Alcock]

On 14 Jul 2022, eugene loh verbalised:

> From: Eugene Loh <eugene.loh at oracle.com>
>
> Generally, tests are deterministically expected to pass or xfail.
>
> Some tests have intermittent results, however, and are tagged
> "unstable".  For such tests, we tolerate xpass and fail outcomes.
> But when such otherwise unexpected results occur, we must debug,
> possibly overlooking the simple explanation that the test is
> unstable to start with.  One must check manually whether the test
> is unstable (assuming one thinks to check).
>
> If a test xpasses or fails, then if it is unstable, just report it
> as xfail and unstable.
> 
> Signed-off-by: Eugene Loh <eugene.loh at oracle.com>

Makes sense, doesn't stop us from using !unstable in tags to stop
unstable tests running entirely, stops spurious XPASS/FAILs from popping
up, and still allows ordinary PASSes: looks good. I like it!

Reviewed-by: Nick Alcock <nick.alcock at oracle.com>

> +        if [[ -z $fail ]] && [[ -n $xfail ]] && [[ "$(extract_options tags $_test)" == *unstable* ]]; then

I didn't know you could use globbing in == like that! (I assumed it
would match filenames, but it doesn't, it expands like case does.)