[DTrace-devel] [PATCH v2 13/61] test: Report unexpected results for unstable tests as xfail

[eugene.loh at oracle.com]

From: Eugene Loh <eugene.loh at oracle.com>

Generally, tests are deterministically expected to pass or xfail.

Some tests have intermittent results, however, and are tagged
"unstable".  For such tests, we tolerate xpass and fail outcomes.
But when such otherwise unexpected results occur, we must debug,
possibly overlooking the simple explanation that the test is
unstable to start with.  One must check manually whether the test
is unstable (assuming one thinks to check).

If a test xpasses or fails, then if it is unstable, just report it
as xfail and unstable.

Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
---
 runtest.sh | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/runtest.sh b/runtest.sh
index af1f67d3..16d86d20 100755
--- a/runtest.sh
+++ b/runtest.sh
@@ -1379,6 +1379,17 @@ for dt in $dtrace; do
             dt_flags="$orig_dt_flags"
         done
 
+        # If it xpasses but is unstable, just call it xfail.
+        if [[ -z $fail ]] && [[ -n $xfail ]] && [[ "$(extract_options tags $_test)" == *unstable* ]]; then
+            fail=t
+
+            if [[ -z $xfailmsg ]]; then
+                xfailmsg="xpassed but unstable"
+            else
+                xfailmsg="xpassed but unstable; $xfailmsg"
+            fi
+        fi
+
         if [[ -z $fail ]]; then
 
             # Success!
@@ -1393,6 +1404,18 @@ for dt in $dtrace; do
             pass "$xfail" "$xfailmsg" "$capturing"
 
         else
+
+            # If it fails, is not xfail, and is unstable, just call it xfail.
+            if [[ -z $xfail ]] && [[ "$(extract_options tags $_test)" == *unstable* ]]; then
+                xfail=t
+
+                if [[ -z $xfailmsg ]]; then
+                    xfailmsg="unstable"
+                else
+                    xfailmsg="unstable; $xfailmsg"
+                fi
+            fi
+
             fail "$xfail" "$xfailmsg" "$failmsg${capturing:+, $capturing}"
         fi
 
-- 
2.18.4