[DTrace-devel] [PATCH] test: Loosen tolerances on args_kernel tests

[Eugene Loh]

On 3/22/23 11:05, Kris Van Hees wrote:

> Question below (just to better understand this)...
>
> On Mon, Feb 27, 2023 at 04:16:10PM -0500, eugene.loh--- via DTrace-devel wrote:
>> From: Eugene Loh <eugene.loh at oracle.com>
>>
>> Signed-off-by: Eugene Loh <eugene.loh at oracle.com>
>> ---
>>   test/unittest/cpc/tst.args_kernel.sh       | 7 +++----
>>   test/unittest/profile-n/tst.args_kernel.sh | 7 +++----
>>   2 files changed, 6 insertions(+), 8 deletions(-)
>>
>> diff --git a/test/unittest/cpc/tst.args_kernel.sh b/test/unittest/cpc/tst.args_kernel.sh
>> index 0cb2480a..c1911588 100755
>> --- a/test/unittest/cpc/tst.args_kernel.sh
>> +++ b/test/unittest/cpc/tst.args_kernel.sh
>> @@ -71,9 +71,8 @@ END {
>>   
>>   # report
>>   status=0
>> -margin=$(($expect / 10))
>> -$utils/check_result.sh $ntotal $expect $margin; status=$(($status + $?))
>> -$utils/check_result.sh $nwarn     0    $margin; status=$(($status + $?))
>> -$utils/check_result.sh $nerror    0       0   ; status=$(($status + $?))
>> +$utils/check_result.sh $ntotal $expect $(($expect /  2)); status=$(($status + $?))
> Does this mean that we're accepting an error margin of 50%?

Yes.

> That seems rather extreme...

Yes.

> I wonder whether there is a way to tighten up the test to not need
> such a wide margin.

That would be great.  Frankly, there are other mysteries as well. It 
would be nice to understand them better.

> I know the test is marked unstable for now, so perhaps
> mostly we need a comment explaining why such a wide margin is acceptable?

I'm not 100% sure of how things unfolded, but I think I had assumed that 
when the "unstable" annotation was added we were no longer pursuing this 
patch.  "50%" was a way of saying "At least something is working right, 
even if we don't understand it all."  In contrast, "unstable" is perhaps 
a better way of expressing the mystery we're seeing here.

>> +$utils/check_result.sh $nwarn     0    $(($expect / 10)); status=$(($status + $?))
>> +$utils/check_result.sh $nerror    0    $(($expect / 10)); status=$(($status + $?))
>>   
>>   exit $status
>> diff --git a/test/unittest/profile-n/tst.args_kernel.sh b/test/unittest/profile-n/tst.args_kernel.sh
>> index d489e554..8ef59893 100755
>> --- a/test/unittest/profile-n/tst.args_kernel.sh
>> +++ b/test/unittest/profile-n/tst.args_kernel.sh
>> @@ -66,9 +66,8 @@ END {
>>   
>>   # report
>>   status=0
>> -margin=$(($expect / 10))
>> -$utils/check_result.sh $ntotal $expect $margin; status=$(($status + $?))
>> -$utils/check_result.sh $nwarn     0    $margin; status=$(($status + $?))
>> -$utils/check_result.sh $nerror    0       0   ; status=$(($status + $?))
>> +$utils/check_result.sh $ntotal $expect $(($expect /  2 )); status=$(($status + $?))
>> +$utils/check_result.sh $nwarn     0    $(($expect / 10 )); status=$(($status + $?))
>> +$utils/check_result.sh $nerror    0    $(($expect / 10 )); status=$(($status + $?))
>>   
>>   exit $status
>> -- 
>> 2.18.4
>>
>>
>> _______________________________________________
>> DTrace-devel mailing list
>> DTrace-devel at oss.oracle.com
>> https://oss.oracle.com/mailman/listinfo/dtrace-devel