Allow test cases to ignore expected value of score or penalty. Issue #1703.

Author: sangwinc

doc/en/STACK_question_admin/Testing.md

@@ -41,6 +41,7 @@ The teacher can also examine by hand the outcomes generated by each step of each
   3. One example of each distinction you wish to make, i.e. if you have added specific feedback then provide an answer you expect to trigger this.
   4. Some "invalid" responses, especially if these are syntactically-valid expressions.  E.g. If the answer is an equation such as \(y=2x+1\), then \(2x+1\) might be invalid if you have chosen the input option "check types".  Adding a test case is useful to confirm this potential problem is caught by the question.  Leave the fields empty and the expected answer note `NULL` to indicate this.
 9. If you leave the penalty field blank it will assume you mean the default penalty for the question.
+10. If you set either the score or penalty field to -1 then the test will pass on any value of that field, effectively ignoring the field for the purposes of this test case.  This is not recommended (as you're not testing anymore!).
 
 Normally test cases are constructed taking into account values of the question variables, so test-case construction can reflect any random variants.  If you start the expected value of an input with the tag `RAW:` (case sensitive) then the remainder of your input will be used as a raw string.  E.g. if your test case is `RAW:2 x` then your input test case will be `2 x`.  Note, this feature does _not_ evaluate the expression further, and values of question variables will not be used as part of constructing the input for this test case.  However, the potential response tree _will_ have access to the question variables later.  For example, if `n:42` in the question variables then typically student's who input `n` will get a validation error.  `RAW:n` allows you to create a test case, which will be displayed as `n` but later the PRTs will pick up the question variables.  This feature is intended to test _invalid_ input rather than evaluation of test cases constructed from the random variables.  We can't have it both ways!
 

stack/questiontestresult.php

@@ -199,8 +199,11 @@ public function get_prt_states() {
                 is_null($state->expectedscore) != is_null($state->score) ||
                     abs($state->expectedscore - $state->score) > 10E-6
             ) {
-                $state->testoutcome = false;
-                $reason[] = stack_string('score');
+                // When the expected score is -1 the test does not fail.
+                if ($state->expectedscore != -1) {
+                    $state->testoutcome = false;
+                    $reason[] = stack_string('score');
+                }
             }
             // If the expected penalty is null then we use the question default penalty.
             $penalty = $state->expectedpenalty;
@@ -216,8 +219,11 @@ public function get_prt_states() {
                     is_null($state->penalty) ||
                         abs($penalty - $state->penalty) > 10E-6
                 ) {
-                    $state->testoutcome = false;
-                    $reason[] = stack_string('penalty');
+                    // When the expected penalty is -1 the test does not fail.
+                    if ($penalty != -1) {
+                        $state->testoutcome = false;
+                        $reason[] = stack_string('penalty');
+                    }
                 }
             }
             [$noteresult, $messages] = $this->test_answer_note(

tests/questiontype_test.php

@@ -143,7 +143,6 @@ public function test_get_possible_responses_test4(): void {
     }
 
     public function test_initialise_question_instance(): void {
-
         $qdata = test_question_maker::get_question_data('stack', 'test3');
         $q = $this->qtype->make_question($qdata);
         $expectedq = test_question_maker::make_question('stack', 'test3');
@@ -193,17 +192,29 @@ public function test_question_tests_test3(): void {
         ));
         $testcases[] = $qtest;
 
-        // This unit test runs a question test, with an input name as
-        // the expected answer, which should work.
+        // This unit test runs a question test, with an input name as the expected answer.
         $qtest = new stack_question_test('', ['ans2' => 'ans2']);
         $qtest->add_expected_result('even', new stack_potentialresponse_tree_state(
             1,
-            true,
+            false,
             1,
             0,
             '',
             ['even-1-T']
         ));
+        $testcases[] = $qtest;
+
+        // This unit test runs a question test, accepting any score and penalty.
+        $qtest = new stack_question_test('', ['ans2' => 'x^2']);
+        $qtest->add_expected_result('even', new stack_potentialresponse_tree_state(
+            1,
+            true,
+            -1,
+            -1,
+            '',
+            ['even-1-T']
+            ));
+        $testcases[] = $qtest;
 
         foreach ($testcases as $testcase) {
             $result = $testcase->test_question($questionid, $seed, context_system::instance());