Testing a node, not the whole tree

[sangwinc]

Currently the question testing only confirms the last note after the whole PRT has been traversed.

Point .5 here: https://docs.stack-assessment.org/en/STACK_question_admin/Testing/ says "Currently only the last Answer Note, not the whole path through the potential response tree, is examined. This is a limitation."

This was a compromise when I set up the testing system some time ago. We need to allow us to specify tests much more flexibly, without breaking back compatibility!

[Mat-Ge]

Thanks for opening this issue @sangwinc!
Here are my suggestions: Testing nodes.pdf

[aharjula]

Well, if we are going there, the whole thing should probably be rewritten... The thing we most definitely should not do is to add to the current way of testing any waypoints that the path through PRT needs to take.

My personal opinion is that a test should define as little as possible so that it is easy to add simple tests checking singular things without having to fix all the PRTs and values at the same time. This is why, in Stateful, the tests are defined by declaring test inputs for each PRT leaf at the node itself, and the placement of the test input directly defines that this input should lead to this node evaluating to true/false. Note that the testing logic there does not fix the score/penalty of that PRT and does not care at all about other PRTs. Stateful does not currently test non-leaf nodes, nor does it test many other things, some because neither does STACK, but some due to me getting there yet...

In my opinion, testing should be closer to the question editing, but as the editing form already has quite a lot of fields, a separate page mirroring the trees and inputs is probably the best we can have. And if we had that, then I would like to see the following things:

1. For each input, one should be able to define inputs that must be considered valid and inputs that need to be considered invalid. This is necessary to catch misconfiguration and changes in the system. Cloning these tests between inputs and questions should be made easy, as in copy-paste.
2. For each PRT, one should be able to define combinations of inputs and the score/penalty they should produce for that PRT. If one wants, one could also define the exit note. Thus making it possible to map old tests to these kinds of test cases.
3. For each PRT-node exit (true/false), one should be able to define the input values needed to trigger this PRT and which need to go through this node and exit it in that direction. Note that we do not define the answernote here, we define the node and exit, thus making it theoretically possible to have multiple nodes producing the same note, but still confirming that the correct node emitted it.

Note that 2 and 3 would also provide implicit valid input tests for inputs, but cannot define invalid input cases.

[aharjula]

Also, we should have "test variables" to ease building up the input values for those test cases. Just one keyval shared by all tests. It is silly that currently constructing complicated samples for special classes is either "keyhole surgery" in the small test input field or needs to happen in the question variables. The test execution would then take three CAS sessions.

1. Loading up the seed, question variables, test variables and all input definitions, generating input "strings".
2. Those "strings" would then go through PHP side input validation and CAS side validation in the second CAS session.
3. Finally, the valid strings would be fed to the PRT functions in the last session, and logic to check if the PRT output matches would be included in that session, so that we do not need to output the full PRT function output for potentially a large number of tests, instead just booleans.

Finally, it might be nice to be able to define conditions for tests, i.e., a boolean expression that would be by default true, but could be used to disable tests if some combination of random parameters makes it nonsensical.

[sangwinc]

Thanks @aharjula, I hadn't thought of a "test variables" keyval field. That's an excellent idea.

[sangwinc]

Lots of ideas here (e.g. "any" for score, new boolean field, dealing with testing invalid inputs,....). Just focusing on deciding if a single PRT evaluation test "passes" we do the following.

Expand the current "expectedanswernote" https://github.com/maths/moodle-qtype_stack/blob/master/db/install.xml#L178 (which needs to be longer the 255).

This field now holds any expected notes from the branches, e.g. prt1-1-F | prt1-3-F | prt1-4-F. Note in this example, there is no expectation from node 2. Individual notes are separated by | as in the current output, but this becomes part of the formal note syntax.

The test passes if and only if the PRT produces those notes in order. We condone any notes from the PRT which don't appear in the test case expectation. Hence, with the expectation prt1-1-F | prt1-3-F | prt1-4-F all the following will pass as tests.

• prt1-1-F | prt1-2-T | prt1-3-F | prt1-4-F
• prt1-1-F | prt1-2-F | prt1-3-F | prt1-4-F
• prt1-1-F | prt1-2-T | prt1-3-F | prt1-4-T | prt1-5-F | prt1-6-F

This is simple and I think authors will actually be able to write tests like this and understand what is being tested when they read other people's test.

1. The current test setup is a special case of this (retaining back compatibility).
2. We now test non-leaf nodes (the underlying purpose of this issue).
3. Simple to author - you don't have to specify the whole route through the tree. (Authoring interface is another issue of course). You can specify the whole route through the tree.
4. This can ignore output from answer tests. By design some answer tests give information about the specific input (e.g. missing items in a set). This is super helpful to teachers and for statistics, but wrecks testing of questions (each input gets a different note). I don't want regular expressions (!) in test case setup....

The simplest authoring interface would be the "confirm current test behavior" button which copies the current note as the test expectation, and allows a teacher to edit (delete) any individual PRT notes which should not form part of the test case. The authoring interface changes from a dropdown (prevents author errors) to a string input (more fragile).

It would be a fun (student project?!) to graphically illustrate the expected and actual route through the tree by expanding the current PRT graph library, or writing something else. (I get ahead of myself of course....)

[aharjula]

Ordered list makes sense, but can we still have some notation for marking that the last element of the list must be the last element of the output? Basically, to have the old behavior, that fixes the test-case to the end of the path. The reasoning for this is that traditionally, expanding the tree has meant that one needs to think about tests of that branch again and those tests have triggered that thinking, but now if the old end is still on the path the test might not trigger unless there are score/penalty differences, but if those can be any...

Separate syntax is needed, as the existing tests are assumed to be fixed to the end, and changing them to match any position might cause unexpected things, if the trees expand.

Basically, if our current value there is an unwrapped list with only one element, maybe we could interpret unwrapped lists as anchored to the end and have some wrapping or special first token to declare that the path does not need to end to the last element of our list. For example, wrapped [prt1-1-F | prt1-3-F | prt1-4-F] would accept prt1-1-F | prt1-2-T | prt1-3-F | prt1-4-T | prt1-5-F | prt1-6-F but unwrapped prt1-1-F | prt1-3-F | prt1-4-F would not as the last element would not match.

[sangwinc]

Good point @aharjula, yes I think we do need to be able "anchor" the start and end of list as you suggest.

Note to self: we will also have to update the "Tidy PRT" script......

[aharjula]

Ohh... Anchoring the start starts to get interesting... Well maybe we could use the range notations: [...] for both ends fixed, (...] for the end, [...) for the start, (...) for neither and (...] would be the default if no wrapping defined.

Just means that one needs to be careful should anyone have those chars in their notes. Actually, is the |-char even forbidden in notes?

[sangwinc]

Just means that one needs to be careful should anyone have those chars in their notes. Actually, is the |-char even forbidden in notes?

Yes, |`-chars are forbidden in notes: https://github.com/maths/moodle-qtype_stack/blob/master/questiontype.php#L2284

[sangwinc]

Finally, it might be nice to be able to define conditions for tests, i.e., a boolean expression that would be by default true, but could be used to disable tests if some combination of random parameters makes it nonsensical.

Do you have a specific example here?

[aharjula]

Finally, it might be nice to be able to define conditions for tests, i.e., a boolean expression that would be by default true, but could be used to disable tests if some combination of random parameters makes it nonsensical.

Do you have a specific example here?

Not really, but suppose you happen to have a PRT subtree that only activates when the parameters of the question are just right. Maybe you go check for some N-th root or some other feature separately, only if the parameters are suitable, that PRT subtree would otherwise be inactive. But you would still want to test it in the cases where it activates. Currently, you cannot test such input cases unless the path returns to the common tree from that potentially inactive subtree, as otherwise the test fails for conditions where that subtree would be inactive. It might also be that the condition would also define whether it is possible to generate a test input matching the test case at all for that particular variant.

Of course, the traditional solution to this has been to select parametrisation so that the whole question works the same for all possible variants, and if that is not possible, then split into multiple questions handling the different categories of variants. But one could build questions that have capabilities that activate if one chooses to pick different variants.

I don't have a good example, but over the years, it has not been uncommon to see someone tuning the randomisation range of an existing question and find out that things break and then split off to another version of the same question where the issues causing the breakage have been handled. Sometimes it is hard to understand the whole story of such sibling questions, and one has to wonder whether being able to keep them together within a single question, where the code documents the reason why they have separate handling for their separate cases, might be better. But as long as testing for such things inside a singular question is impossible, one cannot experiment on whether it would be sensible to bring things together into a shared question codebase, although bigger and typically better documented.

[sangwinc]

OK, good point @aharjula let's keep this in mind. I think the proposals in the current #1703 branch are big improvements, and as we are able to implement more comprehensive question testing the need for the situations you describe above will become more apparent and we can then add them.