Skip to content

Commit 4bbe3e4

Browse files
JackWilbJackWilb
committed
Fix timeline task label hover handling
1 parent 4045eab commit 4bbe3e4

4 files changed

Lines changed: 490 additions & 1 deletion

File tree

‎PARTICIPANT_FEEDBACK_VALIDATION_DESIGN.md‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
# Participant Feedback, Validation, and Progression Design
2+
3+
This document combines the design questions from:
4+
5+
- [Improve Provide Feedback Options for Training](https://github.com/revisit-studies/study/issues/526)
6+
- [Add response validation for all response types](https://github.com/revisit-studies/study/issues/486)
7+
- [Built in validations](https://github.com/revisit-studies/study/issues/751)
8+
- [Extending feedback types and options](https://github.com/revisit-studies/study/issues/637)
9+
10+
The goal is to define a flexible study config interface that lets a Study Designer validate participant responses, evaluate correctness, show participant-facing feedback, and control training progression.
11+
12+
## Proposed Conceptual Model
13+
14+
The default design separates four related concerns:
15+
16+
- `validation`: Determines whether a response is acceptable input. Examples: required answer, valid email, number within range, text matches regex.
17+
- `evaluation`: Determines whether an acceptable response is correct, partially correct, or incorrect. Examples: equals expected answer, selected option is correct, numeric estimate is within tolerance.
18+
- `feedback`: Determines what participant-facing message or hint is shown after validation or evaluation.
19+
- `progression`: Determines whether the Participant can continue, retry, pass a trial, pass a training block, or see a summary.
20+
21+
This separation keeps form validity distinct from training correctness. For example, a numeric answer can be valid because it is inside the allowed input range, but still incorrect because it is outside the target answer tolerance.
22+
23+
## Core Defaults
24+
25+
| Question | Default |
26+
| --- | --- |
27+
| What does validation mean? | `validation` means input acceptability, not correctness. |
28+
| Should correctness be separate? | Yes. Use separate concepts: `validation`, `evaluation`, `feedback`, and `progression`. |
29+
| Does invalid input block submission? | Yes. Invalid input blocks submission. Incorrect but valid input may submit unless `progression` blocks it. |
30+
| Can feedback attach to both invalid and incorrect answers? | Yes. Feedback can be attached to validation failures and evaluation outcomes. |
31+
32+
## Built-In Rule Defaults
33+
34+
| Question | Default |
35+
| --- | --- |
36+
| Which built-ins should v1 support? | `required`, `equals`, `notEquals`, `oneOf`, `notOneOf`, `min`, `max`, `range`, `length`, `regex`, `includes`. |
37+
| Should numeric tolerance be supported? | Support absolute tolerance in v1. Defer percent tolerance unless it is easy to include. |
38+
| Are numeric bounds inclusive? | Yes. Bounds are inclusive by default. |
39+
| What does checkbox `equals` mean? | Exact set equality. Also add explicit `containsAll` and `containsAny` rules. |
40+
| How should object-shaped answers be validated? | Validate the normalized stored answer value by default. Allow an optional `path` for object answers. |
41+
42+
## Rule Combination Defaults
43+
44+
| Question | Default |
45+
| --- | --- |
46+
| How are multiple rules combined? | Multiple rules default to `AND`. |
47+
| Are `OR` groups required in v1? | No. Defer `OR` groups until there is a clear use case. |
48+
| Should rules have names? | Yes. Every rule may have an optional `id`; the system can generate one if omitted. |
49+
| Can one response have multiple targets? | Yes. One response can have multiple validation or evaluation targets. |
50+
51+
## Feedback Defaults
52+
53+
| Question | Default |
54+
| --- | --- |
55+
| Where can feedback be configured? | Per rule and per option. Trial-level fallback feedback applies only when no more specific message exists. |
56+
| Which feedback states are built in? | `valid`, `invalid`, `correct`, `incorrect`, and `partial`. |
57+
| Should feedback support templates? | Yes. Templates should support participant answer, expected answer, option label, attempt number, score, and tolerance. |
58+
| Should feedback reveal the correct answer? | Only when explicitly configured. |
59+
| When does feedback appear? | Validation feedback appears after submit or blur. Correctness feedback appears after submit. |
60+
| Should directional numeric feedback be built in? | Yes. Numeric evaluation should support `tooLow`, `tooHigh`, and `withinTolerance`. |
61+
62+
## Progression Defaults
63+
64+
| Question | Default |
65+
| --- | --- |
66+
| Which trial-level progression modes should exist? | `allowContinue`, `blockUntilValid`, `retryUntilCorrect`, and `retryUntilMaxAttempts`. |
67+
| Where is max attempts configured? | Per trial by default, with optional response-level override. |
68+
| What happens after max attempts? | Allow continue, but mark the trial failed unless configured otherwise. |
69+
| How does a training section pass by default? | Percent correct. Default threshold is `100%` unless specified. |
70+
| Is a built-in summary page required for v1? | No. Expose enough state for summaries in v1; a built-in summary page can be a follow-up. |
71+
| Can progression rules apply outside training? | Yes, but documentation should present them as training-focused. |
72+
73+
## Custom Function Defaults
74+
75+
| Question | Default |
76+
| --- | --- |
77+
| How are custom functions provided? | Referenced from approved study or library code, not inline JavaScript strings. |
78+
| Are async functions supported in v1? | No. Custom functions are sync only in v1. |
79+
| What inputs does a function receive? | Answer, response config, trial config, attempt number, prior attempts, and current participant answers. |
80+
| What does a function return? | A structured result such as `{ valid, score, feedback, metadata }` or `{ evaluation, score, feedback, metadata }`. |
81+
| Are custom functions allowed in deployed studies? | Yes, but only through the existing trusted config/library mechanism. Arbitrary inline deployment code is not allowed. |
82+
83+
## Data and Analysis Defaults
84+
85+
| Question | Default |
86+
| --- | --- |
87+
| What answer data is stored? | Final submitted answers plus attempt history when validation, evaluation, feedback, or progression is configured. |
88+
| What result data is stored? | Validation result, evaluation result, feedback id or message shown, rule ids, score, and attempt number. |
89+
| Should blocked invalid attempts be distinguishable? | Yes. Analysts should be able to distinguish blocked invalid attempts from submitted incorrect answers. |
90+
| Should feedback and progression enter provenance? | Yes. Feedback and progression events should be included in provenance tracking. |
91+
92+
## Config Shape Defaults
93+
94+
| Question | Default |
95+
| --- | --- |
96+
| Where does response-level config live? | Under response-level `validation`, `evaluation`, `feedback`, and `progression` fields. |
97+
| Is block-level progression needed? | Yes. Add sequence or block-level `progression` for aggregate training pass/fail behavior. |
98+
| Where does option-level feedback live? | Embedded directly on options by default, with rule references allowed for advanced cases. |
99+
| How is existing right/wrong behavior handled? | Preserve existing behavior by translating it into `evaluation`. |
100+
| Should unsupported rules be rejected by schema? | Yes for clearly unsupported built-ins. Custom functions can target arbitrary values. |
101+
102+
## Participant UX Defaults
103+
104+
| Question | Default |
105+
| --- | --- |
106+
| Where do validation messages appear? | Inline near the response. |
107+
| Where does correctness feedback appear? | In the existing feedback area. |
108+
| Should blocked progression explain itself? | Yes. The UI should explain why Next is unavailable. |
109+
| Should numeric inputs silently clamp? | No. Preserve typed invalid values where feasible and show a warning. |
110+
| Which feedback severities are built in? | `error`, `warning`, `info`, `success`, and `hint`. |
111+
112+
## Draft Config Sketch
113+
114+
This sketch is illustrative. Field names and exact types should be refined during implementation.
115+
116+
```json
117+
{
118+
"response": [
119+
{
120+
"id": "estimate",
121+
"type": "numerical",
122+
"prompt": "Estimate the correlation.",
123+
"validation": [
124+
{
125+
"id": "required-estimate",
126+
"type": "required",
127+
"message": "Enter an estimate before continuing."
128+
},
129+
{
130+
"id": "estimate-range",
131+
"type": "range",
132+
"min": -1,
133+
"max": 1,
134+
"message": "Enter a value between -1 and 1."
135+
}
136+
],
137+
"evaluation": [
138+
{
139+
"id": "target-correlation",
140+
"type": "equals",
141+
"value": 0.62,
142+
"tolerance": 0.05,
143+
"feedback": {
144+
"correct": "Correct.",
145+
"incorrect": "That estimate is outside the acceptable range.",
146+
"tooLow": "Try a higher value.",
147+
"tooHigh": "Try a lower value."
148+
}
149+
}
150+
],
151+
"progression": {
152+
"mode": "retryUntilMaxAttempts",
153+
"maxAttempts": 3
154+
}
155+
}
156+
]
157+
}
158+
```
159+
160+
## Open Design Decisions
161+
162+
The defaults above still leave these higher-level choices to confirm:
163+
164+
- Whether `evaluation` is the right name, or whether the config should use `grading`, `scoring`, or another term.
165+
- Whether percent tolerance belongs in v1.
166+
- Whether `OR` rule groups are needed immediately.
167+
- Whether a built-in training summary page should be part of the first implementation.
168+
- Whether custom functions should use the same mechanism as existing dynamic functions.
169+
- Which exact values custom functions can access without exposing too much participant or study state.
170+
- How much existing right/wrong training config exists and how it should migrate.

‎RESPONSE_ONE_FORM_ARCHITECTURE_PLAN.md‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
# Trial-Level Response Form Architecture Plan
2+
3+
## Goal
4+
5+
Evaluate and plan a larger refactor from one Mantine form per `ResponseBlock` location to one Mantine form per trial/component. This is a potential later change after `codex/response-validation-cleanup` and the smaller trial-validation answer helper follow-up.
6+
7+
The intended benefit is to make the trial answer object a direct product of one form instead of reconstructing it from per-location snapshots.
8+
9+
## Current Shape
10+
11+
Today, each rendered response location owns its own Mantine form:
12+
13+
- `ResponseBlock` for `aboveStimulus`
14+
- `ResponseBlock` for `belowStimulus`
15+
- `ResponseBlock` for `sidebar`
16+
17+
Each block writes a plain snapshot into `trialValidation[identifier][location]`:
18+
19+
```ts
20+
{
21+
valid: boolean,
22+
values: object,
23+
provenanceGraph?: TrrackedProvenance,
24+
}
25+
```
26+
27+
Other code then merges those snapshots for:
28+
29+
- response issue summaries
30+
- feedback and correct-answer checks
31+
- participant answer persistence
32+
33+
## Proposed End State
34+
35+
Create one trial-level response form that owns all response values for the current component. Individual `ResponseBlock` instances render filtered views of that shared form.
36+
37+
Conceptually:
38+
39+
```txt
40+
TrialResponseFormProvider
41+
useAnswerField(allResponsesForTrial)
42+
values for all response locations
43+
validation for all response fields
44+
45+
ResponseBlock aboveStimulus
46+
renders fields from shared form
47+
48+
ResponseBlock belowStimulus
49+
renders fields from shared form
50+
51+
ResponseBlock sidebar
52+
renders fields from shared form
53+
```
54+
55+
The saved answer path could then read from one canonical form snapshot instead of merging three response-block snapshots.
56+
57+
## Potential Benefits
58+
59+
- One canonical `values` object for all responses in a trial.
60+
- One validation call can evaluate all response fields.
61+
- Less answer copying when saving.
62+
- Less duplication between feedback checks, summaries, and persistence.
63+
- Easier reasoning about cross-location response dependencies.
64+
- A clearer future path for field-level validation summaries and submit behavior.
65+
66+
## Tradeoffs
67+
68+
This is a larger ownership change than the immediate follow-up plan.
69+
70+
The current per-location forms map naturally to the rendered UI structure and to per-location provenance. A trial-level form would need a new owner above all response blocks, likely in the component or trial controller layer.
71+
72+
Open questions:
73+
74+
- Where should the trial-level form provider live?
75+
- How should per-location provenance graphs be preserved?
76+
- Should `trialValidation` keep per-location validity, or become a trial-level response validation snapshot?
77+
- How should stimulus validation continue to compose with response validation?
78+
- How should analysis mode read saved provenance and historical values?
79+
- How should hidden responses, reactive responses, matrix answers, ranking answers, and custom responses synchronize with one shared form?
80+
- What happens if a response location is not mounted because of layout or sidebar state?
81+
82+
## Recommended Migration Strategy
83+
84+
Do not replace all per-location forms in one PR. Use staged changes.
85+
86+
### Stage 1: Stabilize Plain Snapshots
87+
88+
Complete the immediate follow-up plan:
89+
90+
- Add one shared helper for constructing the saved trial answer from `trialValidation`.
91+
- Keep per-location forms.
92+
- Keep behavior identical.
93+
94+
This creates a baseline for comparing old and new answer snapshots.
95+
96+
### Stage 2: Extract a Response Form Boundary
97+
98+
Introduce a hook or provider that can own the shared response form without changing rendering behavior yet.
99+
100+
Possible API:
101+
102+
```ts
103+
function useTrialResponseForm({
104+
responses,
105+
currentStep,
106+
storedAnswer,
107+
customResponseValidators,
108+
customResponseLoadErrors,
109+
}: UseTrialResponseFormOptions): UseTrialResponseFormResult;
110+
```
111+
112+
It should own:
113+
114+
- `useAnswerField`
115+
- initial values
116+
- custom response validators
117+
- custom response load errors
118+
- reactive answer merging
119+
- matrix answer merging
120+
- ranking answer merging
121+
122+
At this stage, `ResponseBlock` can still receive the form through props or context and render only its location.
123+
124+
### Stage 3: Move Validation Publishing
125+
126+
Move the Mantine-to-`trialValidation` synchronization out of each `ResponseBlock`.
127+
128+
The publisher should write either:
129+
130+
- one trial-level response validation snapshot, or
131+
- per-location snapshots derived from the same shared form
132+
133+
Prefer preserving the existing `trialValidation` shape at first to reduce migration risk.
134+
135+
### Stage 4: Replace Answer Save Source
136+
137+
Once one trial-level form snapshot exists, update the save path so `useNextStep` reads the canonical trial response answer rather than merging per-location snapshots.
138+
139+
This should happen only after tests prove the form snapshot and current merged answer are identical for representative studies.
140+
141+
### Stage 5: Simplify ResponseBlock
142+
143+
After the shared form and publishing boundary are stable, simplify `ResponseBlock` so it mostly handles rendering:
144+
145+
- filter responses by location
146+
- render `ResponseSwitcher`
147+
- render feedback alerts
148+
- render local layout
149+
150+
Avoid mixing persistence, full-trial answer aggregation, and form ownership back into the component.
151+
152+
## Test Plan
153+
154+
Unit tests:
155+
156+
- Shared form initializes values for all response locations.
157+
- Location-specific `ResponseBlock` renders only its assigned responses.
158+
- Form values update correctly from each location.
159+
- Hidden responses do not create visible errors but preserve expected answer behavior.
160+
- Matrix, ranking, reactive, custom response, `withOther`, and `withDontKnow` cases keep current validation behavior.
161+
- Trial-level answer snapshot matches the current `trialValidation` merged answer.
162+
163+
Integration tests:
164+
165+
- Participant can complete a study with responses above stimulus, below stimulus, and in sidebar.
166+
- Correct-answer feedback reads answers from all locations.
167+
- Next button blocks on unresolved required responses across all locations.
168+
- Saved participant answer matches the pre-refactor shape.
169+
- Analysis mode can replay/read saved answers.
170+
171+
Browser scenarios to request before merge:
172+
173+
- required short text
174+
- checkbox and dropdown min/max
175+
- matrix partial completion
176+
- ranking response
177+
- `Other`
178+
- `I don't know`
179+
- custom response validation
180+
- feedback/check-answer training flow
181+
- responses split across above, below, and sidebar
182+
183+
## Risk Assessment
184+
185+
This is a high-risk refactor because it changes the owner of response form state. It should be treated as an architectural migration, not a cleanup patch.
186+
187+
Highest-risk areas:
188+
189+
- provenance graph preservation
190+
- answer persistence shape
191+
- analysis/replay behavior
192+
- side-channel state for matrix, ranking, and reactive responses
193+
- correct-answer feedback and failed-training progression
194+
- unmounted or hidden response locations
195+
196+
## Recommendation
197+
198+
Use the immediate follow-up PR to remove duplicated answer aggregation first. Then revisit the one-form architecture with current behavior pinned by tests.
199+
200+
The one-form architecture is probably the cleaner end state, but it should only proceed once the project has enough coverage to prove that saved answers, validation gating, feedback, and analysis mode remain unchanged.

0 commit comments

Comments
 (0)