Skip to content

Feature/sig fig#15

Open
m-messer wants to merge 6 commits into
masterfrom
feature/sig_fig
Open

Feature/sig fig#15
m-messer wants to merge 6 commits into
masterfrom
feature/sig_fig

Conversation

@m-messer

Copy link
Copy Markdown
Member

Requires #14

Problem

IsSimilar should also support significant figures and tolerances.

Changes

  • Added a sig_figs (significant_figures) param that rounds both the response and answer to N significant figures and requires them to match, rather than checking a continuous tolerance band.
  • sig_figs is mutually exclusive with atol/rtol — supplying both raises an exception, since a rounding-based check doesn't combine sensibly with the existing additive tolerance formula.
  • Refactored app/evaluation.py: split tolerance and sig-figs logic into dedicated _evaluate_tolerance/_evaluate_sig_figs functions, and extracted shared _is_number/_result helpers to remove duplication. evaluation_function is now just param validation + dispatch.
  • Added 13 new tests covering correct/incorrect matches, negative numbers, zero, invalid sig_figs values, mutual-exclusivity errors, synonym keys, and non-numeric response/answer handling.
  • Updated readme.md, app/docs/user.md, and app/docs/dev.md to document the new param and its exclusivity rule.

Comment thread app/docs/dev.md
Comment thread app/docs/dev.md
}
```

### `atol`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here, etc.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in #14

@peterbjohnson peterbjohnson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major comment:

  • I think we need to sit with Phil and discuss what we mean by sig fig and what the requirements are here. I've made a more detailed note inline in the code.

Minor comment:

  • can update all docs and code to the new parameter names (maybe do a sweep) for atol/rtol?

Comment thread app/evaluation.py

rounded_answer = _round_to_sig_figs(answer, sig_figs)
rounded_response = _round_to_sig_figs(response, sig_figs)
is_correct = abs(rounded_response - rounded_answer) <= spacing(abs(rounded_answer))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to discuss with Phil what we mean by sig figs.

Example:
answer = 0.123
response = 0.1231

Is the response correct to 3 sig fig? Is this a scientific question of expressing a number to 3 sig fig? If so then no, the response is not correct. It should have been rounded to 0.123. If we're asking for the two numbers to be similar after rounding to 3 s.f. that's a different criterion?

Similarly 0.1230 could be interpreted as equal to 0.123, or not equal because it implies a different level of precision. (e.g. https://ccnmtl.columbia.edu/projects/mmt/frontiers/web/chapter_5/6665.html)

I suspect that we need two types of sig fig checking? The computational type and the scientific type? The latter would need to operate on the string before it is converted to a float or integer as a prelude.

I also wonder if we should incorporate feedback when there is a level of known equality (this point applies to the wider isSimilar), e.g. 'correct within the allowed tolerance'.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback, I am happy to discuss with Phil further. However, I think the link you gave provides the detail that I missed in my first implementation. Currently, the implementation doesn't support trailing 0s for decimal numbers as the conversion will truncate them.

I think adding feedback throughout would also be beneficial. I'll make the edits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants