Skip to content

Add text validation and choice consistency checks#11

Merged
rsh52 merged 1 commit into
mainfrom
codex/metadata-defined-validations
Jul 16, 2026
Merged

Add text validation and choice consistency checks#11
rsh52 merged 1 commit into
mainfrom
codex/metadata-defined-validations

Conversation

@rsh52

@rsh52 rsh52 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

This update expands build_quality_report() with metadata-driven validation findings, corrects several false-positive scenarios, and improves report execution performance.

Changes

  • Added invalid_choice_value findings for radio, dropdown, Yes/No, and True/False fields using raw REDCap values and metadata-defined choice codes.
  • Added invalid_validation_format findings for supported text validation types configured through text_validation_type_or_show_slider_number.
  • Enforced user-configured text_validation_min and text_validation_max bounds through outside_validation_range.
  • Added strict validation for numeric, date, datetime, time, email, and phone formats.
  • Accepted numeric decimals without a leading zero, such as .850 and -.850.
  • Updated date and datetime validation to reflect REDCap API behavior:
    • Dates use canonical YYYY-MM-DD.
    • Datetimes use canonical YYYY-MM-DD HH:MM or YYYY-MM-DD HH:MM:SS.
    • DMY, MDY, and YMD metadata suffixes are treated as data-entry display settings.
  • Prevented malformed temporal values from also producing range or future-date findings.
  • Removed orphaned_branching_reference because valid cross-form, event-qualified, and smart-variable references could not be reliably distinguished from missing references.
  • Excluded checkbox fields from invalid_choice_value:
    • Checkbox metadata codes continue to identify exported option columns.
    • Checkbox cells remain separate 0/1 values.
    • Existing checkbox_no_values_selected and checkbox_none_with_other findings remain.
  • Ensured REDCap record imports explicitly request raw field headers, raw choice values, and raw checkbox values.

Performance

  • Vectorized text-validation parsing instead of parsing each cell independently.
  • Reused parsed validation values across format, range, IQR, and future-date checks.
  • Cached form applicability and represented-field calculations within each report.
  • Parsed choice metadata once per report.
  • Vectorized and cached derived checkbox parent values used by field and record summaries.
  • Reduced a local validation-heavy benchmark from approximately 8.4 seconds to 0.8–1.1 seconds.
  • Reduced a checkbox-heavy benchmark from approximately 5.7 seconds to 2.1 seconds.

Documentation

  • Updated build_quality_report() documentation with supported validation behavior.
  • Expanded the quality-report article to explain:
    • Metadata-driven validation rules and bounds.
    • Canonical REDCap API date and datetime formats.
    • Leading-decimal numeric values.
    • Checkbox choice handling.
    • Removal of branching-reference findings.
  • Regenerated roxygen documentation, pkgdown output, and mock report fixtures.

Testing

  • Added regression coverage for:
    • Supported choice types and invalid codes.
    • Leading-decimal numeric values.
    • Numeric precision and comma-decimal formats.
    • Canonical date and datetime exports across metadata display orders.
    • Temporal bounds and future-date findings.
    • Unsupported or unconfigured text validations.
    • Checkbox exclusion from invalid-choice findings.
    • Cross-form, event-qualified, smart-variable, and missing branching references.
    • Repeating-event and repeating-instrument finding context.
  • Completed full test suite, Air, styler, lintr, pkgdown build, and package check.
  • R CMD check: 0 errors, 0 warnings, 0 notes.

Implements metadata-driven validation for REDCap text fields and choice values.

Adds new validation.R module with parsers for supported text validation types: integers, decimals, dates, datetimes, times, emails, and phone numbers. Consistency checks now detect invalid choice codes in radio, dropdown, Yes/No, and True/False fields, and invalid formats in validated text fields.

Enhances data import to request raw codes with type guessing disabled, ensuring choice values like '0' and '1' remain character rather than being coerced to logical. Adds caching for validation parsing and form applicability to improve performance.

Removes orphaned_branching_reference check in favor of stricter validation and choice enforcement. Updates quality report documentation and test coverage.
@rsh52 rsh52 added bug Something isn't working documentation Improvements or additions to documentation labels Jul 16, 2026
@rsh52 rsh52 self-assigned this Jul 16, 2026
@rsh52
rsh52 marked this pull request as ready for review July 16, 2026 16:47
@rsh52
rsh52 merged commit bc6a1cd into main Jul 16, 2026
3 checks passed
@rsh52
rsh52 deleted the codex/metadata-defined-validations branch July 16, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant