Iterating on Questions to ask of SAP

[TeMeta]

While we are in model discovery stage we want to make sure nothing gets missed:

• Can't be too prescriptive too early
• Can't be too open-ended, need some idea of the types of things we are looking for

Suggestion is to iterate on the questions that we ask of a SAP. By providing a set of examples and definitions as context input we assert a worldview that allows the LLM to respond in kind.

The plan is to parse batches of SAP PDFs from CT.gov (open to suggestions how to select them, but most recent probably makes sense) and then ask questions of them, building up a body of answers to those questions which we can then review and use to feed back into the prompt.

At this early stage we could even experiment with a set of disparate worldviews and see what we get back, since, it could provide useful insights and connections.

[TeMeta]

[Notes from Kirsten]

Give me analysis description text (the text that describes which model applied and which estimates to produce) for primary analysis, as I need to process it for analysis concept tokenisation.

Use STATO to tokenise the text, save results back

Tagging of analysis text.

Given this description of a planned analysis in a clinical trial: The primary analysis of the ADAS-Cog (11) at Week 24 will use the efficacy population with LOCF imputation for any missing values at Week 24. An ANCOVA model will be used with the baseline score, site, and treatment included as independent variables. Treatment will be included as a continuous variable, and results for a test of dose response will be produced. *Interaction terms will not be investigated. Can you tag/mark the words/sentences that would 'map' to an Analysis concept? Elements of an Analysis concept should, if possible, refer to STATO :https://bioportal.bioontology.org/ontologies/STATO?p=classes *

Claude tagging:

Original Text with Analysis Concept Tags

[PRIMARY ANALYSIS] of the [ADAS-Cog (11)] at [Week 24] will use the [EFFICACY POPULATION] with [LOCF IMPUTATION] for any [MISSING VALUES] at Week 24.

An [ANCOVA MODEL] will be used with the [BASELINE SCORE], [SITE], and [TREATMENT] included as [INDEPENDENT VARIABLES].

[TREATMENT] will be included as a [CONTINUOUS VARIABLE], and results for a [TEST OF DOSE RESPONSE] will be produced.

[INTERACTION TERMS] will not be investigated.

…and

Mapping to Analysis Concepts and STATO Terms
Statistical Methods and Models
ANCOVA MODEL → STATO:0000555 "analysis of covariance"
TEST OF DOSE RESPONSE → STATO:0000304 "dose-response modeling"
Data Handling and Population
PRIMARY ANALYSIS → STATO:0000000 "primary analysis" (root statistical analysis concept)
EFFICACY POPULATION → Analysis population specification
LOCF IMPUTATION → STATO:0000405 "last observation carried forward imputation"
MISSING VALUES → STATO:0000402 "missing data"
Variables and Measurements
ADAS-Cog (11) → Primary outcome measure/endpoint
BASELINE SCORE → STATO:0000093 "baseline measurement"
INDEPENDENT VARIABLES → STATO:0000251 "independent variable"
CONTINUOUS VARIABLE → STATO:0000252 "continuous variable"
TREATMENT → STATO:0000258 "treatment variable"
SITE → Stratification/blocking variable
Temporal Elements
Week 24 → Time point specification for analysis
Design Decisions
INTERACTION TERMS → STATO:0000347 "interaction effect" (explicitly excluded)
Analysis Concept Categories

1. Statistical Method Specification
   ANCOVA model selection
   Dose-response testing approach
2. Data Management Strategy
   Population definition (efficacy)
   Missing data handling (LOCF)
3. Variable Definition and Role
   Outcome variable (ADAS-Cog)
   Covariates (baseline, site)
   Treatment parameterization
4. Analysis Scope and Limitations
   Primary analysis designation
   Explicit exclusion of interactions
   Specific time point (Week 24)

---

Then the question used for creating the example AC based on the above tagged text:

I am trying to make an example of an Analysis Concept for a clinical trial analysis text tagged above. I would like to separate display/visualisation from the analysis data - so disregard the display only return the analysis part.

Please return the analysis concept as a json file I can load into https://jsoncrack.com/editor.

---

If we get a set of json files for a sample of analyses (ANOVAs as above), then perhaps we can come up with an AC common structure and combine the above prompt with the description of the AC as we discussed as well. An iterative way of coming to a structure for AC templates.