Update to process ASA24 dietary records

Polyphenol Estimator can now process prospective dietary records, including multi-day records or multiple one-day records.

Updates also clarify packages for functions used.

Author: SWi1

README.md

@@ -1,6 +1,6 @@
 # Polyphenol Estimator
 
-This repository contains scripts to automate the estimation of dietary polyphenol intake and calculation of the dietary inflammatory index from ASA24 or NHANES diet recalls. 
+This repository contains scripts to automate the estimation of dietary polyphenol intake and calculation of the dietary inflammatory index from ASA24 recalls, ASA24 records, and NHANES WWEIA recalls.
 
 ### Releases
 - November 20, 2025 - Tutorial Draft Release
@@ -21,7 +21,7 @@ Want to estimate polyphenols in your dietary data? Please review our start-up gu
         
 | Inputs      | Provided | About   |
 |------------ |--------- |---------|
-| Diet Data  | No     	| ASA24 Items File, or NHANES<br>Note: Current pipeline requires each participant to have at least two recalls. 	|
+| Diet Data  | No     	| ASA24 Items File, or NHANES<br>Note: Current pipeline requires each participant to have at least two recalls, records, or record days. 	|
 | FDA Food Disaggregation Database V 3.1   | Yes  |   FDA's Food Disaggregation Database contains Ingredients and their percentages within FNDDS food codes. |
 | FooDB food polyphenol content  | Yes      	|   Contains polyphenol content in foods. Polyphenols were determined based off structure (an aromatic ring with at least two hydroxyl groups) with 9 compounds manually added to better reflect microbial enzyme substrates.   |
 | FooDB polyphenol list   | Yes      	| List of 3072 polyphenols. File includes FooDB compound ID, compound name, SMILES, InChI key, and taxonomic class. Taxonomic class is from ClassyFire, an automated chemical taxonomic classification application based on chemical structure.  |

_config.yml

@@ -1,5 +1,5 @@
 title: Polyphenol Estimator
-description: A tutorial on how to obtain estimates of polyphenol intake and DII from diet recall data.
+description: A tutorial on how to obtain estimates of polyphenol intake and DII from diet recall or record data.
 theme: just-the-docs
 markdown: kramdown
 

functions/calculate_DII.R

@@ -17,7 +17,7 @@ calculate_DII = function(report = c("none", "html", "md")) {
 
   # The Polyphenol Estimation Pipeline Needs to run first.
   # Output from this pipeline kicks off the DII calculation scripts
-  starting_file = "outputs/Recall_Disaggregated_mapped.csv.bz2"
+  starting_file = "outputs/Diet_Disaggregated_mapped.csv.bz2"
 
   # Check if it was by confirming Disaggregated Dietary Data File exists
   if (!file.exists(starting_file)) {

functions/estimate_polyphenols.R

@@ -59,16 +59,34 @@ estimate_polyphenols = function(
     stop("The diet data file must contain the column '", id_var, "' for ", type, ".")
   }
 
-  # Obtain recall counts by user
-  recalls_per_user = diet_dat %>%
-    group_by(.data[[id_var]]) %>%
-    summarise(n_recalls = n_distinct(RecallNo), .groups = "drop")
+  # Determine if Recall or Record
+  if ("RecallNo" %in% names(diet_dat)) {
+    # Recall dataset
+    n_events = diet_dat %>%
+      group_by(.data[[id_var]]) %>%
+      summarise(n_events = n_distinct(RecallNo), .groups = "drop")
+    
+  } else if ("RecordNo" %in% names(diet_dat) || "RecordDayNo" %in% names(diet_dat)) {
+    # Record dataset
+    n_events = diet_dat %>%
+      group_by(.data[[id_var]]) %>%
+      summarise(
+        n_events = max(
+          n_distinct(RecordNo %||% 0),
+          n_distinct(RecordDayNo %||% 0)
+        ),
+        .groups = "drop"
+      )
+    
+  } else {
+    stop("Dataset has neither RecallNo nor RecordNo/RecordDayNo columns.")
+  }
 
   # Recall Status Message
-  if (max(recalls_per_user$n_recalls, na.rm = TRUE) < 2) {
-    stop("The diet recall file does not contain multiple recalls per participant.")
+  if (max(n_events$n_events, na.rm = TRUE) < 2) {
+    stop("Your diet file does not contain multiple recalls or records per participant.")
   } else {
-    message("Multiple recalls detected across participants.\n")
+    message("Multiple recalls or records detected across participants.\n")
   }
 
   ##########################################

index.md

@@ -6,7 +6,7 @@ nav_order: 1
 
 # Polyphenol Estimator
 
-This start guide shows you how to take your ASA24 or NHANES dietary data and estimate polyphenol intake using the [FooDB](https://foodb.ca/) and calculate the dietary inflammatory index[^1]. Example ASA24 data, borrowed from the DietDiveR Repository[^2], is provided for you to test. Check out [the example file here](https://github.com/SWi1/polyphenol_pipeline/blob/main/user_inputs/VVKAJ_Items.csv) to see the input structure required for Polyphenol Estimator.
+This start guide shows you how to take your ASA24 or NHANES dietary data and estimate polyphenol intake using the [FooDB](https://foodb.ca/) and calculate the dietary inflammatory index [(Shivapppa et al. 2013)](10.1017/S1368980013002115). Example ASA24 data, borrowed from the [DietDiveR Repository](https://computational-nutrition-lab.github.io/DietDiveR/), is provided for you to test. Check out [the example file here](https://github.com/SWi1/polyphenol_pipeline/blob/main/user_inputs/VVKAJ_Items.csv) to see the input structure required for Polyphenol Estimator.
 
 ### 1. Download the entire repository directly [here](https://github.com/SWi1/polyphenol_pipeline/archive/refs/heads/main.zip) then unzip the folder. 
 The repository contains files and scripts used in the tutorial.
@@ -28,16 +28,12 @@ Find a list of expected outputs below:
 <details>
 <summary>Reports: See What's in Each Script</summary>
   <ul>
-  For every script that was run, a report was generated in the reports folder. 
-  This online tutorial actually shows you what the reports look like if you navigate to pages under "Polyphenol Estimator" and "DII Calculation" in your sidebar.
+  If you opt to generate md or html reports, then a readable report of each script used will be placed into your reports folder. You can preview the latest reports generated by navigating to pages under "Polyphenol Estimator" and "DII Calculation" in your sidebar. Report generation may be helpful to keep a record of which scripts were used as the tool periodically will be updated.
   </ul>
 </details>
 
 ### Want to test NHANES data instead?
 `estimate_polyphenols` can also be run on NHANES WWEIA data. To generate NHANES WWEIA data, follow the instructions in ["Preparing Diet Data - NHANES diet recalls"](https://swi1.github.io/polyphenol_pipeline/webpages/preparing_diet_data_NHANES.html#prepare-nhanes-diet-recall-data). When you've finished: 
 1. Come back to run_pipeline.R and update `diet_input_file` with the NHANES output file name. 
 2. In `estimate_polyphenols`, change type to "NHANES"
-3. Run the script. 
-
-[^1]: [Shivapppa et al. 2013. Designing and developing a literature-derived, population-based dietary inflammatory index](10.1017/S1368980013002115)
-[^2]: [DietDiveR Repo](https://computational-nutrition-lab.github.io/DietDiveR/).
+3. Run the script. 

scripts/DII_STEP1_Eugenol.Rmd

@@ -1,7 +1,7 @@
 ---
 title: "DII_PREP - Eugenol"
 author: "Stephanie Wilson"
-date: "November 2025"
+date: "February 2026"
 output:
   md_document:
     variant: gfm
@@ -18,16 +18,16 @@ output:
 ---
 
 ## Calculate Eugenol Intake
-This script takes in your disaggregated dietary data and FooDB-linked descriptions to calculate eugenol intake per recall and subject.
+This script takes in your disaggregated dietary data and FooDB-linked descriptions to calculate eugenol intake per recall (or record) and subject.
 
 #### INPUTS
 
--   **Recall_Disaggregated_mapped.csv.bz2** - Disaggregated dietary data, mapped to FooDB foods, From Step 2 of the polyphenol estimation pipeline
+-   **Diet_Disaggregated_mapped.csv.bz2** - Disaggregated dietary data, mapped to FooDB foods, From Step 2 of the polyphenol estimation pipeline
 -   **FooDB_Eugenol_Content_Final.csv** - Eugenol Content for foods in FooDB, Provided File
 
 #### OUTPUTS
 
--   **Recall_DII_eugenol_by_recall.csv**: Sum eugenol content for each participant recall
+-   **Diet_DII_eugenol_by_entry.csv**: Sum eugenol content for each participant recall or record
 
 ## SCRIPT
 
@@ -54,16 +54,30 @@ Load data
 source("provided_files.R")
 
 # Load Dietary data that has been disaggregated and connected to FooDB
-input_mapped = vroom::vroom('outputs/Recall_Disaggregated_mapped.csv.bz2', 
+input_mapped = vroom::vroom('outputs/Diet_Disaggregated_mapped.csv.bz2', 
                             show_col_types = FALSE)
 
 # Eugenol Content in FooDB
 # Note: Eugenol doesn't have retention factors from Phenol Explorer
 eugenol = vroom::vroom(FooDB_eugenol, show_col_types = FALSE) %>%
-  select(-c(source_type, food_name:orig_source_name))
+  dplyr::select(-c(source_type, food_name:orig_source_name))
 ```
 
 
+### Specify grouping variables
+Column grouping depends on whether output is from a record or recall.
+```{r}
+if ("RecallNo" %in% names(input_mapped)) {
+  group_vars = c("subject", "RecallNo")
+  
+} else if ("RecordNo" %in% names(input_mapped)) {
+  group_vars = c("subject", "RecordNo", "RecordDayNo")
+  
+} else {
+  stop("Data must contain RecallNo or RecordNo.")
+}
+```
+
 ### Merge FooDB-matched Ingredient Codes to FooDB Eugenol Content File.
 
 - Link between FooDB Polyphenol Content and code-matched data is *food_id*.
@@ -73,20 +87,22 @@ input_mapped_content = input_mapped %>%
   # Bring in the Polyphenol Content
   dplyr::left_join(eugenol, by = 'food_id') %>%
   # Remove content that is NA
-  filter(!is.na(orig_content_avg)) %>%
+  dplyr::filter(!is.na(orig_content_avg)) %>%
   # Calculate eugenol amount consumed in milligrams
-  mutate(eugenol_mg = (orig_content_avg * 0.01) * FoodAmt_Ing_g) %>%
-  # Sum eugenol content by subject and recall
-  group_by(subject, RecallNo) %>%
-  # Calculate EUGENOL summed per recall, and rename for dietaryindex function
-  mutate(EUGENOL= sum(eugenol_mg)) %>%
-  ungroup() %>%
-  select(c(subject, RecallNo, EUGENOL)) %>%
+  dplyr::mutate(eugenol_mg = (orig_content_avg * 0.01) * FoodAmt_Ing_g) %>%
+  # Recall - Sum by Subject, Recall
+  # Record - Sum by Subject, Record Number, Day in Record Number
+  dplyr::group_by(across(all_of(group_vars))) %>%
+  # Calculate EUGENOL summed per recal or record
+  # with rename for dietaryindex function
+  dplyr::mutate(EUGENOL = sum(eugenol_mg)) %>%
+  dplyr::ungroup() %>%
+  dplyr::select(c(subject, any_of(c("RecallNo", "RecordNo", "RecordDayNo")), EUGENOL)) %>%
   # Keep distinct entries
-  distinct(subject, RecallNo, .keep_all = TRUE)
+  dplyr::distinct(across(all_of(group_vars)), .keep_all = TRUE)
 ```
 
 ### Export Eugenol Intake File for DII Calculation
 ```{r}
-vroom::vroom_write(input_mapped_content, 'outputs/Recall_DII_eugenol_by_recall.csv', delim = ",")
+vroom::vroom_write(input_mapped_content, 'outputs/Diet_DII_eugenol_by_entry.csv', delim = ",")
 ```

scripts/DII_STEP2_Polyphenol_Subclass.Rmd

@@ -1,7 +1,7 @@
 ---
 title: "DII_PREP - Polyphenol Subclasses"
 author: "Stephanie Wilson"
-date: "November 2025"
+date: "February 2026"
 output:
   md_document:
     variant: gfm
@@ -18,16 +18,16 @@ output:
 ---
 
 ## Calculate DII Polyphenol Subclasses
-This script takes your data that has been mapped to FooDB polyphenol content, extracts compounds categorized under the six required DII subclasses (flavan-3-ols, Flavones, Flavonols, Flavonones, Anthocyanidins, Isoflavones), and calculates the total intake of these subclasses per participant recall. 
+This script takes your data that has been mapped to FooDB polyphenol content, extracts compounds categorized under the six required DII subclasses (flavan-3-ols, Flavones, Flavonols, Flavonones, Anthocyanidins, Isoflavones), and calculates the total intake of these subclasses per participant recall or record. 
 
 #### INPUTS
 
--   **Recall_FooDB_polyphenol_content.csv.bz2**: Disaggregated dietary data, mapped to FooDB polyphenol content, at the compound-level
+-   **Diet_FooDB_polyphenol_content.csv.bz2**: Disaggregated dietary data, mapped to FooDB polyphenol content, at the compound-level
 -   **FooDB_DII_polyphenol_list.csv** - Polyphenols under the six polyphenol subclasses required for DII-2014, Provided File
 
 #### OUTPUTS
 
--   **Recall_DII_subclass_by_recall.csv**: Sum DII polyphenol subclass content for each participant recall
+-   **Diet_DII_subclass_by_entry.csv**: Sum DII polyphenol subclass content for each participant recall or record
 
 ## SCRIPT
 
@@ -54,20 +54,33 @@ Load data
 source("provided_files.R")
 
 # Load dietary data mapped to polyphenol content
-input_polyphenol_content = vroom::vroom('outputs/Recall_FooDB_polyphenol_content.csv.bz2',
+input_polyphenol_content = vroom::vroom('outputs/Diet_FooDB_polyphenol_content.csv.bz2',
                                         show_col_types = FALSE)
 
-# Eugenol Content in FooDB
-# Note: Eugenol doesn't have retention factors from Phenol Explorer
+# Polyphenol classifications
 subclasses = vroom::vroom(FooDB_DII_subclasses, show_col_types = FALSE)
 ```
 
 ### Filter in relevant DII subclass polyphenols
 ```{r}
 input_polyphenol_content_filtered = input_polyphenol_content %>%
-  filter(compound_public_id %in% subclasses$compound_public_id) %>%
+  dplyr::filter(compound_public_id %in% subclasses$compound_public_id) %>%
   # Merge the class information so subclasses are grouped correctly
-  left_join(subclasses)
+  dplyr::left_join(subclasses)
+```
+
+### Specify grouping variables
+Column grouping depends on whether output is from a record or recall.
+```{r}
+if ("RecallNo" %in% names(input_polyphenol_content_filtered)) {
+  group_vars = c("subject", "RecallNo", "component")
+  
+} else if ("RecordNo" %in% names(input_polyphenol_content_filtered)) {
+  group_vars = c("subject", "RecordNo", "RecordDayNo", "component")
+  
+} else {
+  stop("Data must contain RecallNo or RecordNo.")
+}
 ```
 
 
@@ -76,18 +89,20 @@ The column `component` contains the DII category.
 
 ```{r}
 subclass_intakes = input_polyphenol_content_filtered %>%
-  group_by(subject, RecallNo, component) %>%
+  dplyr::group_by(across(all_of(group_vars))) %>%
   # Sum polyphenol category intake, mg by recall
-  mutate(component_sum = sum(pp_consumed)) %>%
-  ungroup() %>%
+  dplyr::mutate(component_sum = sum(pp_consumed)) %>%
+  dplyr::ungroup() %>%
   # Keep distinct entries
-  distinct(subject, RecallNo, component, .keep_all = TRUE) %>%
+  dplyr::distinct(across(all_of(group_vars)), .keep_all = TRUE) %>%
   # Minimize the number of columns for pivoting to wide format
-  select(c(subject, RecallNo, component, component_sum)) %>%
+  dplyr::select(c(subject,
+                  any_of(c("RecallNo", "RecordNo", "RecordDayNo")),
+                  component, component_sum)) %>%
   # pivot to wide version
-  pivot_wider(names_from = component, values_from = component_sum) %>%
+  tidyr::pivot_wider(names_from = component, values_from = component_sum) %>%
   # Rename the columns to match the DII category names in the DietaryIndex function
-  rename(
+  dplyr::rename(
     ISOFLAVONES = Isoflavones,
     "FLA3OL" = "Flavan-3-ols",
     "FLAVONES" = "Flavones",
@@ -98,6 +113,6 @@ subclass_intakes = input_polyphenol_content_filtered %>%
 
 ### Export Polyphenol Subclass Intakes for DII Calculation
 ```{r EXPORT}
-vroom::vroom_write(subclass_intakes, 'outputs/Recall_DII_subclass_by_recall.csv', delim = ",")
+vroom::vroom_write(subclass_intakes, 'outputs/Diet_DII_subclass_by_entry.csv', delim = ",")
 ```