Revise LLM-assisted extraction project details (#152)

dlebauer · web-flow · commit dede67634dc1 · 2026-02-16T18:00:22.000-05:00
* Revise LLM-assisted extraction project details

Updated project description for LLM-assisted data extraction, add detail on methodology and expected outcomes.

* minor changes

Removed redundant text and clarify.

* Change difficulty rating to High

Updated difficulty rating to High.

* add db links
diff --git a/src/pages/gsoc_ideas.mdx b/src/pages/gsoc_ideas.mdx
@@ -177,13 +177,17 @@ Medium (175hr) or Large (350 hr) depending on number of deliverables
 Medium
 
 ---
-### 5. LLM-Assisted Extraction of Agronomic Experiments into BETYdb{#llm-betydb}
 
-Manual extraction of agronomic and ecological experiments from scientific literature into BETYdb is slow, error-prone, and labor-intensive. Researchers must interpret complex experimental designs, reconstruct management timelines, identify treatments and controls, handle factorial structures, and link outcomes with correct covariates and uncertainty estimates—tasks that require scientific judgment beyond simple text extraction. Current manual workflows can take hours per paper and introduce inconsistencies that compromise downstream data quality and meta-analyses.
+### 5. LLM-Assisted Extraction of Agronomic and Ecological Experiments into Structured Data {#llm-betydb}
 
-This project proposes a human-supervised, LLM-based system to accelerate BETYdb data entry while preserving scientific rigor and traceability. The system will ingest PDFs of scientific papers and produce upload-ready BETYdb entries (sites, treatments, management time series, traits, and yields) with every field labeled as extracted, inferred, or unresolved and linked to provenance evidence in the source document. The system leverages existing labeled training data (scientific papers with ground-truth BETYdb entries).
+Manual extraction of agronomic and ecological experiments from scientific literature into a structured format that can be used to calibrate and validate models is slow, error-prone, and labor-intensive. Researchers must interpret complex experimental designs, reconstruct management timelines, identify treatments and controls, handle factorial structures, and link outcomes with correct covariates and uncertainty estimates. Data are often reported as summary statistics (for example mean and standard error) in text, tables, or figures and require additional context from disturbance or management time series. These tasks require scientific judgment beyond simple text extraction.
+Current manual workflows can take hours per paper and introduce inconsistencies that compromise downstream data quality and meta-analyses.
 
-The architecture follows a two-layer design: (1) a schema-validated intermediate representation (IR) preserving evidence links, confidence scores, and flagged conflicts, and (2) a BETYdb materialization layer that enforces BETYdb semantics, validation rules, and generates upload-ready CSVs or API payloads with full audit trails. Implementation is flexible—ranging from agentic LLM workflows to fine-tuned specialist models to an adaptive hybrid—and should be informed by empirical evaluation during the project.
+This project proposes a human-supervised, LLM-based system to accelerate data extraction while preserving scientific rigor and traceability. It will leverage existing labeled training data (scientific papers with ground‑truth entries), including aligned PDF‑to‑structured‑data records from [BETYdb](https://betydb.org) and [ForC](https://forc-db.github.io/index.html), which represent expert‑curated, production‑quality datasets. Combined, these resources include over 80,000 plant and ecosystem observations from more than 1,000 sources, providing high-quality supervision for extraction from text, tables, and figures. Evaluation should include held-out, out-of-sample papers. The system will ingest PDFs of scientific papers and produce tables compatible with the [spreadsheet used to upload data to BETYdb](https://docs.google.com/spreadsheets/d/e/2PACX-1vSAa7jBHSaas-bH0ARxQjVLKhz3Iq03t97wrxMZrgVVi98L5bYQi5ZUC0b57xIZBlHEkPH9qYf22xQS/pubhtml) (sites, treatments, management time series, traits+yields bulk upload table) with every field labeled as extracted, inferred, or unresolved and linked to provenance evidence in the source document.
+
+The architecture follows a two-layer design: (1) a schema-validated intermediate representation (IR) preserving evidence links, confidence scores, and flagged conflicts, and (2) a materialization layer that enforces semantics, validation rules, and generates upload-ready CSVs or API payloads with full audit trails. Implementation is flexible—ranging from agentic LLM workflows to fine-tuned specialist models to an adaptive hybrid—and should be informed by empirical evaluation during the project.
+
+Implementation is flexible—ranging from agentic LLM workflows to fine‑tuned specialist models to an adaptive hybrid—and should be informed by empirical evaluation during the project.
 
 **Expected outcomes:**
 
@@ -194,15 +198,15 @@ A successful project would complete the following tasks:
 * Independent validators for BETYdb semantics, unit consistency, temporal logic, and required fields
 * BETYdb export module producing upload-ready management CSVs and bulk trait upload formats with full provenance preservation
 * Scientist-in-the-loop review interface for approving, correcting, or rejecting extracted entries with inline evidence and confidence scores
-* Evaluation harness with automated metrics for extraction accuracy, inference quality, coverage, and time savings on held-out test papers
+* Evaluation harness with automated metrics for extraction accuracy, inference quality, coverage, and time savings relative to manual curation on held‑out test papers
 * Documentation covering IR schema specification, developer guidance for adding new extraction components, and user guidance for the review interface
 
 **Prerequisites:**
 
-- Required: R Shiny, Python (familiarity with scientific literature and experimental design concepts)
-- Helpful: experience with LLM APIs (Anthropic, OpenAI) or fine-tuning frameworks, knowledge of BETYdb schema and workflows, familiarity with agronomic or ecological experimental designs
+- Required: Python; familiarity with natural language processing, information extraction, and machine learning
+- Helpful: experience with LLM APIs and fine-tuning frameworks, knowledge of BETYdb schema and workflows, familiarity with scientific writing and agronomic or ecological experimental design/analysis
 
-**Contact person:**
+**Contact persons:**
 
 Nihar Sanda (@koolgax99), David LeBauer (@dlebauer)
 
@@ -212,123 +216,4 @@ Large (350 hr)
 
 **Difficulty:**
 
-Medium to High
-
-<!--
-
-
-# This comment section for ideas that may be potentially viable in future (with revision)
-
-
----
-
-### 4. Development of Notebook-based PEcAn Workflows{#notebook}
-
-The PEcAn workflow is currently run using either a web based user interface, an API, or custom R scripts. The web based user interface is easiest to use, but has limited functionality whereas the custom R scripts and API are more flexible, but require more experience.
-
-This project will focus on building Quarto notebooks that provide an interface to PEcAn that is both welcoming to new users and flexible enough to be a starting point for more advanced users. It will build on existing [Pull Request 1733](https://github.com/PecanProject/pecan/pull/1733).
-
-**Expected outcomes:**
-
-- Two or more template workflows for running the PEcAn workflow. 
-- Written vignette and video tutorial introducing their use.
-
-**Prerequisites:**
-
-- Familiarity with R. 
-- Familiarity with R studio and Quarto or Rmarkdown is a plus.
-
-**Contact person:**
-David LeBauer @dlebauer, Nihar Sanda @koolgax99
-
-**Duration:**
-Medium (175hr)
-
-**Difficulty:**
-Medium
-
-
-#### BETYdb R data package
-
-BETYdb's web front end is built on a version of Ruby on Rails that is functional byt no longer supported. A key feature of BETYdb is that the data is open and accessible. 
-
-Building an R data package would make the Trait and Yield data currently in BETYdb more accessible to users beyond the PEcAn community.
-
-**Expected outcomes:**
-
-A successful project would complete a subset of the following tasks:
-
-- An R package containing the data currently hosted in BETYdb.
-- Documentation and examples of use.
-- Updates to BETYdb documentation.
-
-**Prerequisites:**
-
-- Required: R
-- Helpful: R package development; familiarity with relational databases and SQL.
-
-**Contact person:**
-
-David LeBauer (@dlebauer)
-
-**Duration:**
-
-Medium (175hr) to Large (350hr) depending on scope of proposal.
-
-**Difficulty:**
-
-Medium
-
----
-
-#### [Optimize PEcAn for freestanding use of single packages [R package development]](#freestanding)
-
-PEcAn was designed as a system of independent modules, each implemented as its own R package that was intended to be usable either standalone or as part of the full PEcAn system. Subsequent development focused on the most common cross-module workflows has lead to tighter coupling between modules than was originally intended, so that in practice many of the modules are now challenging to use, test, or develop without a full understanding of their interdependencies. Further, some packages expect inputs and outputs in data structures that are only generated by other PEcAn packages but might be more easily provided in standard well-known formats. We seek proposals to re-loosen these couplings by revisiting the design and interface of PEcAn packages through one or more of:
-
-1. Refactoring code to remove unneeded dependencies, simplify package interfaces, and exchange data in standard formats
-2. Identifying exported functions that are not core to the functionality of the package, and removing them or making them internal
-3. Writing tests and examples that demonstrate freestanding use of the package
-4. Developing methods for tracking the dependencies between packages that cannot be eliminated, including how these change between package versions
-   Proposals for this project should choose a subset of these approaches and apply them to a specified subset of the PEcAn packages. Strong proposals will clearly show why each chosen package should be a priority, how it will become more independent at the completion of the project, and what interface changes will be needed to achieve this.
-
-**Expected outcome:**
-
-- One or more PEcAn packages can be installed, used, and/or tested without the user needing to know [something previously important] about [another package].
-
-**Prerequisites:**
-
-- Familiarity with R, especially how it manages dependencies between packages, and with concepts of software package development. Helpful resources: [rOpenSci packages](https://devguide.ropensci.org/index.html) and [R packages](https://r-pkgs.org). Experience with multi-package code bases will be very helpful.
-
-**Contact person:**
-Chris Black @infotroph, Shashank Singh @moki1202
-
-**Duration:**
-Flexible to work as either a Small (175hr) or Large (350 hr)
-
-**Difficulty:**
-Medium, Large
----
-
-#### [PEcAn model coupling and development [Data Science]](#coupling)
-
-PEcAn has the capability to interface multiple ecological models. The goal of this project is to improve the coupling of existing models to PEcAn (specifically FATES) and add new models (specifically a simple vegetation model that is under development). It is also possible to contribute to the development of the simple vegetation model which is written in Fortran.
-
-**Expected outcome:**
-
-- New or improved PEcAn model packages.
-
-**Prerequisites:**
-
-- R, Fortran is an advantage.
-
-**Contact person:**
-Hui Tang @Hui Tang, Istem Fer @istfer
-
-**Duration:**
-Flexible to work as either a Small (175hr) or Large (350 hr)
-
-**Difficulty:**
-Medium
-
----
--->
+High
