parse_pubmed_paragraph does not propagate parent section titles for nested <sec>

[wzhou122]

Hi,

I am trying to use the parse_pubmed_paragraph function to find section names and text and use a regex expression to find methods sections based on section name. However, looking into on the currently implementation, I found it might not do the thing that I expected.

I just want to double check that if I understand it correctly.

Currently, the parse_pubmed_paragraph function in pubmed_oa_parser.py uses paragraph.find("../title") which only goes up one level to get the immediate parent section. This means paragraphs in nested subsections (e.g., Methods > Step 1 > Step 2) would only get "Step 2" as their section name, not "Methods", causing them to be missed when filtering for "Methods" sections.

For example, given the below xml

<title>Methods</title>

Direct paragraph in Methods

<title>Step 1</title>

Paragraph in Methods > Step 1

<title>Step 2</title>

Paragraph in Methods > Step 1 > Step 2

<title>Results</title>

Paragraph in Results

Paragraph in "Methods" should have section = "Methods" ✓
Paragraph in "Step 1" should have section = "Step 1" (missing parent "Methods") ✗
Paragraph in "Step 2" should have section = "Step 2" (missing parents "Methods > Step 1") ✗

Please let me know if I am understanding it correctly. Thanks!

[Prunoideae]

Yes, this does not handle nested sections, which could be annoying. For a quick fix, I used Gemini to generate some XML parsing code that can handle:

import xml.etree.ElementTree as ET
import re
from typing import Dict, List, Union

# Define the recursive type for our parsed document
DocumentType = Dict[str, List[Union["DocumentType", str]]]


def clean_text(element: ET.Element) -> str:
    """Extracts and cleans text from an XML element and all its sub-elements."""
    # .itertext() grabs all text, even inside nested tags like <italic> or <xref>
    raw_text = "".join(element.itertext())
    # Remove extra spaces, newlines, and tabs created by XML formatting
    return re.sub(r"\s+", " ", raw_text).strip()


def parse_sec(sec_elem: ET.Element) -> DocumentType:
    """Recursively parses a <sec> element into a dictionary."""
    # Find the title of the section
    title_elem = sec_elem.find("./title")
    title = clean_text(title_elem) if title_elem is not None else "Untitled Section"

    contents = []

    for child in sec_elem:
        if child.tag == "p":
            p_text = clean_text(child)
            if p_text:
                contents.append(p_text)
        elif child.tag == "sec":
            # Recursively parse nested sections
            contents.append(parse_sec(child))

    return {title: contents}


def parse_oa_xml(xml_string: str) -> DocumentType:
    """Parses standard Pubmed OA XML into a nested dictionary structure."""
    root = ET.fromstring(xml_string)

    # We only care about the body element content for the text
    body = root.find(".//body")
    if body is None:
        return {}

    result = {}

    # Process all root-level sections in the body
    for sec in body.findall("./sec"):
        parsed_section = parse_sec(sec)
        result.update(parsed_section)

    return result


# --- Example usage ---
if __name__ == "__main__":
    xml_data = open("sample_pmc.xml", "r").read()

    import json

    parsed_doc = parse_oa_xml(xml_data)
    # Print pretty JSON representation
    print(json.dumps(parsed_doc, indent=4))

And tested on real PubMed articles:

{
    "Materials and Methods": [
        {
            "Sediment Sampling": [
                "The sampling was carried out onboard the research vessel R/V Seiyo Maru in early June 2017 at Sagami Bay, Japan (35\u00b002\ua78c52\ua78c\ua78cN 139\u00b019\ua78c46\ua78c\ua78cE)...",
                "Pore water sampling (see below for details) from two cores (SB4 and SB5) was carried out immediately after sampling, on board the research vessel (Table 1). ..."
            ]
...

Though this does not handle things like references (because I use them for my own purposes), you might need to modify the code to add more functionality.