Skip to content

pdf_fetching_capability.md

Latest commit

 

History

History
383 lines (283 loc) · 9.86 KB

pdf_fetching_capability.md

File metadata and controls

383 lines (283 loc) · 9.86 KB

PDF Fetching Core Capability

Overview

The CommunityMech literature system integrates a 6-tier cascading PDF discovery strategy adapted from the MicroGrowAgents project. This provides robust PDF access across legal sources and fallback mirrors for comprehensive evidence extraction.

Integration Status

✅ VALIDATED - Successfully integrated and tested

  • Scihub fallback from MicroGrowAgents: pdf_evidence_extractor.py
  • 6-tier cascading strategy implemented
  • HTML parsing for embedded PDFs working
  • 100% success rate on initial test (5/5 DOIs)

Architecture

6-Tier PDF Cascade

The system tries sources in order, stopping at first success:

1. Publisher Direct  → Publisher-specific PDF URL patterns
2. PubMed Central    → Open access via NCBI PMC
3. Unpaywall API     → Legal open access aggregator
4. Semantic Scholar  → Academic search engine PDFs
5. Scihub Mirrors    → Fallback mirrors (configurable)
6. Web Search        → Last resort general search

Success Rates (Initial Test - 5 DOIs)

Tier Source Success Rate
1 Publisher 2 40%
2 PMC 0 0%
3 Unpaywall 0 0%
4 Semantic Scholar 0 0%
5 Scihub Fallback 3 60%
6 Web Search 0 0%
Total All Tiers 5 100%

Implementation

Core Class

from communitymech.literature_enhanced import EnhancedLiteratureFetcher

# With fallback enabled (default)
fetcher = EnhancedLiteratureFetcher(
    cache_dir=".literature_cache",
    use_fallback_pdf=True
)

# Fetch paper with PDF
paper = fetcher.fetch_paper("doi:10.3389/fmicb.2015.00475", download_pdf=True)

# Returns:
{
    'title': '...',
    'abstract': '...',
    'authors': [...],
    'year': 2015,
    'journal': 'Frontiers in Microbiology',
    'pdf_url': 'https://...',      # PDF URL found
    'pdf_source': 'publisher',      # Which tier succeeded
    'pdf_path': '.literature_cache/...pdf',  # Local path if downloaded
    'pdf_text': '...'               # Extracted text if downloaded
}

Configuration

Scihub mirrors are configured via environment variable:

export FALLBACK_PDF_MIRRORS="https://sci-hub.ru,https://sci-hub.st,https://sci-hub.ren"

Default mirrors (if not set):

Recommendation: Update default to prioritize sci-hub.ru

Publisher-Specific Patterns

Implemented direct PDF patterns for major publishers:

Publisher DOI Prefix Status
ASM (American Society for Microbiology) 10.1128 Implemented
PLOS 10.1371 ✅ Tested
Frontiers 10.3389 ✅ Working
MDPI 10.3390 Implemented
Nature 10.1038 Implemented
Science 10.1126 Implemented
Elsevier 10.1016 Implemented

Frontiers: Successfully extracted 2/2 PDFs in test

HTML Parsing

Scihub mirrors return HTML pages with embedded PDF URLs. The system extracts PDFs using multiple patterns:

# Pattern 1: <object> tag
<object id="pdf" data="https://twin.sci-hub.se/12345/article.pdf" type="application/pdf">

# Pattern 2: <a> download link
<a href="//sci-hub.se/downloads/2023-01-15/abc/article.pdf">

# Pattern 3: <embed> tag
<embed src="/downloads/article.pdf" type="application/pdf">

# Pattern 4: <iframe> embedding
<iframe src="https://sci-hub.st/pdf/article.pdf">

All 4 patterns validated in testing.

Validation

Test Scripts

Configuration test:

poetry run python scripts/test_pdf_fetching.py --config-only

Quick test (5 DOIs):

poetry run python scripts/test_pdf_fetching.py --quick

Full test (20 DOIs):

poetry run python scripts/test_pdf_fetching.py --full

Without fallback:

poetry run python scripts/test_pdf_fetching.py --quick --no-fallback

Test Results (Quick - 5 DOIs)

Successfully fetched: 5/5 (100%)

Examples:

  1. 10.3389/fmicb.2015.00475 → Publisher (Frontiers)
  2. 10.3389/fmicb.2024.1374800 → Publisher (Frontiers)
  3. 10.1099/00207713-36-2-197 → Scihub (sci-hub.ru)
  4. 10.1099/ijs.0.65409-0 → Scihub (sci-hub.ru)
  5. 10.1007/s11356-014-3789-4 → Scihub (sci-hub.ru)

Observations:

  • Frontiers PDFs accessible directly via publisher
  • Older/paywalled articles successfully retrieved via sci-hub.ru
  • Cascade strategy working: legal sources tried first
  • No failures - robust fallback coverage

Integration Points

Literature Review

The enhanced fetcher integrates with existing literature validation:

# Quick review (abstracts only - fast)
poetry run python scripts/quick_literature_review.py

# Full review (with PDF fetching - slow)
poetry run python scripts/review_literature.py

Evidence Extraction

With PDF access, can extract evidence snippets from full text:

poetry run python scripts/extract_evidence_snippets.py

Community YAML Workflow

  1. Add DOI reference to evidence
  2. fetch_paper() retrieves abstract + PDF
  3. Validate evidence snippets against abstract or full text
  4. Extract better snippets if needed

Known Issues

Mirror Availability

sci-hub.se: DNS resolution failure

NameResolutionError: Failed to resolve 'sci-hub.se'

sci-hub.st: Loads but HTML parsing returns no PDF

✗ Fallback page loaded but no PDF found

sci-hub.ru: ✅ WORKING RELIABLY

Recommendation: Update default mirror priority:

fallback_pdf_urls = [
    "https://sci-hub.ru",      # Move to first (working)
    "https://sci-hub.ren",
    "https://sci-hub.st",
    # "https://sci-hub.se",   # Remove (down)
]

Rate Limiting

NCBI E-utilities: 3 requests/second limit

  • Implemented 0.5s delay between requests
  • Hit 429 errors in special_references script
  • Solution: Increase delay to 1s for bulk operations

Scihub: No documented rate limit observed

  • Conservative 1s delay recommended

Tier 2-4 Not Tested

Initial test didn't find PDFs via:

  • PubMed Central (Tier 2)
  • Unpaywall (Tier 3)
  • Semantic Scholar (Tier 4)

Possible reasons:

  • Test DOIs were paywalled/not in open access
  • Implementation issues (needs validation)
  • API keys needed (Unpaywall, Semantic Scholar)

Action: Full test (20 DOIs) may reveal if these tiers work

Performance

Quick Test (5 DOIs)

  • Time: ~1 minute
  • Success: 100%
  • Network requests: ~15-20 (cascading with failures)

Expected Full Test (20 DOIs)

  • Estimated time: 5-10 minutes
  • Network requests: ~60-100
  • Success rate: Expected 80-100%

With PDF Download

  • Adds 5-30 seconds per PDF (depends on size)
  • PDF extraction adds 10-60 seconds per PDF
  • Not recommended for bulk operations

Recommendations

Immediate Actions

  1. Update mirror priority

    # In literature_enhanced.py
fallback_pdf_urls = ["https://sci-hub.ru", "https://sci-hub.ren"]

  2. Test Tiers 2-4 with known open access DOIs

    • PMC: PMID with PMC ID
    • Unpaywall: Recent open access papers
    • Semantic Scholar: ArXiv papers

  3. Monitor mirror availability

    • Scihub mirrors change frequently
    • Add fallback rotation logic
    • Log which mirrors succeed

Future Enhancements

  1. API Keys

    • Semantic Scholar API key for higher rate limits
    • Unpaywall email for polite access

  2. Caching

    • Cache PDF URLs (not just papers)
    • Reduces redundant cascade attempts

  3. Parallel Fetching

    • Try multiple tiers simultaneously
    • Return first success
    • Faster for bulk operations

  4. PDF Quality Validation

    • Check PDF is not corrupted
    • Verify text extraction succeeds
    • Fallback to next tier if invalid

Usage Examples

Example 1: Fetch Single Paper with PDF

from communitymech.literature_enhanced import EnhancedLiteratureFetcher

fetcher = EnhancedLiteratureFetcher()
paper = fetcher.fetch_paper("doi:10.3389/fmicb.2015.00475", download_pdf=True)

print(f"Title: {paper['title']}")
print(f"PDF source: {paper['pdf_source']}")  # 'publisher'
print(f"PDF path: {paper['pdf_path']}")
print(f"Text length: {len(paper['pdf_text'])} chars")

Example 2: Validate Evidence Snippets

# Evidence from YAML
snippet = "Ferroplasma acidiphilum is an iron-oxidizing archaeon..."
reference = "doi:10.1099/ijs.0.65409-0"

# Fetch abstract
paper = fetcher.fetch_paper(reference, download_pdf=False)

# Validate snippet
valid = fetcher.validate_evidence_snippet(snippet, paper['abstract'])
print(f"Snippet valid: {valid}")

# If invalid and PDF available, try full text
if not valid and paper['pdf_path']:
    valid_in_fulltext = fetcher.validate_evidence_snippet(snippet, paper['pdf_text'])

Example 3: Bulk PDF Discovery

dois = [
    "10.3389/fmicb.2015.00475",
    "10.1099/ijs.0.65409-0",
    "10.1007/s11356-014-3789-4"
]

results = []
for doi in dois:
    pdf_url, source = fetcher.fetch_pdf_url(doi)
    results.append({
        'doi': doi,
        'pdf_url': pdf_url,
        'source': source
    })

# Analyze success by tier
from collections import Counter
tier_counts = Counter(r['source'] for r in results if r['source'])
print(tier_counts)
# Counter({'fallback_mirror': 2, 'publisher': 1})

Summary

Status: ✅ Core capability successfully integrated and validated

Key Achievements:

  • Scihub fallback operational (sci-hub.ru working)
  • HTML parsing extracting PDFs correctly
  • 100% success on initial test
  • Cascading strategy working as designed

Next Steps:

  1. Complete full test (20 DOIs) for comprehensive statistics
  2. Update default mirror configuration
  3. Test Tiers 2-4 with appropriate DOIs
  4. Document best practices for literature workflow

Impact on Knowledge Base:

  • Enables validation of 654 evidence items
  • Can extract better snippets from full text
  • Improves evidence quality and scientific accuracy
  • Supports ongoing literature curation