Fixed unable to download images (#233)

Author: AlexJSully

.github/workflows/code-qa.yaml .github/workflows/code-qa-js.yaml.github/workflows/code-qa.yaml renamed to .github/workflows/code-qa-js.yaml

@@ -1,10 +1,28 @@
-name: Code Quality Assurance
+name: Code Quality Assurance - JavaScript/TypeScript
 
 on:
     push:
         branches: [master]
+        paths:
+            - "**/*.js"
+            - "**/*.json"
+            - "**/*.ts"
+            - ".github/workflows/code-qa-js.yaml"
+            - "eslint.config.js"
+            - "jest.config.js"
+            - "package*.json"
+            - "tsconfig*.json"
     pull_request:
         branches: [master]
+        paths:
+            - "**/*.js"
+            - "**/*.json"
+            - "**/*.ts"
+            - ".github/workflows/code-qa-js.yaml"
+            - "eslint.config.js"
+            - "jest.config.js"
+            - "package*.json"
+            - "tsconfig*.json"
 
 permissions:
     contents: read
@@ -19,13 +37,13 @@ jobs:
 
         steps:
             - name: Checkout repository
-              uses: actions/checkout@v5
+              uses: actions/checkout@v6
             - name: Use Node.js ${{ matrix.node-version }}
-              uses: actions/setup-node@v5
+              uses: actions/setup-node@v6
               with:
                   node-version: ${{ matrix.node-version }}
             - name: Cache Node.js modules
-              uses: actions/cache@v4
+              uses: actions/cache@v5
               with:
                   path: ~/.npm
                   key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}

.github/workflows/markdown-lint.yaml .github/workflows/code-qa-markdown.yaml.github/workflows/markdown-lint.yaml renamed to .github/workflows/code-qa-markdown.yaml

@@ -1,20 +1,22 @@
-name: Markdown Lint
+name: Code Quality Assurance - Markdown
 
 on:
     push:
         branches: [master]
         paths:
             - "**/*.md"
+            - ".github/workflows/markdown-lint.yaml"
             - ".markdownlint.json"
             - ".markdownlintignore"
-            - ".github/workflows/markdown-lint.yaml"
+            - "package*.json"
     pull_request:
         branches: [master]
         paths:
             - "**/*.md"
+            - ".github/workflows/markdown-lint.yaml"
             - ".markdownlint.json"
             - ".markdownlintignore"
-            - ".github/workflows/markdown-lint.yaml"
+            - "package*.json"
 
 permissions:
     contents: read
@@ -24,13 +26,13 @@ jobs:
 
         steps:
             - name: Checkout repository
-              uses: actions/checkout@v5
+              uses: actions/checkout@v6
             - name: Use Node.js ${{ matrix.node-version }}
-              uses: actions/setup-node@v5
+              uses: actions/setup-node@v6
               with:
                   node-version: ${{ matrix.node-version }}
             - name: Cache Node.js modules
-              uses: actions/cache@v4
+              uses: actions/cache@v5
               with:
                   path: ~/.npm
                   key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}

README.md

@@ -121,17 +121,17 @@ For comprehensive documentation, see the [`docs/`](docs/) folder:
 
 ## Maintenance Mode
 
-This project is currently in maintenance mode. This means that:
+This project is currently in **maintenance mode**. This means that:
 
-- Only critical bug fixes and security updates will be addressed.
-- New feature requests are unlikely to be implemented.
+- ✅ **Critical bug fixes** will be addressed
+- ✅ **Security updates** will be implemented promptly
+- ✅ **Minor improvements** to existing functionality may be accepted
+- ❌ **New features** are unlikely to be implemented
+- ❌ **Major refactoring** or architectural changes will not be pursued
 
-## Sponsorship
+**Response Time:** While we strive to address issues promptly, response times may vary. Critical security issues will be prioritized.
 
-If you want to support my work, you can do so through the following methods:
-
-- [BTC](3Lp4pwF5nXqwFA62BYx4DSvDswyYpskBog) - 3Lp4pwF5nXqwFA62BYx4DSvDswyYpskBog
-- [ETH](0xc6EB17BD7cbe5976Bfc4f845669cD66Ff340a1A2) - 0xc6EB17BD7cbe5976Bfc4f845669cD66Ff340a1A2
+**Contributing:** Pull requests for bug fixes and security patches are still welcome. Please review the [Contributing Guidelines](CONTRIBUTING.md) before submitting.
 
 ## Authors
 

docs/architecture/index.md

@@ -143,56 +143,45 @@ graph TD
 
 ### 4. Parse Module (`src/processor/parseFigures.ts`)
 
-Processes XML article data to extract figure information:
+Processes XML article data to extract PMC IDs and orchestrate package downloads:
 
 ```mermaid
 graph LR
-    A[XML Article Data] --> B[Parse XML Structure]
-    B --> C[Extract Article Metadata]
-    C --> D[Find Figure Elements]
-    D --> E[Extract Figure URLs]
-    E --> F[Process Each Figure]
-    F --> G[Download Figure]
-    G --> H[Save to File System]
+        A[XML Article Data] --> B[Parse XML Structure]
+        B --> C[Extract Article Metadata]
+        C --> D[Locate PMC ID]
+        D --> E[Request Article Package]
+        E --> F[Extract Images from Package]
+        F --> G[Save Images to File System]
 ```
 
-**XML Structure Navigation:**
+**XML Structure Navigation (PMC ID extraction):**
+
+The parser locates the PMC identifier in the article front matter (see implementation: [`src/processor/parseFigures.ts`](../src/processor/parseFigures.ts)).
 
 ```xml
 <pmc-articleset>
-  <article>
-    <front>
-      <article-meta>
-        <article-id pub-id-type="pmc">PMC123456</article-id>
-      </article-meta>
-    </front>
-    <body>
-      <fig>
-        <graphic xlink:href="figure1.jpg"/>
-      </fig>
-    </body>
-  </article>
+    <article>
+        <front>
+            <article-meta>
+                <article-id pub-id-type="pmcid">PMC123456</article-id>
+            </article-meta>
+        </front>
+    </article>
 </pmc-articleset>
 ```
 
-### 5. Download Module (`src/processor/downloadImage.ts`)
+### 5. Download Module (`src/processor/downloadArticlePackage.ts`)
 
-Handles actual file downloads with proper error handling:
+Downloads a complete PMC article package (.tar.gz) and extracts image files. The implementation fetches a package URL from the OA Web Service API, downloads the archive, extracts media, and selects the highest-priority image format per basename before copying results to the output directory (see implementation: [`src/processor/downloadArticlePackage.ts`](../src/processor/downloadArticlePackage.ts)).
 
-```mermaid
-stateDiagram-v2
-    [*] --> Validate_URL
-    Validate_URL --> Create_Directory
-    Create_Directory --> Download_File
-    Download_File --> Success : HTTP 200
-    Download_File --> Retry : Network Error
-    Download_File --> Skip : HTTP 404
-    Retry --> Download_File : Max 3 attempts
-    Retry --> Failed : Exceeded retries
-    Success --> [*]
-    Skip --> [*]
-    Failed --> [*]
-```
+Key implementation behaviors (implementation proof):
+
+- Fetches OA package metadata via the OA API and converts FTP links to HTTPS (see [`src/processor/fetchPackageUrl.ts`](../src/processor/fetchPackageUrl.ts)).
+- Downloads the package archive and extracts it to a temporary directory (see [`src/processor/downloadArticlePackage.ts`](../src/processor/downloadArticlePackage.ts)).
+- Groups files by basename and keeps the highest-priority extension using the `IMAGE_EXTENSIONS` priority map (see [`src/constants.ts`](../src/constants.ts)).
+
+Console-level messages written by the implementation include `Fetching package URL for <PMCID>`, `Package downloaded. Extracting images...`, `Extracted image: <filename>`, and `Successfully extracted <N> images from package.` (see [`src/processor/downloadArticlePackage.ts`](../src/processor/downloadArticlePackage.ts)).
 
 ## Data Flow Architecture
 
@@ -211,13 +200,13 @@ graph TD
 
     subgraph "Content Processing"
         D --> E[XML Parsing]
-        E --> F[Figure URL Extraction]
-        F --> G[URL Validation]
+        E --> F[PMC ID Extraction]
+        F --> G[Request Article Package]
+        G --> H[Extract Images from Package]
     end
 
     subgraph "File Operations"
-        G --> H[Directory Creation]
-        H --> I[Figure Download]
+        H --> I[Directory Creation]
         I --> J[File System Storage]
     end
 

docs/faq.md

@@ -11,12 +11,12 @@ The Publication Figure Retrieval Tool is an open-source Node.js application that
 ### What formats are supported?
 
 - **Input**: Species names (scientific and common names)
-- **Output**: JPEG images from publication figures
+- **Output**: Image formats extracted from article packages; supported extensions are defined in the code (`IMAGE_EXTENSIONS`) and include common formats such as `jpg`, `png`, `tiff`, `gif`, and `svg` (see [`src/constants.ts`](../src/constants.ts)).
 - **Data**: JSON metadata files with article and figure information
 
 ### Is this tool free to use?
 
-Yes, this is an open-source tool released under the MIT License. However, please respect the NCBI API usage guidelines and publication copyright restrictions.
+The project is open-source; consult the repository [`package.json`](../package.json) for the declared license. Users must comply with NCBI API usage guidelines and any applicable publication copyright restrictions.
 
 ## Installation and Setup
 
@@ -99,16 +99,16 @@ For the example above:
 
 ### Q: Where are the downloaded figures saved?
 
-**A:** Figures are saved in the `src/output/` directory, organized by species:
+**A:** At runtime the tool writes extracted images to the `build/output/` directory (when running the compiled JavaScript). The layout is organized by species and PMC ID; the package extraction and write behavior are implemented in [`src/processor/parseFigures.ts`](../src/processor/parseFigures.ts) and [`src/processor/downloadArticlePackage.ts`](../src/processor/downloadArticlePackage.ts). Example:
 
 ```text
-src/output/
+build/output/
+├── cache/
+│   └── id.json
 ├── Homo_sapiens/
-│   ├── figures/
-│   │   ├── PMC123456_figure1.jpg
-│   │   └── PMC123456_figure2.jpg
-│   └── metadata/
-│       └── PMC123456_metadata.json
+│   ├── PMC123456/
+│   │   ├── figure1.jpg
+│   │   └── figure2.png
 ```
 
 ## Troubleshooting

docs/index.md

@@ -38,8 +38,8 @@ graph TD
     D --> E[Get Article PMC IDs]
     E --> F[Fetch Article Details]
     F --> G[Parse XML Response]
-    G --> H[Extract Figure URLs]
-    H --> I[Download Figures]
+    G --> H[Download Article Package]
+    H --> I[Extract Images from Package]
     I --> J[Save to Species/PMCID Directory]
     J --> K{More Species?}
     K -->|Yes| C
@@ -102,10 +102,10 @@ npm run start
 
 The tool will:
 
-1. Read species from `src/data/species.json`
-2. Search PMC for each species
-3. Download figures to `build/output/[species]/[pmcid]/`
-4. Cache progress in `build/output/cache/id.json`
+1. Read species from [`src/data/species.json`](../src/data/species.json)
+2. Search PMC for each species (see [`src/processor/searchArticleBySpecies.ts`](../src/processor/searchArticleBySpecies.ts))
+3. For each article: fetch article XML, identify the PMC ID, download the article package (.tar.gz) and extract images into `build/output/[species]/[pmcid]/` (see [`src/processor/parseFigures.ts`](../src/processor/parseFigures.ts) and [`src/processor/downloadArticlePackage.ts`](../src/processor/downloadArticlePackage.ts))
+4. Cache progress in `build/output/cache/id.json` to enable resume
 
 ### With API Key (Recommended)
 
@@ -141,10 +141,9 @@ sequenceDiagram
     Fetch->>PMC: efetch.fcgi?db=pmc&id=batch
     PMC-->>Fetch: XML article data
 
-    Fetch->>Parse: parseFigures(xmlData)
-    Parse->>Parse: extractFigureUrls()
-    Parse->>Download: downloadImage(figureUrl)
-    Download-->>Parse: Downloaded figure
+    Fetch->Parse: parseFigures(xmlData)
+    Parse->>Download: downloadArticlePackage(pmcId)
+    Download-->>Parse: Extracted images saved to disk
     Parse-->>User: Organized files
 ```
 

docs/usage/api/searchArticleBySpecies.md

@@ -27,7 +27,7 @@ graph TD
 ### Signature
 
 ```typescript
-export async function searchArticlesBySpecies(throttle: any, species: string): Promise<string[]>;
+export async function searchArticlesBySpecies(throttle: ThrottleFunction, species: string): Promise<string[]>;
 ```
 
 ### Parameters
@@ -40,8 +40,8 @@ export async function searchArticlesBySpecies(throttle: any, species: string): P
 ### Return Value
 
 - **Type**: `Promise<string[]>`
-- **Description**: Array of PMC IDs (without "PMC" prefix)
-- **Example**: `["123456", "789012", "345678"]`
+- **Description**: The function returns the ID list provided by the NCBI response at `response.data.esearchresult.idlist`. The implementation returns the value directly from the API response (see [`src/processor/searchArticleBySpecies.ts`](../src/processor/searchArticleBySpecies.ts)).
+- **Example**: `["PMC123456", "PMC789012"]` (exact contents depend on the API response)
 
 ### API Integration
 
@@ -155,11 +155,11 @@ const pmcIds = await searchArticlesBySpecies(throttleWithKey, "Cannabis_sativa")
 ```typescript
 // From the actual implementation
 try {
-    const response = await throttle(async () => await axios.get(url));
-    return response.data.esearchresult.idlist; // Returns array of PMC IDs
+	const response = await throttle(async () => await axios.get(url));
+	return response.data.esearchresult.idlist; // Returns array of PMC IDs
 } catch (error) {
-    console.error("Error fetching articles:", error);
-    return []; // Returns empty array on error
+	console.error("Error fetching articles:", error);
+	return []; // Returns empty array on error
 }
 ```
 

docs/usage/index.md

@@ -54,10 +54,10 @@ npm run start
 
 The tool will:
 
-1. **Load species configuration** from `src/data/species.json`
+1. **Load species configuration** from [`src/data/species.json`](../src/data/species.json)
 2. **Initialize rate limiting** (3 requests/second without API key)
 3. **Process each species** sequentially
-4. **Download figures** to `build/output/[species]/[pmcid]/`
+4. **Download article packages and extract images** into `build/output/[species]/[pmcid]/` (see [`src/processor/parseFigures.ts`](../src/processor/parseFigures.ts) and [`src/processor/downloadArticlePackage.ts`](../src/processor/downloadArticlePackage.ts))
 5. **Cache progress** for resume capability
 
 ### Example Output
@@ -67,10 +67,13 @@ Searching articles for the species: Arabidopsis_thaliana...
 Found 1,247 articles for Arabidopsis_thaliana
 Fetching Arabidopsis thaliana article details for batch 1-50...
 Processing article PMC ID: PMC123456
-Found 3 figures in the article.
-Downloaded image: figure1.jpg
-Downloaded image: figure2.png
-Downloaded image: supplementary1.tiff
+Fetching package URL for PMC123456...
+Downloading package from https://.../PMC123456.tar.gz
+Package downloaded. Extracting images...
+Extracted image: figure1.jpg (priority: jpg)
+Extracted image: figure2.png (priority: png)
+Successfully extracted 2 images from package.
+Successfully processed article package for PMC123456
 ```
 
 ## Configuration Options