updated README

Author: AlexJSully

README.md

@@ -1,16 +1,14 @@
 # Publication Figure Retrieval Tool
 
-This tool provides a method for scraping through NCBI's [PMC](https://www.ncbi.nlm.nih.gov/labs/pmc/) publications and retrieving (downloading) the figures from open access and publicly available publications.
+This tool provides a method for retrieving figures from NCBI's [PMC](https://www.ncbi.nlm.nih.gov/labs/pmc/) publications using the Entrez API.
 
 [![Follow on Twitter](https://img.shields.io/twitter/follow/alexjsully?style=social)](https://twitter.com/alexjsully)
-[![GitHub repo size](https://img.shields.io/github/repo-size/AlexJSully/Publication-Figures-Web-Scraping)](https://github.com/AlexJSully/Publication-Figures-Web-Scraping)
-[![GitHub](https://img.shields.io/github/license/AlexJSully/Publication-Figures-Web-Scraping)](https://github.com/AlexJSully/Publication-Figures-Web-Scraping)
+[![GitHub repo size](https://img.shields.io/github/repo-size/AlexJSully/Publication-Figure-Retrieval)](https://github.com/AlexJSully/Publication-Figure-Retrieval)
+[![GitHub](https://img.shields.io/github/license/AlexJSully/Publication-Figure-Retrieval)](https://github.com/AlexJSully/Publication-Figure-Retrieval)
 
 ## Disclaimer
 
-Please note that as of August 20, 2024, scraping figures from PMC (PubMed Central) publications is no longer permitted under their Copyright Notice ([https://www.ncbi.nlm.nih.gov/pmc/about/copyright/](https://www.ncbi.nlm.nih.gov/pmc/about/copyright/)). Attempting to do so may result in a 403 error and an IP ban from NIH/PMC. Users are fully responsible for ensuring their use of this tool complies with current PMC policies and applicable laws. Use at your own risk.
-
-This code is maintained for educational and historical reference purposes only. While it remains functional, it is not intended for use in scraping PMC data, as this is no longer allowed under current policies. The tool was originally developed for academic research, but due to these policy changes, it is not recommended for use in this capacity anymore.
+This code is maintained for educational and historical reference purposes only. The tool was originally developed for academic research. Please note that the use of this tool for retrieving figures from PMC publications is subject to NCBI's policies. Use at own risk.
 
 ## Requirements
 
@@ -20,25 +18,41 @@ This code is maintained for educational and historical reference purposes only.
 
 ## Installation & Setup
 
-If you would like to run or modify the publication figure web scraping tool locally, clone the repository with git by running the following command:
+If you would like to run or modify the publication figure retrieval tool locally, clone the repository with git by running the following command:
+
+```bash
+git clone https://github.com/AlexJSully/Publication-Figure-Retrieval.git
+```
+
+Then run
+
+```bash
+npm install
+```
+
+followed by
 
-```git
-git clone https://github.com/AlexJSully/Publication-Figures-Web-Scraping.git
+```bash
+npm start
 ```
 
-Then run `npm install` then `npm start`. This tool runs within your node environment. On Windows, this script needs to run in an administrator mode.
+This tool runs within your Node.js environment. On Windows, this script may need to run in administrator mode.
+
+### Usage
 
-The images are downloaded then downloaded locally within this containing directory under [src/data/figures/{species}/{PMC ID}](./src/data/figures).
+The images are downloaded locally within the `build/processor/output` directory.
 
-If you would like to run against commercial use publications, you will need to download [`oa_comm_use_file.list.txt`](https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_comm_use_file.list.txt) from [ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/](https://ftp.ncbi.nlm.nih.gov/pub/pmc/) then run `npm run process`. Once that is done, set [index.js](./src/index.js) init function to true (`await init(true);`)
+### API Key
 
-The publication figure scraper will resume where you last left off. If you would like to reset the scraper, empty [species-pmid-list.json](./src/data/species-pmid-list.json), [data-retrieved.json](./src/data/data-retrieved.json) and [data-empty-pubs.json](./src/data/data-empty-pubs.json) to contain only just an empty JSON object (`{}`).
+If you have an API key, create a `.env` file in the root directory and add your API key as follows:
 
-If you would like to add more species support for publications to be scraped, add the species to [species.json](./src/data/species.json) and then run `npm start`. Currently, this JSON includes species' common aliases which are not currently being used but may be useful in the future. If you would like to scrape a single species, then change `speciesList` in [index.js](./src/index.js) to an array of species scientific name(s) to scrape. For example: `speciesList = ['Arabidopsis thaliana']; // Or whatever species name(s) you would like to scrape`. Currently, it is set to scrape all species within the [species.json](./src/data/species.json) file.
+```bash
+NCBI_API_KEY=your_api_key_here
+```
 
-If in the instance that you do not have an internet connection/speed greater than 7mb/s, you will need to change all the Axios request timeouts in [data-retrieval.js](./src/scripts/data-retrieval.js) to a value of at least half of your speed (e.g. down speed of 10mb/s, set timeout to 5s).
+With an API key, the tool can retrieve up to 10 calls per second instead of 3. Details on obtaining an API key can be found [here](https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities/).
 
-## Known issues
+## Known Issues
 
 We aim to make this tool as perfect as possible but unfortunately, there may be some unforeseen bugs. If you manage to find one that is not here, feel free to create a bug report so we can fix it.
 
@@ -48,6 +62,12 @@ We aim to make this tool as perfect as possible but unfortunately, there may be
 
 Please read [CONTRIBUTING.md](CONTRIBUTING.md) for more details.
 
+Before contributing, ensure that all tests pass by running:
+
+```bash
+npm run validate
+```
+
 ## License
 
 [GLP-2.0](LICENSE.md)
@@ -61,7 +81,7 @@ This project is currently in maintenance mode. This means that:
 
 ## Sponsorship
 
-If you want to support my work, you can through the following methods:
+If you want to support my work, you can do so through the following methods:
 
 -   [BTC](3Lp4pwF5nXqwFA62BYx4DSvDswyYpskBog) - 3Lp4pwF5nXqwFA62BYx4DSvDswyYpskBog
 -   [ETH](0xc6EB17BD7cbe5976Bfc4f845669cD66Ff340a1A2) - 0xc6EB17BD7cbe5976Bfc4f845669cD66Ff340a1A2

package.json

@@ -47,11 +47,11 @@
 	],
 	"private": true,
 	"bugs": {
-		"url": "https://github.com/AlexJSully/Publication-Figures-Web-Scraping/issues"
+		"url": "https://github.com/AlexJSully/Publication-Figure-Retrieval/issues"
 	},
 	"repository": {
 		"type": "git",
-		"url": "git+https://github.com/AlexJSully/Publication-Figures-Web-Scraping.git"
+		"url": "git+https://github.com/AlexJSully/Publication-Figure-Retrieval.git"
 	},
 	"keywords": [
 		"science",
@@ -60,5 +60,5 @@
 		"publication-data"
 	],
 	"license": "GNU General Public License v2.0",
-	"homepage": "https://github.com/AlexJSully/Publication-Figures-Web-Scraping#readme"
+	"homepage": "https://github.com/AlexJSully/Publication-Figure-Retrieval#readme"
 }

src/index.ts

@@ -5,7 +5,7 @@ import { searchArticlesBySpecies } from "./processor/searchArticleBySpecies";
 
 // Check if NCBI API key is present in environment variables
 /** The API key for the NCBI E-utilities. */
-const ncbiApiKey = process.env.NCBI_API_KEY;
+const ncbiApiKey = process?.env?.NCBI_API_KEY;
 /**
  * The number of API calls allowed per second.
  * If an API key is provided, we can make up to 10 calls per second.

src/processor/extractFigureUrls.ts

@@ -57,6 +57,7 @@ export function extractFigureUrls(
 						// Construct the correct absolute URL
 						/** The absolute URL of the figure graphic. */
 						const absoluteUrl = `https://www.ncbi.nlm.nih.gov/pmc/articles/PMC${pmcId}/bin/${figureUrl}`;
+
 						figureUrls.push(absoluteUrl);
 					}
 				});

src/processor/fetchArticleDetails.ts

@@ -34,7 +34,12 @@ export async function fetchArticleDetails(
 		/** Comma-separated list of PMCIDs. */
 		const ids = batch.join(",");
 		/** The URL to fetch article details for the current batch. */
-		const url = `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=${ids}&retmode=xml`;
+		let url = `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=${ids}&retmode=xml`;
+		// Check if there is a NCBI API key available and if so, add it to the URL
+		if (process?.env?.NCBI_API_KEY) {
+			url += `&api_key=${process.env.NCBI_API_KEY}`;
+		}
+
 		console.log(`Fetching article details for batch ${i + 1}-${i + batch.length}...`);
 
 		try {

src/processor/searchArticleBySpecies.ts

@@ -22,9 +22,13 @@ export async function searchArticlesBySpecies(
 ): Promise<string[]> {
 	// Construct query for species and open-access filter
 	const query = `${species}[organism]`;
-	const url = `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pmc&term=${encodeURIComponent(
+	let url = `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pmc&term=${encodeURIComponent(
 		query,
-	)}&retmode=json&retmax=100000`;
+	)}&retmode=json&retmax=1000000`;
+	// Check if there is a NCBI API key available and if so, add it to the URL
+	if (process?.env?.NCBI_API_KEY) {
+		url += `&api_key=${process.env.NCBI_API_KEY}`;
+	}
 
 	try {
 		// Make HTTP request to NCBI E-utilities API to search for articles