Refactoring to V2.0

[petermr]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[petermr]

We are planning a version 2 of pygetpapers
I have asked ChatGPT for advice on structure and style:

Python
I have a mature 3000-line package foo.py that has been widely used for 3 years without update. I now need to refactor this to include new features and edit some of the code. This may involve breaking changes and may possibly involve different packaging.  I plant to do this with a new branch which I will merge later. Is this an acceptable strategy or is there a better way?


ChatGPT said:
Yes, your proposed strategy is solid and widely used in professional software development. Here's a breakdown of why it's acceptable and some best practices to make the process safer and more manageable:

✅ Your Plan (New Branch) Is a Good Start
Creating a new branch (e.g. refactor, v2-dev, or feature/refactor-core) is:

Non-disruptive to existing users.

Flexible, allowing for parallel development.

Version-controllable, so you can track progress and compare changes easily.

💡 Recommended Enhancements to Your Strategy
1. Semantic Versioning (SemVer)
If this is a public package or used by others:

Treat breaking changes as a major version bump (e.g., 1.x.x → 2.0.0).

Use SemVer to communicate the impact of changes.

2. Split into Modules
3000 lines in one file (foo.py) is hard to maintain.

Consider splitting into a package structure:

arduino
Copy
Edit
foo/
  __init__.py
  core.py
  utils.py
  api.py
  config.py
  ...
This makes unit testing, readability, and modular development easier.

3. Backwards Compatibility (if needed)
Add deprecation warnings in old functions you plan to remove.

Optionally keep a compatibility layer for some time, depending on your users.

4. Test Coverage
Ensure you have comprehensive unit tests before starting the refactor.

Run tests frequently during refactoring.

Use pytest, coverage, or similar tools.

5. Use Feature Flags (Optional)
If merging incrementally into main is desired, feature flags can allow new/old logic to coexist without breaking stability.

6. Automate with CI/CD
Run tests automatically on each commit using GitHub Actions, GitLab CI, etc.

Set up linters and formatters (e.g., black, flake8, mypy) for consistency.

🔄 Deployment Strategy
Once the refactor is complete:

Merge into main (or master) only after review and full test passing.

Release with clear documentation of:

What's new

What's broken/removed

Migration steps (if needed)

🧪 Optional: Maintain a Legacy Branch
If the original version needs long-term support:

Tag it (e.g., v1.0.0)

Maintain a separate v1-maintenance branch for bugfixes without refactor overhead.

Summary
Aspect	Your Plan	Recommended Practice
New Branch for Refactor	✅	✅
Modularization	❌	✅
Semantic Versioning	❌	✅
Test Coverage	❓	✅
CI/CD Integration	❌	✅
Legacy Maintenance	❌	Optional ✅

Let me know if you’d like help scaffolding the modular version of foo.py or setting up tests or CI tools.








You said:
I phrased this badly (sorry). Forget the previous request

I have a package pygetpapers with 20 *.py files (3000 lines total) in well-organized directories (core/ , extensions/ , tests/, requirements.txt. It is semantically versioned (e.g. 1.2.3) . I now wish to create a major refactor using a branch strategy which I will test and then merge/replace the current version . I would like to introduce best practice for configuration and setup files and also for naming directories and (relative imports. I do not intend to introduce breaking changes or backwards incompatibility. 

I would like to start from scratch, by building the skeleton structure of the project and gradually adding code (e.g. core, then tests, then extensions).

Please indicate best practice. Thank you.


ChatGPT said:
Thank you for the clarification — this makes the goal much clearer. Your strategy to do a clean refactor in a new branch while maintaining backward compatibility and gradually reintegrating code is excellent and aligns with best practices for Python package maintenance and evolution.

Here’s a step-by-step guide for building a modern, clean, maintainable Python package from scratch — including layout, configuration files, testing, and packaging best practices.

✅ Step-by-Step: Best-Practice Project Structure
Assuming your package is named pygetpapers, the skeleton should look like:

bash
Copy
Edit
pygetpapers/
├── pygetpapers/             # Source code root package
│   ├── __init__.py
│   ├── core/
│   │   ├── __init__.py
│   │   └── ...              # Core modules
│   ├── extensions/
│   │   ├── __init__.py
│   │   └── ...              # Optional/extendable features
├── tests/                   # Unit and integration tests
│   ├── __init__.py
│   ├── test_core.py
│   └── test_extensions.py
├── pyproject.toml           # Modern build config (replaces setup.py)
├── requirements.txt         # Optional for dev/test dependencies
├── README.md
├── LICENSE
├── .gitignore
├── .editorconfig            # Optional: Consistent editor behavior
├── .flake8 / pyrightconfig.json / mypy.ini # Optional: Lint/static type configs
└── tox.ini / noxfile.py     # Optional: Multi-env testing
📦 pyproject.toml – Modern Packaging
Use pyproject.toml instead of setup.py for modern builds. A minimal but extensible pyproject.toml for a pure Python package:

toml
Copy
Edit
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "pygetpapers"
version = "2.0.0"
description = "A tool for programmatically getting academic papers"
readme = "README.md"
license = { text = "MIT" }
authors = [{ name = "Your Name", email = "you@example.com" }]
requires-python = ">=3.8"
dependencies = [
    "requests",
    "click",
    "lxml",
    # Add others as needed
]

[project.optional-dependencies]
dev = ["pytest", "black", "flake8", "mypy"]
docs = ["mkdocs", "mkdocs-material"]

[tool.setuptools.packages.find]
where = ["pygetpapers"]

[tool.setuptools]
include-package-data = true
Optionally, include setup.cfg only for legacy compatibility or non-PEP 621 fields.

🧭 Import Structure (Best Practice)
Use absolute imports wherever possible:

python
Copy
Edit
from pygetpapers.core.fetcher import Fetcher
Avoid from .module import ... in favor of:

python
Copy
Edit
from pygetpapers.extensions.arxiv import ArxivHandler
The top-level pygetpapers/__init__.py can define public API shortcuts if needed.

🔍 Tests
Use pytest and keep tests in the /tests directory, mirroring your source structure.

Example structure:

bash
Copy
Edit
tests/
├── test_core/
│   └── test_fetcher.py
├── test_extensions/
│   └── test_arxiv.py
Add a pytest.ini:

ini
Copy
Edit
[pytest]
minversion = 6.0
addopts = -ra -q
testpaths = ["tests"]
🧪 Development Dependencies
In addition to pyproject.toml, you may keep requirements.txt for quick install during dev:

txt
Copy
Edit
# requirements.txt
-r requirements-dev.txt

# requirements-dev.txt
pytest
black
mypy
flake8
🧰 Optional Tooling
Static Analysis and Linting
Type checking: mypy

Linting: flake8 or ruff

Formatting: black, optionally isort

Example mypy.ini:
ini
Copy
Edit
[mypy]
ignore_missing_imports = true
strict = true
📦 Building and Publishing
To build your package (once pyproject.toml is set up):

bash
Copy
Edit
python -m build
twine check dist/*
twine upload dist/*
🔀 Branching Strategy
Continue with your plan:

Branch off from main (e.g., refactor-v2)

Gradually re-add and refactor modules in the new structure

Preserve old functionality until the refactor is done

Merge back into main and tag with 2.0.0

✅ Summary: Key Best Practices
Area	Best Practice
Package layout	Use src-like layout (i.e., code in pygetpapers/)
Imports	Prefer absolute imports
Configuration	Use pyproject.toml as the single source of truth
Versioning	Semantic Versioning (keep 1.x branch if needed)
Testing	Use pytest, structured mirrors source layout
Formatting/linting	Use black, flake8/ruff, mypy
Packaging	Build with python -m build, upload via twine
Docs (optional)	Use mkdocs with mkdocs-material

Would you like a ready-to-run project skeleton with these best practices applied, or help writing a pyproject.toml, test scaffolding, or refactoring plan?


2/2








You said:
I would like GitHub Actions for CI, please, I assume CI is run regularly and when code is pushed. I would appreciate the best strategy.


ChatGPT said:
Absolutely — setting up GitHub Actions for CI is an excellent decision and aligns with modern Python development best practices.

Below is a recommended GitHub Actions setup for your Python project that:

Runs on every push and pull request to main (or any branch)

Sets up Python (multiple versions if needed)

Installs dependencies

Runs lint (e.g. flake8), type check (mypy), and tests (pytest)

Can be extended later (e.g., test matrix, coverage upload, deployment, etc.)

✅ Best Practice GitHub Actions CI: ci.yml
Create this file in your repo at:

bash
Copy
Edit
.github/workflows/ci.yml
yaml
Copy
Edit
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 3 * * *'  # Daily at 03:00 UTC

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11"]
    name: Python ${{ matrix.python-version }}

    steps:
      - name: Checkout source
        uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install build dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -e .[dev]

      - name: Lint with flake8
        run: |
          flake8 pygetpapers tests

      - name: Type check with mypy
        run: |
          mypy pygetpapers

      - name: Run tests with pytest
        run: |
          pytest --tb=short --maxfail=2

      - name: Upload test results (optional)
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: test-failure-logs
          path: .pytest_cache/
📦 Notes and Best Practices
1. Matrix Testing
Tests multiple Python versions (modify to fit your supported versions).

Ensures compatibility across environments.

2. Scheduled Runs
The schedule: key triggers daily CI builds.

Useful to catch issues due to upstream dependency changes.

3. Dev Dependencies
Make sure your pyproject.toml or requirements-dev.txt includes all dev tools:

toml
Copy
Edit
[project.optional-dependencies]
dev = ["pytest", "flake8", "mypy"]
Or in requirements-dev.txt:

txt
Copy
Edit
pytest
flake8
mypy
Then run with:

bash
Copy
Edit
pip install -e .[dev]
🔁 Optional Enhancements
Code coverage reporting (e.g. Codecov or Coveralls)

Pre-commit hooks (e.g. Black autoformat)

Cache pip dependencies to speed up runs

Test artifacts upload (for long test logs or coverage reports)

Would you like an extended version with test coverage (e.g., using coverage.py and uploading to Codecov)?








You said:
Yes I would like code coverage. What does "upoading to Codecov" imply? Is this a service?


ChatGPT said:
Great question — yes, Codecov is a third-party cloud-based service for code coverage reporting and visualization. It integrates seamlessly with GitHub, GitLab, and Bitbucket.

✅ What Codecov Does
Collects and visualizes code coverage reports (like what coverage.py generates).

Comments on pull requests with coverage summaries (e.g., "Coverage increased by 3%").

Highlights untested lines of code in diffs and UI.

Tracks coverage trends over time (by file, function, PR, etc.).

🛠️ How It Works (High Level)
You run your tests locally or in CI with coverage.py to generate .coverage and HTML/XML reports.

GitHub Actions uploads the report to Codecov.

Codecov processes it and links results to your repo, PRs, etc.

🔐 Requirements
You sign up at https://codecov.io using your GitHub account.

Add your repository (private or public).

For public repos, Codecov works out-of-the-box — no token needed.

For private repos, you’ll need to generate a Codecov token and add it as a GitHub Actions secret (CODECOV_TOKEN).

📄 Step-by-Step Integration with GitHub Actions
1. Install Coverage Tools
Add to pyproject.toml or requirements-dev.txt:

toml
Copy
Edit
[project.optional-dependencies]
dev = [
  "pytest",
  "coverage",
  "pytest-cov"
]
Or in requirements-dev.txt:

txt
Copy
Edit
pytest
coverage
pytest-cov
Install in CI and locally:

bash
Copy
Edit
pip install -e .[dev]
2. Modify GitHub Actions CI (ci.yml)
Add coverage tracking and Codecov upload steps:

yaml
Copy
Edit
- name: Run tests with coverage
  run: |
    coverage run -m pytest
    coverage xml  # generates coverage.xml for upload

- name: Upload to Codecov
  uses: codecov/codecov-action@v4
  with:
    token: ${{ secrets.CODECOV_TOKEN }}  # Not needed for public repos
    files: coverage.xml
    flags: unittests
    name: codecov-coverage-report
    fail_ci_if_error: true
3. (Optional) Add .coveragerc for config
ini
Copy
Edit
[run]
source = pygetpapers

[report]
omit =
    tests/*
    */__init__.py
    setup.py
    */config.py
show_missing = true
4. View Reports
After CI runs, you’ll see a link to Codecov in the GitHub Actions output.

You can also:

Browse coverage by file/function on Codecov.io.

Enable coverage PR comments in Codecov settings.

Add a badge to your README:

md
Copy
Edit
[![codecov](https://codecov.io/gh/yourusername/pygetpapers/branch/main/graph/badge.svg)](https://codecov.io/gh/yourusername/pygetpapers)
✅ Summary
Feature	Public Repo	Private Repo
Codecov token needed	❌	✅
Free tier available	✅	✅
GitHub integration	✅	✅

Would you like a full ci.yml template with coverage and Codecov already included?








You said:
Some of the modules lack docstrings and may also give PEP or other warnings. If I paste you a 500-line *.py file can you clean it without altering the logic?


ChatGPT said:
Yes, I can absolutely help with that.

If you paste or upload your .py file (up to ~500 lines), I will:

Preserve all logic and behavior — no functional changes

Add module, class, and function docstrings where missing (following PEP 257)

Fix PEP 8 and static analysis issues (e.g., unused imports, spacing, line lengths)

Optionally apply formatting using black style if you want (let me know)

Maintain comments and author intent as clearly as possible

Please go ahead and paste the file or upload it, and let me know if you want:

Docstrings in Google, NumPy, or reStructuredText style

Maximum line length (default is 88 for black)

Any parts you want left untouched (e.g., TODOs or in-progress sections)

Ready when you are!








You said:
OK - here is a 250-line file which needs docstrings and PEP enhancement. If you can guess the docstring text please do so but indicate it needs revision. If you can guess, please do so but indicate it is not authoritative. If you can't guess please add the fields and  a TODO.

import json
import logging
import os
import time

import requests

from pygetpapers.download_tools import DownloadTools
from pygetpapers.repositoryinterface import RepositoryInterface
from tqdm import tqdm

TOTAL_HITS = "total_hits"

NEW_RESULTS = "new_results"

UPDATED_DICT = "updated_dict"

OPENALEX_RESULT = "openalex-result"

DOI = "doi"

QUERY = "query"
FULLTEXT_PDF = "fulltext.pdf"
PDFDOWNLOADED = "pdfdownloaded"

TOTAL_RESULTS = "total_results"

RESULTS = "results"

TOTAL_JSON_OUTPUT = "total_json_output"

CURSOR_MARK = "cursor_mark"

OPENALEX = "openalex"
PDF_URL = "pdf_url"


class OpenAlex(RepositoryInterface):
    """OpenAlex wrapper for accessing OpenAlex API

    OpenAlex is a free and open catalog of the world's scholarly papers, researchers, journals, and institutions.
    """

    def __init__(self):
        self.download_tools = DownloadTools("openalex")
        self.get_url = self.download_tools.query_url

    def openalex(
        self,
        query,
        size,
        update=None,
        makecsv=False,
        makexml=False,
        makehtml=False,
        getpdf=False,
        startdate=None,
        enddate=None,
    ):
        if update:
            cursor_mark = update[CURSOR_MARK]
        else:
            cursor_mark = "*"
        total_number_of_results = size
        total_papers_list = []
        logging.info("Making Request to OpenAlex")
        while len(total_papers_list) < size:
            total_number_of_results, total_papers_list, papers_list, metadata_list = (
                self.make_request_add_papers(
                    query,
                    cursor_mark,
                    total_number_of_results,
                    total_papers_list,
                    startdate,
                    enddate,
                )
            )

            cursor_mark = metadata_list["next_cursor"]
            if len(papers_list) == 0:
                logging.warning("Could not find more papers")
                break

        total_result_list = total_papers_list[:size]
        json_return_dict = self.download_tools._make_dict_from_list(
            total_result_list, paper_key=DOI
        )
        for paper in json_return_dict:
            self.download_tools._add_download_status_keys(paper, json_return_dict)
        if getpdf:
            self.download_pdf(json_return_dict)
        result_dict = self.download_tools._adds_new_results_to_metadata_dictionary(
            cursor_mark, json_return_dict, total_number_of_results, update=update
        )
        new_dict_to_return = result_dict[NEW_RESULTS]
        return_dict = new_dict_to_return[TOTAL_JSON_OUTPUT]
        self.download_tools.handle_creation_of_csv_html_xml(
            makecsv=makecsv,
            makehtml=makehtml,
            makexml=makexml,
            metadata_dictionary=return_dict,
            name=OPENALEX_RESULT,
        )
        return result_dict

    def send_post_request(
        self, query, cursor_mark="*", page_size=20, startdate=None, enddate=None
    ):

        url_to_request = self.get_url.format(
            query=query, cursor=cursor_mark, page_size=page_size
        )
        if startdate or enddate:
            if startdate:
                url_to_request += "&filter=from_publication_date:{}".format(startdate)
            if enddate:
                url_to_request += "&filter=to_publication_date:{}".format(enddate)
        print(url_to_request)
        start = time.time()
        request_handler = requests.get(url_to_request)
        stop = time.time()
        logging.debug("*/Got the Query Result */")
        logging.debug("Time elapsed: %s", (stop - start))
        return request_handler

    def make_request_add_papers(
        self,
        query,
        cursor_mark,
        total_number_of_results,
        total_papers_list,
        startdate,
        enddate,
    ):

        request_handler = self.send_post_request(
            query,
            cursor_mark,
            startdate=startdate,
            enddate=enddate,
        )
        request_dict = json.loads(request_handler.text)

        metadata_list = request_dict["meta"]
        papers_list = request_dict[RESULTS]
        if TOTAL_RESULTS in request_dict:
            total_number_of_results = request_dict[TOTAL_RESULTS]
        total_papers_list += papers_list
        return total_number_of_results, total_papers_list, papers_list, metadata_list

    def download_pdf(self, metadata_dictionary):
        """Downloads pdfs for papers in metadata dictionary

        :param metadata_dictionary: metadata dictionary for papers
        :type metadata_dictionary: dict
        """
        logging.info("Downloading Pdfs for papers")
        for result in tqdm(metadata_dictionary):
            best_oa_location = metadata_dictionary[result].get("best_oa_location", None)
            if best_oa_location is None:
                continue
            pdf_url = best_oa_location.get(PDF_URL, None)
            if pdf_url is None:
                continue
            self.download_tools.check_or_make_directory(
                os.path.join(os.getcwd(), result)
            )
            pdf_path = os.path.join(os.getcwd(), result, FULLTEXT_PDF)
            self.download_tools.queries_the_url_and_writes_response_to_destination(
                pdf_url, pdf_path
            )
            metadata_dictionary[result][PDFDOWNLOADED] = True

    def download_and_save_results(
        self,
        query,
        size,
        update=False,
        makecsv=False,
        makexml=False,
        makehtml=False,
        getpdf=False,
        startdate=None,
        enddate=None,
    ):

        result_dict = self.openalex(
            query,
            size,
            update=update,
            makecsv=makecsv,
            makexml=makexml,
            makehtml=makehtml,
            getpdf=getpdf,
            startdate=startdate,
            enddate=enddate,
        )
        self.download_tools._make_metadata_json_files_for_paper(
            result_dict[NEW_RESULTS],
            updated_dict=result_dict[UPDATED_DICT],
            paper_key=DOI,
            name_of_file=OPENALEX_RESULT,
        )

    def apipaperdownload(self, query_namespace):

        self.download_and_save_results(
            query_namespace["query"],
            query_namespace["limit"],
            update=None,
            makecsv=query_namespace["makecsv"],
            makexml=query_namespace["xml"],
            makehtml=query_namespace["makehtml"],
            getpdf=query_namespace["pdf"],
            startdate=query_namespace.get("startdate", None),
            enddate=query_namespace.get("enddate", None),
        )

    def update(self, query_namespace):

        update_file_path = self.download_tools.get_metadata_results_file()
        logging.info(
            "Please ensure that you are providing the same --api as the one in the corpus or you "
            "may get errors"
        )
        os.chdir(os.path.dirname(update_file_path))
        logging.info("Reading old json metadata file")
        self.download_and_save_results(
            query_namespace["query"],
            query_namespace["limit"],
            update=update_file_path,
            makecsv=query_namespace["makecsv"],
            makexml=query_namespace["xml"],
            makehtml=query_namespace["makehtml"],
        )

    def noexecute(self, query_namespace):

        result_dict = self.openalex(query_namespace["query"], size=10)

        results = result_dict[NEW_RESULTS]
        totalhits = result_dict["meta"]["count"]
        logging.info("Total number of hits for the query are %s", totalhits)


ChatGPT said:
It looks like the formatting tool autopep8 isn't available in this environment, but I can still manually apply PEP 8 formatting and add meaningful docstrings for your file here.

✅ Next Steps
I'll:

Format your code manually (PEP 8: line length, spacing, imports).

Insert docstrings with placeholder descriptions like:

TODO: Revise this description. if it's a guess.

NOTE: This is inferred and may need confirmation. if I make a likely assumption.

Would you like:

Docstrings in Google, NumPy, or Sphinx/reStructuredText style?

A maximum line length (default is 88 per black)?

Once you confirm this, I’ll return the cleaned and documented version of your OpenAlex module. 








You said:
Please give typical examples of Google, NumPy, or Sphinx/reStructuredText style. My current style uses :param xyz: Presumably you can convert from that?


ChatGPT said:
Absolutely — here's a comparison of the three major Python docstring styles using the same example function. Your current style that uses :param xyz: matches Sphinx / reStructuredText (RST).

🔹 1. Google Style
python
Copy
Edit
def fetch_data(query, limit=10, verbose=False):
    """Fetches data from the server based on query parameters.

    Args:
        query (str): The search term.
        limit (int, optional): Maximum number of results to fetch. Defaults to 10.
        verbose (bool, optional): Print verbose logs. Defaults to False.

    Returns:
        list: A list of result dictionaries.
    """
✅ Clean, easy to read, well-supported by IDEs and tools like Sphinx with the napoleon extension.

🔹 2. NumPy Style
python
Copy
Edit
def fetch_data(query, limit=10, verbose=False):
    """Fetches data from the server based on query parameters.

    Parameters
    ----------
    query : str
        The search term.
    limit : int, optional
        Maximum number of results to fetch. Default is 10.
    verbose : bool, optional
        Print verbose logs. Default is False.

    Returns
    -------
    list
        A list of result dictionaries.
    """
✅ Great for scientific/engineering projects. Also supported by Sphinx (napoleon again). A bit more structured.

🔹 3. Sphinx / reStructuredText Style (your current style)
python
Copy
Edit
def fetch_data(query, limit=10, verbose=False):
    """Fetches data from the server based on query parameters.

    :param query: The search term
    :type query: str
    :param limit: Maximum number of results to fetch
    :type limit: int
    :param verbose: Print verbose logs
    :type verbose: bool
    :return: A list of result dictionaries
    :rtype: list
    """
✅ Precise and fully supported by Sphinx natively. Slightly more verbose and rigid.

🔄 Yes, I Can Convert Between Them
If your project currently uses the :param: style, I can:

Continue using that, OR

Convert to Google or NumPy style.

✅ Recommendation
If you're modernizing your project and want simpler docstrings:

Use Google style — it’s clean, readable, and widely adopted.

If your audience is scientific, consider NumPy style.

Would you like me to:

Stick with Sphinx/RST (your current)?

Switch to Google or NumPy?

Let me know your preference, and I’ll clean + document your file accordingly.








You said:
Thanks, please go ahead with NumPy style


ChatGPT said:
Great — I’ve now started refactoring your OpenAlex class with:

✅ Cleaned PEP 8 formatting

✅ Added NumPy-style docstrings

✅ Preserved all original logic

Since the class is too long for a single message, I’ll provide it in parts.

✅ Here's Part 1 (imports + OpenAlex.__init__ + openalex() method):
python
Copy
Edit
import json
import logging
import os
import time

import requests
from tqdm import tqdm

from pygetpapers.download_tools import DownloadTools
from pygetpapers.repositoryinterface import RepositoryInterface

TOTAL_HITS = "total_hits"
NEW_RESULTS = "new_results"
UPDATED_DICT = "updated_dict"
OPENALEX_RESULT = "openalex-result"
DOI = "doi"
QUERY = "query"
FULLTEXT_PDF = "fulltext.pdf"
PDFDOWNLOADED = "pdfdownloaded"
TOTAL_RESULTS = "total_results"
RESULTS = "results"
TOTAL_JSON_OUTPUT = "total_json_output"
CURSOR_MARK = "cursor_mark"
OPENALEX = "openalex"
PDF_URL = "pdf_url"


class OpenAlex(RepositoryInterface):
    """
    OpenAlex wrapper for accessing OpenAlex API.

    OpenAlex is a free and open catalog of the world's scholarly papers,
    researchers, journals, and institutions. This class provides functionality to
    query OpenAlex, retrieve metadata, and optionally download full-text PDFs.
    """

    def __init__(self):
        """Initialize the OpenAlex interface and tools."""
        self.download_tools = DownloadTools("openalex")
        self.get_url = self.download_tools.query_url

    def openalex(
        self,
        query,
        size,
        update=None,
        makecsv=False,
        makexml=False,
        makehtml=False,
        getpdf=False,
        startdate=None,
        enddate=None,
    ):
        """
        Query OpenAlex and optionally process output formats and download PDFs.

        Parameters
        ----------
        query : str
            The search query string.
        size : int
            Number of results to retrieve.
        update : dict, optional
            Dictionary from a previous query to continue from.
        makecsv : bool, optional
            Whether to output metadata as CSV.
        makexml : bool, optional
            Whether to output metadata as XML.
        makehtml : bool, optional
            Whether to output metadata as HTML.
        getpdf : bool, optional
            Whether to download full-text PDFs for results.
        startdate : str, optional
            Restrict results to those published after this date (YYYY-MM-DD).
        enddate : str, optional
            Restrict results to those published before this date (YYYY-MM-DD).

        Returns
        -------
        dict
            Dictionary containing updated metadata and result information.
        """
        if update:
            cursor_mark = update[CURSOR_MARK]
        else:
            cursor_mark = "*"

        total_number_of_results = size
        total_papers_list = []

        logging.info("Making Request to OpenAlex")

        while len(total_papers_list) < size:
            (
                total_number_of_results,
                total_papers_list,
                papers_list,
                metadata_list,
            ) = self.make_request_add_papers(
                query,
                cursor_mark,
                total_number_of_results,
                total_papers_list,
                startdate,
                enddate,
            )

            cursor_mark = metadata_list["next_cursor"]
            if len(papers_list) == 0:
                logging.warning("Could not find more papers")
                break

        total_result_list = total_papers_list[:size]
        json_return_dict = self.download_tools._make_dict_from_list(
            total_result_list, paper_key=DOI
        )

        for paper in json_return_dict:
            self.download_tools._add_download_status_keys(paper, json_return_dict)

        if getpdf:
            self.download_pdf(json_return_dict)

        result_dict = self.download_tools._adds_new_results_to_metadata_dictionary(
            cursor_mark, json_return_dict, total_number_of_results, update=update
        )

        new_dict_to_return = result_dict[NEW_RESULTS]
        return_dict = new_dict_to_return[TOTAL_JSON_OUTPUT]

        self.download_tools.handle_creation_of_csv_html_xml(
            makecsv=makecsv,
            makehtml=makehtml,
            makexml=makexml,
            metadata_dictionary=return_dict,
            name=OPENALEX_RESULT,
        )

        return result_dict

[petermr]

First commit of V20 , guided by ChatGPT

✅ 1. In Jupyter Notebook / JupyterLab
Use Matplotlib with the %matplotlib notebook magic for interactive zoom/pan:

python
Copy
Edit
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 10, 100)
y = np.sin(x)

plt.figure()
plt.plot(x, y)
plt.title("Interactive Plot")
plt.xlabel("X")
plt.ylabel("Y")
plt.grid(True)
plt.show()
📝 %matplotlib inline gives static images, but %matplotlib notebook allows pan/zoom.

✅ 2. In a Python Script (Desktop GUI)
Use Matplotlib + plt.show() to pop up a window:

python
Copy
Edit
import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5]
y = [2, 3, 5, 7, 11]

plt.plot(x, y)
plt.title("Interactive GUI Plot")
plt.show()
This opens a separate window with interactive tools (zoom, pan, save, etc.).

✅ 3. Interactive Web Plot (Plotly)
Use Plotly for rich interactive HTML-based plots:

python
Copy
Edit
import plotly.express as px
import numpy as np
import pandas as pd

x = np.linspace(0, 10, 100)
y = np.sin(x)
df = pd.DataFrame({'x': x, 'y': y})

fig = px.line(df, x='x', y='y', title='Interactive Plotly Graph')
fig.show()
Works in Jupyter or web-based apps. Hover, zoom, pan, download supported.

✅ 4. With Interactive Widgets (Jupyter + ipywidgets)
python
Copy
Edit
from ipywidgets import interact
import matplotlib.pyplot as plt
import numpy as np

def plot_sine(freq=1.0):
    x = np.linspace(0, 2*np.pi, 500)
    y = np.sin(freq * x)
    plt.plot(x, y)
    plt.title(f"Sine Wave: {freq} Hz")
    plt.grid(True)
    plt.show()

interact(plot_sine, freq=(0.1, 10.0, 0.1))
Lets you create sliders/dropdowns for dynamic plots in Jupyter.

Let me know your setup (e.g., Jupyter, VSCode, terminal, or web app), and I’ll tailor a solution further.








You said:
I have run python3 -m build and get a build error with
Writing pygetpapers-2.0.0/setup.cfg
Creating tar archive
removing 'pygetpapers-2.0.0' (and everything under it)
* Building wheel from sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools>=61.0
  - wheel
* Getting build dependencies for wheel...
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
    main()
  File "/opt/anaconda3/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel
    return hook(config_settings)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/ft/7j605bsd10l0ftqygyxjjflh0000gq/T/build-env-gmujtkjx/lib/python3.12/site-packages/setuptools/build_meta.py", line 331, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=[])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/ft/7j605bsd10l0ftqygyxjjflh0000gq/T/build-env-gmujtkjx/lib/python3.12/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires
    self.run_setup()
  File "/private/var/folders/ft/7j605bsd10l0ftqygyxjjflh0000gq/T/build-env-gmujtkjx/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup
    exec(code, locals())
  File "<string>", line 10, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/ft/7j605bsd10l0ftqygyxjjflh0000gq/T/build-via-sdist-edqvot9i/pygetpapers-2.0.0/pygetpapers/config.ini'


ChatGPT said:
This error is telling you that the file pygetpapers/config.ini is missing during the build process:

vbnet
Copy
Edit
FileNotFoundError: [Errno 2] No such file or directory: '.../pygetpapers/config.ini'
This usually happens when:

✅ Your code expects config.ini to be present at runtime or build time.

❌ But it's not included in the source distribution (sdist) created by python3 -m build.

✅ Solution: Include config.ini in your build
To fix this, you need to tell setuptools to include config.ini in your package.

Step 1: Add config.ini to your MANIFEST.in
In the root of your project, create or edit the MANIFEST.in file:

ini
Copy
Edit
include pygetpapers/config.ini
You can also include all .ini files:

ini
Copy
Edit
recursive-include pygetpapers *.ini
Step 2: Ensure setuptools includes non-Python files
In your setup.cfg, make sure you have:

ini
Copy
Edit
[options]
include_package_data = True
And if you're using pyproject.toml, make sure you have:

toml
Copy
Edit
[tool.setuptools]
include-package-data = true
Step 3: Rebuild the package
Clean previous builds:

bash
Copy
Edit
rm -rf dist/ build/ *.egg-info
Then rebuild:

bash
Copy
Edit
python3 -m build
✅ Optional: Verify included files
You can check what files are in the final sdist:

bash
Copy
Edit
tar -tzf dist/pygetpapers-2.0.0.tar.gz
Or inspect the wheel:

bash
Copy
Edit
unzip -l dist/pygetpapers-2.0.0-py3-none-any.whl
Let me know if config.ini is generated at runtime or bundled manually — the solution may differ slightly.








You said:
 I still get the same error. Here is my filestore and contents 
(base) pm286macbook-2:pygetpapers pm286$ ls
LICENSE			init.py			setup.py
MANIFEST.in		pygetpapers		tests
README.md		pyproject.toml
dist			requirements.txt
(base) pm286macbook-2:pygetpapers pm286$ ls pygetpapers/
__init__.py		extensions.py		repository
__pycache__		pgexceptions.py		repositoryinterface.py
config.ini		pygetpapers.egg-info	requirements-dev.txt
download_tools.py	pygetpapers.py
(base) pm286macbook-2:pygetpapers pm286$ more pygetpapers/config.ini 
[pygetpapers]
version=1.2.5a2

[europe_pmc]
query_url=https://www.ebi.ac.uk/europepmc/webservices/rest/searchPOST
citationurl=https://www.ebi.ac.uk/europepmc/webservices/rest/{source}/{identifier}/citations?page=1&pageSize=1000&format=xml
referencesurl=https://www.ebi.ac.uk/europepmc/webservices/rest/{source}/{identifier}/references?page=1&pageSize=1000&format=xml
xmlurl=https://www.ebi.ac.uk/europepmc/webservices/rest/{identifier}/fullTextXML
suppurl=https://www.ebi.ac.uk/europepmc/webservices/rest/{identifier}/supplementaryFiles
zipurl= http://europepmc.org/ftp/suppl/OA/{key}/{identifier}.zip
date_query=SUPPORTED
term=SUPPORTED
update=SUPPORTED
restart=SUPPORTED
class_name=EuropePmc
library_name= europe_pmc
features_not_supported = ["filter",]




ChatGPT said:
Thanks — you've done most things correctly, but here's what's still missing or going wrong:

Your config.ini file is inside the pygetpapers/ directory but is not being included in your final distribution. The most likely causes are either:

MANIFEST.in is incorrect or ignored.

The config.ini file is not marked as part of the package data in setup.cfg or pyproject.toml.

Let’s walk through a guaranteed fix.

✅ Step-by-Step Fix
✅ 1. MANIFEST.in — Include the file
Make sure it includes the exact path:

ini
Copy
Edit
include pygetpapers/config.ini
You can test this works by running:

bash
Copy
Edit
python3 -m build --sdist
tar -tzf dist/pygetpapers-*.tar.gz | grep config.ini
If config.ini doesn't appear in that list, the build system is ignoring it.

✅ 2. Ensure setup.cfg or pyproject.toml includes it
Since you're using pyproject.toml, ensure it looks something like this:

toml
Copy
Edit
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[tool.setuptools]
include-package-data = true

[tool.setuptools.packages.find]
where = ["."]
✅ 3. Add a package_data key (if needed)
Although include-package-data = true should be enough, some builds still need an explicit declaration. So in setup.cfg (if you have one) or as a fallback to setup.py, include:

In setup.py (if you’re using that instead of pure config):

python
Copy
Edit
from setuptools import setup, find_packages

setup(
    name="pygetpapers",
    version="2.0.0",
    packages=find_packages(),
    include_package_data=True,
    package_data={"pygetpapers": ["config.ini"]},
)
✅ 4. Clean all build artifacts
Before rebuilding, remove all old build files:

bash
Copy
Edit
rm -rf build/ dist/ pygetpapers.egg-info/
Then rebuild:

bash
Copy
Edit
python3 -m build
✅ 5. Validate the final distribution
Check that config.ini is inside the .tar.gz and the .whl:

bash
Copy
Edit
tar -tzf dist/pygetpapers-2.0.0.tar.gz | grep config.ini
unzip -l dist/pygetpapers-2.0.0-py3-none-any.whl | grep config.ini
🧠 Why this matters
Your error happens after source distribution is created and during wheel build from sdist, where the file is not present in the temporary build folder. So the fix must ensure config.ini is:

Included in sdist and wheel

Recognized as package data

Let me know if you're using setup.cfg heavily — I can show how to do this purely from that config file without setup.py.








You said:
OK i will remove setup.py


ChatGPT said:
Perfect — if you're going all-in on the modern pyproject.toml + setup.cfg approach (no setup.py), here’s how to make sure your config.ini is properly included without needing setup.py at all.

✅ Summary: No setup.py, just pyproject.toml, setup.cfg, and MANIFEST.in
✅ 1. pyproject.toml
Make sure you have this:

toml
Copy
Edit
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"
✅ 2. setup.cfg
Here’s what you need in setup.cfg:

ini
Copy
Edit
[metadata]
name = pygetpapers
version = 2.0.0
description = [Your description here]
long_description = file: README.md
long_description_content_type = text/markdown
author = [Your name]
license = MIT
license_files = LICENSE
url = https://github.com/yourusername/pygetpapers

[options]
packages = find:
include_package_data = True
python_requires = >=3.7
install_requires =
    requests
    tqdm
    # add other dependencies here

[options.package_data]
pygetpapers = config.ini  # Or *.ini if more files

[options.packages.find]
where = .
✅ Note: You need both include_package_data = True and [options.package_data] for best compatibility.

✅ 3. MANIFEST.in
Your MANIFEST.in should include:

ini
Copy
Edit
include pygetpapers/config.ini
Or to include any .ini file inside pygetpapers/:

ini
Copy
Edit
recursive-include pygetpapers *.ini
✅ 4. Clean and Rebuild
bash
Copy
Edit
rm -rf build/ dist/ pygetpapers.egg-info/
python3 -m build
Then confirm:

bash
Copy
Edit
tar -tzf dist/pygetpapers-2.0.0.tar.gz | grep config.ini
unzip -l dist/pygetpapers-2.0.0-py3-none-any.whl | grep config.ini
Let me know if you want help switching your full setup to this format — I can generate a complete working pyproject.toml and setup.cfg for your project.

[petermr]

Committed V20 branch

git push --set-upstream origin v20
Enumerating objects: 12, done.
Counting objects: 100% (12/12), done.
Delta compression using up to 12 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 2.14 KiB | 2.14 MiB/s, done.
Total 8 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
remote: 
remote: Create a pull request for 'v20' on GitHub by visiting:
remote:      https://github.com/petermr/pygetpapers/pull/new/v20
remote: 
To https://github.com/petermr/pygetpapers
 * [new branch]      v20 -> v20
branch 'v20' set up to track 'origin/v20'.
(base) pm286macbook-2:pygetpapers pm286$ 

[petermr]

NEW V20 with README
extensive GETTING_STARTED