SLR: scopus raw query support

[LoayTarek5]

Related issues and pull requests

PR Description

performRawSearchQueryPaged sends the query string as Scopus's query parameter, bypassing ScopusQueryTransformer, so SLR researchers can use catalog-native syntax, URL building is extracted into a shared buildSearchURL helper used by both the translated and raw paths, also the raw path intentionally bypasses client-side year filtering (same as IEEE), guaranteed by resetting the transformer

Steps to test

AI usage

claude-opus-4.6 to investigate files and review my code

Checklist

• I own the copyright of the code submitted and I license it under the MIT license
• If AI tools were used, I disclosed them in the "AI usage" section and reviewed, understood, and take full ownership of all AI-generated code
• I manually tested my changes in running JabRef (always required)
• I added JUnit tests for changes (if applicable)
• [/] I added screenshots in the PR description (if change is visible to the user)
• [/] I added a screenshot in the PR description showing a library with a single entry with me as author and as title the issue number
• [/] I described the change in CHANGELOG.md in a way that can be understood by the average user (if change is visible to the user)
• [/] I checked the user documentation for up to dateness and submitted a pull request to our user documentation repository

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Scopus: add raw (native) query support for paged SLR searches
✨ Enhancement 🧪 Tests 🕐 20-40 Minutes

Description

• Add paged raw-query search path for Scopus, bypassing query transformation.
• Extract shared Scopus URL construction into a reusable helper.
• Add a regression test for blank raw queries returning an empty page.

Diagram

graph TD
  A["SLR search caller"] --> B["Scopus fetcher"]
  B --> C["ScopusQueryTransformer"] --> D["buildSearchURL"] --> E{{"Scopus Search API"}} --> F["Scopus parser"] --> G["Page<BibEntry>"]
  B --> D --> E

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Disable filtering by setting transformer = null in raw path

• ➕ More explicit signal to shouldIncludeEntry that no filtering should occur
• ➕ Avoids any future reliance on transformer default state
• ➖ Requires ensuring no other code assumes transformer is non-null after any search call
• ➖ May diverge from IEEE’s established pattern of always instantiating a transformer

2. Refactor common raw-paged search logic into PagedSearchBasedParserFetcher default method

• ➕ Reduces duplicated try/catch and stream-handling across fetchers (IEEE/Scopus/others)
• ➕ Centralizes consistent error handling for raw search implementations
• ➖ Harder to fit differing URL parameter conventions and per-fetcher concerns (e.g., year/journal params)
• ➖ Larger change footprint than this PR’s focused enhancement

Recommendation: Current approach is reasonable and consistent with IEEE: introduce a dedicated raw paged method and reuse a shared URL builder. The one improvement to consider is making the “no year filtering” intent more explicit (e.g., transformer = null) rather than relying on a fresh transformer instance having empty bounds; however, the existing implementation is acceptable given current transformer semantics and the added blank-query guard/test.

Files changed (2) +43 / -6

Enhancement (1) +36 / -6

Scopus.javaAdd paged raw-query search and shared URL builder +36/-6

Add paged raw-query search and shared URL builder

• Implements performRawSearchQueryPaged to send the raw query directly as Scopus’s 'query' parameter, bypassing ScopusQueryTransformer. Extracts URL construction into buildSearchURL for reuse by both translated and raw query flows, and wraps malformed URL / IO / parse errors into FetcherException with more specific messages. Resets the transformer for the raw path to avoid stale year bounds influencing shouldIncludeEntry.

jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java

Tests (1) +7 / -0

ScopusTest.javaTest that blank raw query returns an empty page +7/-0

Test that blank raw query returns an empty page

• Adds a unit test verifying performRawSearchQueryPaged("", 0) returns a Page with empty content.

jablib/src/test/java/org/jabref/logic/importer/fetcher/ScopusTest.java

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Transformer data race ✓ Resolved 🐞 Bug ☼ Reliability

Description

Scopus stores year-filter state in the instance field transformer, which is mutated in both
getURLForQuery and the new performRawSearchQueryPaged, while parsing later reads the same field
in shouldIncludeEntry. Because SLR crawling uses nested parallelStream() over queries and
fetchers, the same Scopus instance can be called concurrently and overwrite transformer
mid-parse, causing incorrect inclusion/exclusion of entries (notably year filtering) or
non-deterministic results.

Code

jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java[R69-80]

+    public Page<BibEntry> performRawSearchQueryPaged(String rawQuery, int pageNumber) throws FetcherException {
+        if (rawQuery.isBlank()) {
+            return new Page<>(rawQuery, pageNumber, List.of());
+        }
+        // reset the transformer so the parser's year filter (shouldIncludeEntry) carries no stale bounds
+        // the raw path intentionally bypasses year filtering, consistent with IEEE
+        transformer = new ScopusQueryTransformer();
+        try {
+            URL url = buildSearchURL(rawQuery, pageNumber);
+            try (var stream = getUrlDownload(url).asInputStream()) {
+                return new Page<>(rawQuery, pageNumber, getParser().parseEntries(stream));
+            }

Evidence

StudyFetcher runs queries and fetchers concurrently using parallelStream(), which can invoke the
same fetcher instance from multiple threads at once. Scopus stores per-request filtering state in
an instance field that is written before network/parse and read during parsing, so concurrent
invocations can overwrite that state mid-parse and change filtering behavior.

jablib/src/main/java/org/jabref/logic/crawler/StudyFetcher.java[38-67]
jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java[45-88]
jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java[110-193]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`Scopus` keeps request-specific filtering state in a mutable instance field (`transformer`). The new `performRawSearchQueryPaged` also assigns to this field, and parsing consults it later (`shouldIncludeEntry`). In the SLR crawler, fetchers are reused across concurrent parallel streams, so multiple requests can race and overwrite `transformer` while another request is still parsing, producing incorrect/non-deterministic results.
### Issue Context
- `StudyFetcher.crawl()` uses nested `parallelStream()` and reuses the same fetcher instances across queries.
- `Scopus` parsing-time filtering reads mutable shared state.
### Fix Focus Areas
- jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java[45-88]
- jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java[110-193]
- jablib/src/main/java/org/jabref/logic/crawler/StudyFetcher.java[38-67]
### Suggested fix approaches (pick one)
1) **ThreadLocal state (minimal invasive):**
- Replace `@Nullable private ScopusQueryTransformer transformer;` with a `ThreadLocal<@Nullable ScopusQueryTransformer>`.
- In `getURLForQuery` / `performRawSearchQueryPaged`, set the thread-local transformer.
- In `shouldIncludeEntry`, read from the thread-local.
2) **Eliminate shared state (preferred design):**
- Stop using a mutable field for filtering.
- Capture start/end year bounds per request and apply filtering using locals (e.g., filter the parsed list in `performSearchPaged`/`performRawSearchQueryPaged`), or refactor parsing so the filter predicate is constructed per request and does not depend on shared mutable fields.
Either approach should ensure concurrent crawls cannot affect each other’s year filtering.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. buildSearchURL rawQuery misleading ✓ Resolved 📘 Rule violation ⚙ Maintainability

Description

buildSearchURL documents/labels its argument as a raw query that bypasses
ScopusQueryTransformer, but it is also used by getURLForQuery with a transformed query string.
This mismatched naming and documentation can mislead future changes and reduce maintainability.

Code

jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java[R90-99]

+    /// Builds the Scopus search API URL for the given raw query string and page number.
+    /// The raw query is sent verbatim as the `query` parameter, bypassing [ScopusQueryTransformer].
+    ///
+    /// @param rawQuery   the query string to pass directly as the `query` parameter
+    /// @param pageNumber zero based page number; converted to a zero based `start` offset.
+    private URL buildSearchURL(String rawQuery, int pageNumber) throws URISyntaxException, MalformedURLException {
   URIBuilder uriBuilder = new URIBuilder(API_URL);
   importerPreferences.getApiKey(FETCHER_NAME).ifPresent(apiKey -> uriBuilder.addParameter("apiKey", apiKey));
-
-        if (!transformedQuery.isBlank()) {
-            uriBuilder.addParameter("query", transformedQuery);
+        if (!rawQuery.isBlank()) {
+            uriBuilder.addParameter("query", rawQuery);

Evidence

PR Compliance IDs 5 and 6 require intention-revealing naming and clear, consistent documentation. In
the changed code, getURLForQuery passes transformedQuery into buildSearchURL(...), while
buildSearchURL is documented and parameter-named as if it only accepts a raw query that bypasses
the transformer, creating a naming/documentation mismatch.

AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication: AGENTS.md: Follow JabRef Java Style: Naming, SRP, Small Focused Methods, No Duplication
AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful: AGENTS.md: Variable Names Must Be Correctly Spelled and Meaningful
jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java[62-66]
jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java[90-100]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`buildSearchURL` takes a `String rawQuery` and its `///` comment states the query is sent verbatim and bypasses `[ScopusQueryTransformer]`, but the method is also called from `getURLForQuery(...)` with a transformed query string. This makes the parameter name and documentation inaccurate.
## Issue Context
The helper is shared by both the translated-query and raw-query code paths, so the parameter and docs should reflect that it accepts an already-prepared Scopus `query` string (raw or transformed).
## Fix Focus Areas
- jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java[62-66]
- jablib/src/main/java/org/jabref/logic/importer/fetcher/Scopus.java[90-107]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[koppor]

Hi @LoayTarek5 — I pushed two commits to this branch (using maintainer edits):

1. Merged current main — this branch predated SLR: ACM raw query #16139, which was merged in the meantime.
2. Simplified the Scopus raw path to match SLR: ACM raw query #16139.

#16139 hoisted the raw-query fetch loop into the base interfaces: PagedSearchBasedParserFetcher.performRawSearchQueryPaged now handles the blank-query short-circuit + download/parse/post-cleanup, and paged fetchers only supply a getURLForRawQuery(String, int) hook. IEEE and SpringerNatureWebFetcher were migrated to that shape in the same PR.

So this branch's bespoke performRawSearchQueryPaged override (plus its inline fetch loop) was reintroducing the duplication #16139 had just removed. It's now replaced by the single hook:

@Override
public URL getURLForRawQuery(String rawQuery, int pageNumber) throws URISyntaxException, MalformedURLException {
    return buildSearchURL(rawQuery, pageNumber);
}

Behavior is unchanged: year filtering stays in performSearchPaged, so the raw path remains unfiltered as before, and getBibEntries adds the standard post-cleanup (a no-op for Scopus). Note that — unlike IEEE — Scopus needs no transformer reset in the hook, because its year filtering is applied to results in performSearchPaged rather than inside the parser.

Verified locally: :jablib:compileJava is clean and ScopusTest.performRawSearchQueryPagedWithBlankQueryReturnsEmptyPage passes on the base-class default. The networked fetcher tests will run in CI.

[InAnYan]

I haven't really run the code, but code-wise seems to be okay