Replace the Java rdf2json with a faithful Rust port#1306
Draft
jamesamcl wants to merge 3 commits into
Draft
Conversation
A performance-motivated, behaviour-preserving rewrite of uk.ac.ebi.rdf2json.RDF2JSON. It reads RDF triples via oxrdfio, builds the OLS node graph, runs the ~22 annotators, and streams the output JSON via serde_json (one entity at a time, so memory stays bounded on very large ontologies). It implements --mergeOutputWith (streamed via struson) to preserve the production incremental-reload safety net. There is no horned-owl dependency. Output matches the existing testcase fixtures on 99/100 cases (only gitIssue502 differs, because it needs network access to fetch its source ontology). The dataload image builds the binary and the dev-testing / test_dataload.sh flows exercise it; JSON is compared by normalized content rather than byte-for-byte, since serde_json's formatting differs from Gson's while the content is identical. This also corrects the reified-bnode handling the Java areSubgraphsIsomorphic bug got wrong: annotated blank-node edges now merge into a single reified edge instead of being duplicated. The reification testcase fixtures are updated to the corrected output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V7yNmdd6CGnLqd85y3xoon
nextflow (ols_dataload.nf) now invokes the rdf2json Rust binary instead of the Java jar, dropping the JVM heap/entity-limit flags; the status-file grep is made whitespace-tolerant so it matches the pretty-printed status JSON. The Java rdf2json maven module (dataload/pom.xml) and its source (dataload/rdf2json/) are removed; nothing else depended on them. dataload/README.md and dev-testing/dev-local.sh are repointed at the Rust binary. The dataload image still builds the remaining Java tools (merge_configs, reporting, extras); removing Java/Maven from the image entirely is a follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V7yNmdd6CGnLqd85y3xoon
jamesamcl
marked this pull request as draft
June 26, 2026 15:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the Java
rdf2jsondataload step (uk.ac.ebi.rdf2json.RDF2JSON) with a faithful Rust port, for performance — without changing the output contract the rest of the dataload pipeline depends on. This is a behaviour-preserving 1:1 replacement, the first of a planned series.How it works
rdf2json(Rust) reads each ontology's RDF as triples viaoxrdfio, builds the OLS node graph, runs the ~22 annotators (label / definition / synonym / direct+hierarchical parents / ancestors / hierarchy flags & metrics / shortForm / curie / relatedTo / isObsolete / reification / …), and streams the output JSON viaserde_jsonone entity at a time, so memory stays bounded even on very large ontologies.--mergeOutputWith(the production incremental-reload safety net) is implemented via streamingstruson. There is nohorned-owldependency.Validation
test_dataload.shnow compares normalized JSON content rather than byte-for-byte, because serde_json's whitespace differs from Gson's while the content is identical. The single difference,gitIssue502, differs only because it needs network access to fetch its source ontology.--mergeOutputWithexercised for the fail→fallback, succeed→fresh, and unrelated→kept cases.ols_create_manifestconsumes the Rust output and produces a byte-identical manifest.test_api.sh/ apitester) runs in CI.Behaviour change (intended)
The Java
areSubgraphsIsomorphicbug duplicated annotated blank-node edges; the Rust port handles this correctly — annotated bnode edges merge into a single reified edge. Thereification/reified-bnodefixtures are updated to the corrected output.Commits
The dataload image still builds the remaining Java tools (
merge_configs,reporting,extras).Not in this PR (roadmap)
merge_configs/reporting/extras/json2sssomto Rust.🤖 Generated with Claude Code
https://claude.ai/code/session_01V7yNmdd6CGnLqd85y3xoon
Generated by Claude Code