Skip to content

[feature] Add ModuleFactory and IndexFactory SPI for auto-discovery of bundled modules and indexes#6551

Open
duncdrum wants to merge 19 commits into
eXist-db:developfrom
duncdrum:dp-module-spi
Open

[feature] Add ModuleFactory and IndexFactory SPI for auto-discovery of bundled modules and indexes#6551
duncdrum wants to merge 19 commits into
eXist-db:developfrom
duncdrum:dp-module-spi

Conversation

@duncdrum

@duncdrum duncdrum commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Introduces ServiceLoader-based SPI for both XQuery modules and index modules so that all bundled implementations are auto-discovered at startup without explicit conf.xml entries. The canonical conf.xml template is trimmed accordingly — new installations get a leaner config by default. Existing hand-maintained configs continue to work unchanged: explicit entries always take precedence over SPI discovery.

Builds on the @enabled attribute from #6550 (itself depending on #6528, #6530, #6531).

Closes #3062 — delivers both parts of the issue: the @enabled attribute (in #6550) and the ServiceLoader/CDI-style autodiscovery requested here.

What Changed

ModuleFactory SPI

  • New org.exist.xquery.ModuleFactory interface: getNamespaceURI() + create(String, Map<String,List<?>>, XQueryContext) matching the existing Module factory contract
  • ServiceLoader<ModuleFactory> wired into Configuration.configureModules(): SPI runs first, then the conf.xml loop; an explicit <module> entry for the same namespace URI overwrites the SPI entry, and enabled="no" suppresses it
  • All 27 bundled XQuery modules register their ModuleFactory implementations in META-INF/services/org.exist.xquery.ModuleFactory
  • Configuration.class.getClassLoader() used explicitly so the loader survives embedding scenarios

IndexFactory SPI

  • New org.exist.indexing.IndexFactory interface: getId() + create(BrokerPool, Path, Element)
  • ServiceLoader<IndexFactory> wired into IndexManager initialisation: SPI entries whose id does not appear as a live <module> entry in conf.xml are registered automatically; enabled="no" suppresses without requiring JAR removal
  • All bundled index modules (Lucene, ngram, range, sort, spatial) wired; index name defaults to the config id when registered via SPI without a conf.xml element

Vector model registry

  • Integrated with Configuration; @enabled on <vector-models> children so individual models can be disabled; suppressed WARN log for unavailable models (logged at DEBUG instead)

exist-distribution/src/main/config/conf.xml (canonical template)

  • <builtin-modules> section trimmed — the 27 bundled modules no longer listed individually
  • <modules> (index) section trimmed — the bundled index modules no longer listed individually
  • Third-party / optional entries retained with comments explaining how to add custom modules

Compatibility

  • Explicit <module> entries in conf.xml always take precedence over SPI discovery for the same namespace URI or index id — zero upgrade action required for hand-maintained configs
  • To suppress a bundled module or index without removing its JAR, add enabled="no" to its entry
  • New META-INF/services/ files are JAR-internal; no classpath changes required by consumers

Test Plan

  • mvn validate -pl exist-distribution -Ddependency-check.skip=true — canonical conf.xml still validates
  • mvn test -pl exist-core -Ddependency-check.skip=true -Ddocker=false — full unit suite passes
  • Manual: start eXist-db with the trimmed conf.xml; all 27 built-in XQuery modules available in eXide
  • Manual: start eXist-db; Lucene, ngram, and range indexes are active without explicit conf.xml entries
  • Manual: add enabled="no" to a module's conf.xml entry; confirm it is not loaded even though SPI would discover it

🤖 Generated with Claude Code

@duncdrum duncdrum requested a review from a team as a code owner July 6, 2026 18:38
@duncdrum duncdrum added the needs documentation Signals issues or PRs that will require an update to the documentation repo label Jul 6, 2026
Comment thread exist-core/src/main/java/org/exist/util/SaxonConfiguration.java Outdated
…antic changes

Add a mvn validate execution (xml:transform@schema-governance) that catches
semantic edits to the seven native XSD schemas without a matching @Version
bump; bump all seven XSDs to reflect post-2023 semantic changes.

Closes eXist-db#6190
duncdrum added 3 commits July 7, 2026 15:32
…consolidate governance CI

Add schemaVersion="x.y.z" to all five canonical config instances so
administrators can see at a glance which XSD version a config file targets,
and so startup code can log a DEBUG or WARN when it doesn't match.
Consolidate the four separate governance shell scripts into a single
governance.xsl, making the version-bump check fully Maven-native. Add
SchemaVersionSyncTest to guard the generated constants against drift.
Include schema/ (conf.xsd, collection.xconf.xsd, descriptor.xsd,
controller-config.xsd, mime-types.xsd, users.xsd, server.xsd,
security-manager.xsd, expath-pkg.xsd and its extensions) in the
tarball, zip, Docker image, IzPack installer, and macOS .app bundle
as $EXIST_HOME/schema/. Also fix the "abrev" -> "abbrev" typo
in expath-pkg.xsd.

Closes eXist-db#6189
Partially addresses eXist-db#6008
…generate SchemaVersion constants

Extend the Maven validate phase to check every schema/*.xsd against the
W3C XSD 1.1 meta-schema (upgrading the bundled XMLSchema.xsd/dtd to the
2009 XSD 1.1 revision); caught and removed five xsi:type="dcterms:W3CDTF"
appinfo annotations with no backing schema, bumping the affected schema
@Version values. Wire a generate-sources execution that reads each
governed XSD's xs:schema/@Version and emits SchemaVersion.java constants
so they can never drift from the schemas.

Closes eXist-db#5541
duncdrum and others added 15 commits July 7, 2026 15:57
…line

Wire the XML Catalog API into Saxon's XsltCompiler so xsl:import,
xsl:include, and document() URIs resolve via the eXist-db entity catalog
(Closes eXist-db#350). Generalize MutableCollection's at-store-time validation to
route through a JAXP 1.1 SchemaFactory when the schema or its catalog-
resolved meta-schema requires XSD 1.1; extract the detection logic into
Xsd11SchemaDetection so validation:jaxp() and org.exist.validation.Validator
share the same probe-compile path. Suppress the spurious xmlresolver WARNING
for the non-existent "default" catalog entry.

Closes eXist-db#5541
Set a Jetty stop timeout so shutdown does not block indefinitely on
active connections; move ShutdownListenerImpl.shutdown() to a daemon
thread so it does not deadlock the Jetty shutdown-hook thread that
also holds the BrokerPool write lock.
…nical at build time

Add a Maven exec plugin execution that transforms the canonical
exist-distribution conf.xml through module-specific XSLT strip-sheets,
writing per-module fixture conf.xml and controller-config.xml files into
src/test/resources-filtered at process-test-resources time. Eliminates
~35 hand-maintained fixture copies that drift from canonical on every
schema edit.
…o URN imports

Register the repo-root schema/catalog.xml in the Maven xml-maven-plugin
catalog chain so XSD-against-XSD validation resolves locally without
network access. Migrate all 35 XSLT strip-sheets from bare http:// imports
to URN-mapped imports via the catalog. Document the new-native-schema
checklist and fix the profile location note in the README.
…X parser features in conf.xml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…yes", scheduler jobs as live enabled="no" entries, @enabled on <parameter> and <property> in conf.xsd; bump schema to 2.5.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…2.6.0

- Add memorySizeType (pattern [0-9]+[KkMmGgTt]?|-1); apply to cacheSize,
  collectionCache, minDiskSpace, recovery/@SiZe
- pool/@min, @max: xs:integer → xs:positiveInteger
- lock-table/@trace-stack-depth: xs:int → xs:nonNegativeInteger
- indexer/create/@type: xs:anySimpleType → xs:string
- transformer/@caching: xs:string → yes_no with default="yes"
- scheduler/job/@period: xs:string → xs:positiveInteger
- vector-models/model/@Dimension: xs:integer → xs:positiveInteger
- Fix swapped docs on raise-error-on-failed-retrieval / enforce-index-use
- Fix "group-comit" typo in recovery/@group-commit doc
- Fix rpc-server in-memory-size default 4196 → 4096
- Fill in TODO docs: indexer/create attrs, flushAfter, n, xquery module
  attrs (class/uri/src), xupdate/@growth-factor

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ure triggers

ParametersExtractor.parseProperties() used case-sensitive .equals("no") while
every other enabled check in Configuration.java uses .equalsIgnoreCase(). An
enabled="NO" parameter would be silently included instead of skipped.

generate-conf-fixture.xsl stripped @enabled from kept module/index nodes but
not from surviving triggers, so BouncyCastle and URLStreamHandler emitted an
explicit enabled="yes" in generated fixtures — inconsistent with the module
treatment and noise in fixture diffs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nonical conf.xml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…abled to <vector-models>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…canonical conf.xml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…f.xml element

AbstractIndex.configure() only sets name from config.getAttribute("id") when
config != null. IndexFactory SPI registration passes config=null, leaving name
null. IndexController.getWorkerByIndexName() matches on name, so SPI-registered
indexes (range-index, ngram-index, sort-index, lucene-index) were never found,
causing NPE at every range:index-keys-for-field() call.

Fix: after configure(), if name is still null, call setName(id) with the
configuration id that was used to key the indexers map.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…es covered by existing imports

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

@line-o line-o left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolving modules to load by default over SPI received pushback on the community call on 2026-07-06
Security concerns need to be addressed before we can consider pulling this in.

@dizzzz specific concern was: "I see autoloading modules that are not mentioned in the configuration as a risk."

My concern is: if bundled modules are left out of the configuration I have to know they are there in order to disable them again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs documentation Signals issues or PRs that will require an update to the documentation repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

modify conf.xml

3 participants