S3DAO fails with HCP (Hitachi Content Platform) S3, Owner field missing in ListBuckets response

[abowman-unige]

Environment

• seb-sps version: v1.2-latest
• S3 provider: Hitachi Content Platform (HCP) v9.7.3.14
• HCP S3 API enabled: yes (Hitachi API for Amazon S3)

Description

S3DAO.init() fails on startup when connecting to a Hitachi Content Platform S3 endpoint.
HCP's ListBuckets response does not include the element, which the MinIO client
requires as mandatory in ListAllMyBucketsResult.

Error log

sps-webservice | INFO [main]:[S3DAO] There is no certificate detected for S3 binding. Try without...
sps-webservice | ERROR [SERVICE_INIT] ---> Error printing buckets in S3 service
org.simpleframework.xml.core.ValueRequiredException: Unable to satisfy
@org.simpleframework.xml.Element(name="Owner", type=void.class, required=true)
on field 'owner' private io.minio.messages.Owner
io.minio.messages.ListAllMyBucketsResult.owner at line 1

Caused by: io.minio.errors.ErrorResponseException:
The XML you provided was not well-formed or was not valid.
at ch.ethz.seb.sps.server.datalayer.dao.impl.S3DAO.isBucketExisting(S3DAO.java:130)
at ch.ethz.seb.sps.server.datalayer.dao.impl.S3DAO.init(S3DAO.java:67)

Root cause

HCP returns a valid S3-compatible ListBuckets response but omits the block.
The MinIO client parses this response and throws because owner is annotated as
required=true

Standard AWS S3 and MinIO always include , but some S3-compatible providers
(HCP and potentially others) consider it optional per the S3 spec.

Expected behavior

Handle a missing field gracefully and continue

[anhefti]

Hi

Hmm it seems that this issue is related with this bug in the Java MinIO API we use: minio/minio-java#921
But this should have been resolved years ago and although we don't use the newest version of the MinIO API for Java, it should have been fixed within the version 8.5.11.

But beside of that, it seems that there are two different errors in play here. One while trying to print the existing bucked list and another one when trying to verify if a bucked with specific name already exists (Caused by). The printing task will just skip if it is not working correctly but the isBucketExisting task will terminate the S3 init when not possible. So it seems this is the problem, but maybe both are rooted in the same cause inside the MinIO API.

If you don't mind, can you please provide us the whole section of the server logs for this case, so we can probably see better what is going on.

Currently I cannot see what I can do against it beside trying to go to the newest version of MinIO API and hope the bug is resolved there and everything else is still working as expected. I can do this for v1.2-latest and test it on our site first, if it is working for us, I can tell you to pull the new image and try on your side.

[anhefti]

I have updated MinIO client to version 8.6.0 (the last version before latest major version with a lot of changes). Locally tested for out environment it is working fine so fare.

Please pull the new image and try if the error still remains or if it is solved now.

[abowman-unige]

Hi,

Thank you for the quick update. Unfortunately the issue still remains with the new image (v1.2-latest-3c252f2).

Here are the full logs:

sps-webservice  | 21.04.2026 09:06:21.874 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:21.874 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:21.874 INFO  [main]:[SERVICE_INIT] ----> Version: v1.2-latest-3c252f2
sps-webservice  | 21.04.2026 09:06:21.875 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:21.875 INFO  [main]:[SERVICE_INIT] ----> Active profiles: [prod]
sps-webservice  | 21.04.2026 09:06:21.875 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:21.876 INFO  [main]:[SERVICE_INIT] ----> Context Path: /
sps-webservice  | 21.04.2026 09:06:21.876 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:21.876 INFO  [main]:[SERVICE_INIT] ----> *********************************************************
sps-webservice  | 21.04.2026 09:06:21.876 INFO  [main]:[SERVICE_INIT] ----> *** Webservice starting up...                         ***
sps-webservice  | 21.04.2026 09:06:21.876 INFO  [main]:[SERVICE_INIT] ----> *********************************************************
sps-webservice  | 21.04.2026 09:06:21.876 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:21.876 INFO  [main]:[SERVICE_INIT] ----> Register Webservice: ba4738ae-143a-4978-a9df-6e7cb1e9106a_v1.2-latest-3c252f2
sps-webservice  | 21.04.2026 09:06:21.881 INFO  [main]:[com.zaxxer.hikari.HikariDataSource] HikariPool-1 - Starting...
sps-webservice  | 21.04.2026 09:06:21.942 INFO  [main]:[com.zaxxer.hikari.pool.HikariPool] HikariPool-1 - Added connection org.mariadb.jdbc.Connection@2d541e01
sps-webservice  | 21.04.2026 09:06:21.951 INFO  [main]:[com.zaxxer.hikari.HikariDataSource] HikariPool-1 - Start completed.
sps-webservice  | 21.04.2026 09:06:22.013 INFO  [main]:[SERVICE_INIT] ----> Successfully register Webservice instance. uuid: ba4738ae-143a-4978-a9df-6e7cb1e9106a_v1.2-latest-3c252f2, address: 172.19.0.5
sps-webservice  | 21.04.2026 09:06:22.013 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.013 INFO  [main]:[SERVICE_INIT] ----> **** Migration check START ******************************
sps-webservice  | 21.04.2026 09:06:22.013 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.013 INFO  [main]:[SERVICE_INIT] ----> Check database status
sps-webservice  | 21.04.2026 09:06:22.036 INFO  [main]:[org.flywaydb.core.FlywayExecutor] Database: jdbc:mariadb://seb-server-mariadb/SEBScreenProctoring (MariaDB 10.5)
sps-webservice  | 21.04.2026 09:06:22.074 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.074 INFO  [main]:[SERVICE_INIT] ----> No pending migrations found. Last migration --> 9 --> screenshot data live cache table v1 2
sps-webservice  | 21.04.2026 09:06:22.086 INFO  [main]:[org.flywaydb.core.internal.command.DbValidate] Successfully validated 8 migrations (execution time 00:00.005s)
sps-webservice  | 21.04.2026 09:06:22.087 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.087 INFO  [main]:[SERVICE_INIT] ----> **** Migration check END ********************************
sps-webservice  | 21.04.2026 09:06:22.087 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.087 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.087 INFO  [main]:[SERVICE_INIT] ----> **** Initialize Services... *****************************
sps-webservice  | 21.04.2026 09:06:22.087 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.088 INFO  [main]:[SERVICE_INIT] ----> SessionServiceHealthControl initialized
sps-webservice  | 21.04.2026 09:06:22.088 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.088 INFO  [main]:[SERVICE_INIT] ----> Initialize Live Proctoring Cache Service
sps-webservice  | 21.04.2026 09:06:22.088 INFO  [main]:[SERVICE_INIT] ---->     with update interval: 1000
sps-webservice  | 21.04.2026 09:06:22.103 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.103 INFO  [main]:[SERVICE_INIT] ----> Initialize ScreenshotStore_S3 Service
sps-webservice  | 21.04.2026 09:06:22.104 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.203 INFO  [main]:[ch.ethz.seb.sps.server.datalayer.dao.impl.S3DAO] There is no certificate detected for S3 binding. Try without...
sps-webservice  | 21.04.2026 09:06:22.907 ERROR [main]:[SERVICE_INIT]  ---> Error printing buckets in S3 service org.simpleframework.xml.core.ValueRequiredException: Unable to satisfy @org.simpleframework.xml.Element(name="Owner", type=void.class, data=false, required=true) on field 'owner' private io.minio.messages.Owner io.minio.messages.ListAllMyBucketsResult.owner for class io.minio.messages.ListAllMyBucketsResult at line 1
sps-webservice  | 21.04.2026 09:06:22.955 ERROR [main]:[org.springframework.boot.SpringApplication] Application run failed
sps-webservice  | java.lang.reflect.UndeclaredThrowableException: Failed to invoke event listener method
sps-webservice  | HandlerMethod details:
sps-webservice  | Bean [ch.ethz.seb.sps.server.datalayer.dao.impl.S3DAO]
sps-webservice  | Method [public void ch.ethz.seb.sps.server.datalayer.dao.impl.S3DAO.init() throws java.lang.Exception]
sps-webservice  | Resolved arguments:
sps-webservice  |
sps-webservice  |       at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:382)
sps-webservice  |       at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:237)
sps-webservice  |       at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:168)
sps-webservice  |       at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
sps-webservice  |       at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
sps-webservice  |       at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
sps-webservice  |       at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:452)
sps-webservice  |       at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:385)
sps-webservice  |       at ch.ethz.seb.sps.server.ServiceInit.onApplicationEvent(ServiceInit.java:121)
sps-webservice  |       at ch.ethz.seb.sps.server.ServiceInit.onApplicationEvent(ServiceInit.java:28)
sps-webservice  |       at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
sps-webservice  |       at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
sps-webservice  |       at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
sps-webservice  |       at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:452)
sps-webservice  |       at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:385)
sps-webservice  |       at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109)
sps-webservice  |       at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80)
sps-webservice  |       at java.base/java.lang.Iterable.forEach(Unknown Source)
sps-webservice  |       at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
sps-webservice  |       at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
sps-webservice  |       at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80)
sps-webservice  |       at org.springframework.boot.SpringApplication.run(SpringApplication.java:349)
sps-webservice  |       at org.springframework.boot.SpringApplication.run(SpringApplication.java:1363)
sps-webservice  |       at org.springframework.boot.SpringApplication.run(SpringApplication.java:1352)
sps-webservice  |       at ch.ethz.seb.sps.server.ScreenProctoringServer.main(ScreenProctoringServer.java:24)
sps-webservice  |       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sps-webservice  |       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
sps-webservice  |       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
sps-webservice  |       at java.base/java.lang.reflect.Method.invoke(Unknown Source)
sps-webservice  |       at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:91)
sps-webservice  |       at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53)
sps-webservice  |       at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58)
sps-webservice  | Caused by: java.lang.Exception: Could not retrieve status of bucket: luniproctortst2
sps-webservice  |       at ch.ethz.seb.sps.server.datalayer.dao.impl.S3DAO.isBucketExisting(S3DAO.java:130)
sps-webservice  |       at ch.ethz.seb.sps.server.datalayer.dao.impl.S3DAO.init(S3DAO.java:67)
sps-webservice  |       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sps-webservice  |       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
sps-webservice  |       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
sps-webservice  |       at java.base/java.lang.reflect.Method.invoke(Unknown Source)
sps-webservice  |       at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:365)
sps-webservice  |       ... 31 common frames omitted
sps-webservice  | Caused by: io.minio.errors.ErrorResponseException: The XML you provided was not well-formed or was not valid.
sps-webservice  |       at io.minio.S3Base$1.onResponse(S3Base.java:803)
sps-webservice  |       at io.minio.S3Base$1.onResponse(S3Base.java:639)
sps-webservice  |       at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
sps-webservice  |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
sps-webservice  |       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
sps-webservice  |       at java.base/java.lang.Thread.run(Unknown Source)
sps-webservice  | 21.04.2026 09:06:22.961 INFO  [main]:[org.apache.coyote.http11.Http11NioProtocol] Stopping ProtocolHandler ["http-nio-0.0.0.0-8090"]
sps-webservice  | 21.04.2026 09:06:22.972 INFO  [main]:[SERVICE_INIT] *********************************************************
sps-webservice  | 21.04.2026 09:06:22.972 INFO  [main]:[SERVICE_INIT] **** Gracefully Shutdown of SEB Server instance 0.0.0.0
sps-webservice  | 21.04.2026 09:06:22.972 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.972 INFO  [main]:[SERVICE_INIT] ----> Unregister Webservice: ba4738ae-143a-4978-a9df-6e7cb1e9106a_v1.2-latest-3c252f2
sps-webservice  | 21.04.2026 09:06:22.988 INFO  [main]:[SERVICE_INIT] ---->
sps-webservice  | 21.04.2026 09:06:22.988 INFO  [main]:[SERVICE_INIT] ----> Webservice down
sps-webservice  | 21.04.2026 09:06:22.995 INFO  [main]:[com.zaxxer.hikari.HikariDataSource] HikariPool-1 - Shutdown initiated...
sps-webservice  | 21.04.2026 09:06:23.003 INFO  [main]:[com.zaxxer.hikari.HikariDataSource] HikariPool-1 - Shutdown completed.

[anhefti]

Hi, thanks, I can see now, that the second error is the one that terminates the process. There is one error in the printing but this error is cached and ignored on happening because it is not of importance. But the second one that checks if the bucked with given name is already existing, fails with an arbitrary "The XML you provided was not well-formed or was not valid" exception.

For me this looks like either a bug or an incompatibility within the Java MinIO client using Hitachi Content Platform (HCP) v9.7.3.14.
Maybe also a configuration mismatch between the two but cannot verify this and also have nothing found with quick search. Are there also logs on HCP side that might show what exactly is wrong with the XML?

I can quickly try to go to the newest major version of the MinIO client that was released a month ago and hope there are no braking changes. Then we can try with the new major version.

[anhefti]

Unfortunately the update to major version of MinIO comes with braking changes that also affects the code (must partially be rewritten). We cannot do this within a patch.

I will try to find time to further investigate into this Issue and if we miss a configuration detail that can cause this error but for now this seems to be an incompatibility.

[abowman-unige]

Thanks for looking into this. No worries at all, we're planning to move to a different S3 provider within
the next year anyway, so this isn't blocking us. We also tested external s3 alternative that we can use if needed.

Thanks again for the quick responses!