Merge pull request #1044 from silk-framework/feature/hashing-CMEM-7596

CMEM-7596: Add per-value hash operator

Author: edufuga

silk-rules/src/main/resources/org/silkframework/rule/plugins/transformer/value/InputHashTransformer.md

@@ -1,2 +1,29 @@
-Calculates the hash sum of the input values. Generates a single hash sum for all input values combined.
-This operator supports using different hash algorithms from the [Secure Hash Algorithms family](https://en.wikipedia.org/wiki/Secure_Hash_Algorithms) (SHA, e.g. SHA256) and two algorithms from the [Message-Digest Algorithm family](https://en.wikipedia.org/wiki/MD5) (MD2 / MD5). Please be aware that some of these algorithms are not secure due the possibility of collision attacks and other attacks.
+The **Combined input hash** operator produces exactly one hash value covering all input values combined, across all connected input ports. However many values arrive and however many ports are connected, the output is always a single string.
+
+## How combining works
+
+All values from all input ports are fed sequentially into a single hash function — port 1 first, then port 2, and so on. Within each port, values are processed in the order they arrive. No separator is inserted between values or between ports. The hash covers the concatenated byte content of all values in that traversal order.
+
+This means the result depends on both the content and the order of values. The same set of values in a different order produces a different hash. Connecting one port with values `["apple", "banana"]` produces the same hash as connecting two ports with `["apple"]` and `["banana"]` respectively, because the bytes are fed in the same sequence either way.
+
+## Output
+
+The output is a single lowercase hexadecimal string. The length depends on the algorithm: 64 characters for SHA-256, 32 for MD5, 40 for SHA-1, 96 for SHA-384, 128 for SHA-512. If the input is empty, the output is the hash of an empty message.
+
+Values are encoded as UTF-8 before hashing.
+
+## Algorithm parameter
+
+The algorithm parameter selects the hash function. The default is SHA-256. The following algorithms from the [SPARQL 1.1 specification](https://www.w3.org/TR/sparql11-query/#func-hash) are supported:
+
+| SPARQL name | Java name | Notes |
+|-------------|-----------|-------|
+| MD5 | MD5 | Weak — vulnerable to collision attacks. Avoid for security-sensitive use. |
+| SHA1 | SHA-1 | Weak — deprecated for most security purposes. |
+| SHA256 | SHA-256 | Recommended default. |
+| SHA384 | SHA-384 | Stronger than SHA-256. |
+| SHA512 | SHA-512 | Strongest in the SPARQL set. |
+
+Additional algorithms available on the JVM (such as SHA-512/256 and SHA-3 variants) are also accepted. The full list is JVM-dependent and visible in the algorithm parameter dropdown.
+
+Note that the Java names use hyphens (SHA-256, SHA-1) where SPARQL uses none (SHA256, SHA1). Both forms are accepted by this operator.

silk-rules/src/main/resources/org/silkframework/rule/plugins/transformer/value/PerValueHashTransformer.md

@@ -0,0 +1,37 @@
+The **Per-value hash** operator hashes each input value independently and returns one hash per value. The output count always equals the input count — cardinality is preserved.
+
+## SPARQL alignment
+
+This operator produces the same output as the SPARQL 1.1 hash functions applied per value. For a single input value, `SHA256(?x)` in SPARQL returns the same result as this operator with the default SHA256 algorithm.
+
+## Single-input constraint
+
+The operator accepts exactly one input port. Connecting more than one port throws an `IllegalArgumentException`. This constraint exists because per-value hashing is defined relative to a single value sequence — combining values across ports would require choosing a port-merging strategy, which is the behaviour of the **Combined input hash** operator instead.
+
+## Output
+
+Each input value produces one lowercase hexadecimal hash string. The output order matches the input order. If the input is empty, the output is empty — no hash is produced.
+
+Values are encoded as UTF-8 before hashing.
+
+## Algorithm parameter
+
+The algorithm parameter selects the hash function. The default is SHA-256. The five algorithms from the [SPARQL 1.1 specification](https://www.w3.org/TR/sparql11-query/#func-hash) are supported:
+
+| SPARQL name | Java name | Notes |
+|-------------|-----------|-------|
+| MD5 | MD5 | Weak — vulnerable to collision attacks. Avoid for security-sensitive use. |
+| SHA1 | SHA-1 | Weak — deprecated for most security purposes. |
+| SHA256 | SHA-256 | Recommended default. |
+| SHA384 | SHA-384 | Stronger than SHA-256. |
+| SHA512 | SHA-512 | Strongest in the SPARQL set. |
+
+Additional algorithms available on the JVM are also accepted. The full list is JVM-dependent and visible in the algorithm parameter dropdown.
+
+Note that the Java names use hyphens (SHA-256, SHA-1) where SPARQL uses none (SHA256, SHA1). Both forms are accepted by this operator.
+
+## Contrast with Combined input hash
+
+The **Combined input hash** operator feeds all values from all ports into a single hash function and returns one hash regardless of input size. Use it when you need a single fingerprint for a set of values taken together.
+
+Use **Per-value hash** when each value needs its own hash — for example, to hash a column of URIs independently, or to replicate `SHA256(?x)` in SPARQL.

silk-rules/src/main/scala/org/silkframework/rule/plugins/RulePlugins.scala

@@ -115,6 +115,7 @@ class RulePlugins extends PluginModule {
         // Dataset
         classOf[FileHashTransformer] ::
         classOf[InputHashTransformer] ::
+        classOf[PerValueHashTransformer] ::
         // Numeric
         classOf[NumReduceTransformer] ::
         classOf[NumOperationTransformer] ::

silk-rules/src/main/scala/org/silkframework/rule/plugins/transformer/value/HashAlgorithmAutoCompletionProvider.scala

@@ -0,0 +1,32 @@
+package org.silkframework.rule.plugins.transformer.value
+
+import org.silkframework.runtime.plugin.{AutoCompletionResult, ParamValue, PluginContext, PluginParameterAutoCompletionProvider}
+import org.silkframework.workspace.WorkspaceReadTrait
+
+import java.security.Security
+import scala.jdk.CollectionConverters.IterableHasAsScala
+
+/**
+ * Provides autocomplete suggestions for hash algorithm names.
+ *
+ * The algorithm list is sourced from the JVM's registered security providers via [[java.security.Security#getAlgorithms]],
+ * scoped to the `MessageDigest` service type. The result is JVM-dependent — different provider configurations
+ * return different sets. The list is built once on first access and cached.
+ */
+case class HashAlgorithmAutoCompletionProvider() extends PluginParameterAutoCompletionProvider {
+
+  private lazy val algorithms: Seq[String] = Security.getAlgorithms("MessageDigest").asScala.toSeq.sorted
+
+  override def autoComplete(searchQuery: String, dependOnParameterValues: Seq[ParamValue],
+                            workspace: WorkspaceReadTrait)
+                           (implicit context: PluginContext): Iterable[AutoCompletionResult] = {
+    val multiSearchWords = extractSearchTerms(searchQuery)
+    algorithms
+      .filter(r => matchesSearchTerm(multiSearchWords, r.toLowerCase))
+      .map(r => AutoCompletionResult(r, None))
+  }
+
+  override def valueToLabel(value: String, dependOnParameterValues: Seq[ParamValue],
+                            workspace: WorkspaceReadTrait)
+                           (implicit context: PluginContext): Option[String] = None
+}

silk-rules/src/main/scala/org/silkframework/rule/plugins/transformer/value/HashTransformer.scala

@@ -0,0 +1,61 @@
+package org.silkframework.rule.plugins.transformer.value
+
+import org.silkframework.rule.input.InlineTransformer
+
+import java.nio.charset.StandardCharsets
+import java.security.MessageDigest
+import java.util.HexFormat
+
+/**
+ * Digest operations for hash transformer plugins.
+ *
+ * Provides [[toHex]], [[hashValue]], and [[withDigest]] as concrete implementations over the abstract
+ * [[algorithm]] parameter, which each concrete plugin supplies via its own plugin parameter. Subclasses
+ * implement [[apply]] according to their combining or per-value semantics:
+ *  - [[hashValue]] is suited to per-value operators: it is a complete, self-contained operation on a
+ *    single string.
+ *  - [[withDigest]] is suited to combining operators: it loans a reset digest for the duration of a
+ *    single lambda, leaving the caller responsible for updates and finalization within that scope.
+ */
+trait HashTransformer extends InlineTransformer {
+  /** The hash algorithm name, passed directly to [[java.security.MessageDigest#getInstance]].
+   *  Must be recognized by the JVM's security provider registry; unrecognized names cause
+   *  [[java.security.NoSuchAlgorithmException]] at runtime. */
+  def algorithm: String
+
+  // @transient: HexFormat is not serializable. lazy: transient fields are null after deserialization;
+  // lazy ensures the instance is recreated on first use rather than remaining null.
+  @transient private lazy val hexFormat: HexFormat = HexFormat.of()
+
+  // @transient: not serialized. lazy: recreated on first use after deserialization.
+  @transient private lazy val threadLocalDigest: ThreadLocal[MessageDigest] =
+    ThreadLocal.withInitial(() => MessageDigest.getInstance(algorithm))
+
+  /** Encodes a byte array as a fixed-length lowercase hex string. Each byte maps to exactly two
+   *  characters; without zero-padding, a byte with value 5 would render as "5" instead of "05",
+   *  producing a variable-length string that breaks any downstream comparison or length check. */
+  def toHex(bytes: Array[Byte]): String =
+    hexFormat.formatHex(bytes)
+
+  /** Updates the digest with the UTF-8 encoding of [[v]]. */
+  def updateWith(digest: MessageDigest, v: String): Unit =
+    digest.update(v.getBytes(StandardCharsets.UTF_8))
+
+  /** Hashes a single string value and returns its hex digest. A thread-local
+   *  [[java.security.MessageDigest]] instance is reused across calls on the same thread:
+   *  [[java.security.MessageDigest]] is stateful and not thread-safe, so each thread maintains
+   *  its own instance rather than allocating a new one per call. */
+  def hashValue(v: String): String =
+    withDigest { digest =>
+      updateWith(digest, v)
+      toHex(digest.digest())
+    }
+
+  /** Resets the thread-local [[java.security.MessageDigest]] and loans it to the given function.
+   *  The instance is shared on the calling thread; do not retain a reference beyond the function's scope. */
+  def withDigest[A](f: MessageDigest => A): A = {
+    val digest = threadLocalDigest.get()
+    digest.reset()
+    f(digest)
+  }
+}

silk-rules/src/main/scala/org/silkframework/rule/plugins/transformer/value/InputHashTransformer.scala

@@ -1,77 +1,113 @@
 package org.silkframework.rule.plugins.transformer.value
 
 import org.silkframework.rule.annotations.{TransformExample, TransformExamples}
-import org.silkframework.rule.input.InlineTransformer
 import org.silkframework.rule.plugins.transformer.replace.MapTransformerWithDefaultInput
 import org.silkframework.runtime.plugin.annotations.{Param, Plugin, PluginReference}
-import org.silkframework.runtime.plugin.{AutoCompletionResult, ParamValue, PluginContext, PluginParameterAutoCompletionProvider}
-import org.silkframework.workspace.WorkspaceReadTrait
 
-import java.nio.charset.StandardCharsets
-import java.security.{MessageDigest, Security}
-import scala.jdk.CollectionConverters.IterableHasAsScala
+import java.security.MessageDigest
 
+
+/**
+ * Combines all input values across all connected ports into a single hash.
+ *
+ * Values are fed sequentially into a single [[java.security.MessageDigest]] instance — port 1 first, then port 2,
+ * and so on; within each port, values are processed in order. No separator is inserted between values or ports.
+ * The output is always exactly one lowercase hexadecimal string, regardless of how many values or ports are provided.
+ *
+ * @see [[PerValueHashTransformer]] for the per-value variant that produces one hash per input value.
+ */
 @Plugin(
   id = InputHashTransformer.pluginId,
   categories = Array("Value"),
-  label = "Input hash",
-  description = """Calculates the hash sum of the input values. Generates a single hash sum for all input values combined.""",
+  label = "Combined input hash",
+  description = """Calculates a single hash value covering all input values combined, across all input ports. Values are fed into the hash function in port order without any separator between them.""",
   documentationFile = "InputHashTransformer.md",
   relatedPlugins = Array(
+    new PluginReference(
+      id = PerValueHashTransformer.pluginId,
+      description = "The Per-value hash plugin hashes each input value independently and returns one hash per value, preserving cardinality. The Combined input hash plugin instead feeds all values into a single hash function, producing one combined hash regardless of input size."
+    ),
     new PluginReference(
       id = MapTransformerWithDefaultInput.pluginId,
-      description = "One hash value is produced for the entire set of inputs by the Input hash plugin. The Map with default plugin instead keeps a value sequence and rewrites it position by position through the mapping, falling back to the second input where no mapping entry is found."
+      description = "One hash value is produced for the entire set of inputs by the Combined input hash plugin. The Map with default plugin instead keeps a value sequence and rewrites it position by position through the mapping, falling back to the second input where no mapping entry is found."
     )
   )
 )
 @TransformExamples(Array(
   new TransformExample(
+    description = "A single input value produces one combined SHA-256 hash.",
     input1 = Array("input value"),
     output = Array("f708c2afff0ed197e8551c4dd549ee5b848e0b407106cbdb8e451c8cd1479362")
   ),
+  new TransformExample(
+    description = "Multiple values on one input are combined into a single hash.",
+    input1 = Array("apple", "banana"),
+    output = Array("5b692305517af54eb5ae12b9ff89eaf89e31f6a6ee208365886a18b81a2fc2f8")
+  ),
+  new TransformExample(
+    description = "Reversing the value order produces a different hash, confirming order-sensitivity.",
+    input1 = Array("banana", "apple"),
+    output = Array("d4183362b538440bb9a5f82359791c647280e6b657a1812f16f7bcc2b8f141ca")
+  ),
+  new TransformExample(
+    description = "Values from multiple ports are combined in port order, producing the same hash as the equivalent single-port sequence.",
+    input1 = Array("apple"),
+    input2 = Array("banana"),
+    output = Array("5b692305517af54eb5ae12b9ff89eaf89e31f6a6ee208365886a18b81a2fc2f8")
+  ),
+  new TransformExample(
+    description = "The algorithm parameter selects the hash function (MD5).",
+    parameters = Array("algorithm", "MD5"),
+    input1 = Array("input value"),
+    output = Array("cee963a28f70ee97751a85ef732e66dd")
+  ),
+  new TransformExample(
+    description = "The algorithm parameter selects the hash function (SHA-1).",
+    parameters = Array("algorithm", "SHA-1"),
+    input1 = Array("apple"),
+    output = Array("d0be2dc421be4fcd0172e5afceea3970e2f3d940")
+  ),
+  new TransformExample(
+    description = "The algorithm parameter selects the hash function (SHA-384).",
+    parameters = Array("algorithm", "SHA-384"),
+    input1 = Array("apple"),
+    output = Array("3d8786fcb588c93348756c6429717dc6c374a14f7029362281a3b21dc10250ddf0d0578052749822eb08bc0dc1e68b0f")
+  ),
+  new TransformExample(
+    description = "The algorithm parameter selects the hash function (SHA-512).",
+    parameters = Array("algorithm", "SHA-512"),
+    input1 = Array("apple"),
+    output = Array("844d8779103b94c18f4aa4cc0c3b4474058580a991fba85d3ca698a0bc9e52c5940feb7a65a3a290e17e6b23ee943ecc4f73e7490327245b4fe5d5efb590feb2")
+  ),
+  new TransformExample(
+    description = "Empty input produces the hash of an empty message.",
+    input1 = Array(),
+    output = Array("e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")
+  ),
+  new TransformExample(
+    description = "Empty algorithm string causes IllegalArgumentException.",
+    parameters = Array("algorithm", ""),
+    input1 = Array("foo"),
+    throwsException = classOf[IllegalArgumentException]
+  ),
 ))
 case class InputHashTransformer(@Param(value = "The hash algorithm to be used.",
                                       autoCompletionProvider = classOf[HashAlgorithmAutoCompletionProvider], allowOnlyAutoCompletedValues = true)
-                                algorithm: String = "SHA256") extends InlineTransformer {
+                                algorithm: String = "SHA256") extends HashTransformer {
 
   require(algorithm.trim.nonEmpty, "Algorithm must not be empty. Please specify an algorithm, such as 'SHA256'.")
 
   override def apply(values: Seq[Seq[String]]): Seq[String] = {
-    val hashSum = MessageDigest.getInstance(algorithm)
-    for(value <- values; v <- value) {
-      hashSum.update(v.getBytes(StandardCharsets.UTF_8))
-    }
-    // Convert the byte array to a hexadecimal string
-    Seq(hashSum.digest().map("%02x".format(_)).mkString)
+    def updateAll(digest: MessageDigest): Unit =
+      values.flatten.foreach(updateWith(digest, _))
+
+    Seq(withDigest { digest =>
+      updateAll(digest)
+      toHex(digest.digest())
+    })
   }
 }
 
 object InputHashTransformer {
   final val pluginId = "inputHash"
 }
-
-/**
- * Auto-completion for project resources.
- */
-case class HashAlgorithmAutoCompletionProvider() extends PluginParameterAutoCompletionProvider {
-
-  private lazy val algorithms = {
-    Security.getAlgorithms("MessageDigest").asScala.toSeq
-  }
-
-  override def autoComplete(searchQuery: String, dependOnParameterValues: Seq[ParamValue],
-                            workspace: WorkspaceReadTrait)
-                           (implicit context: PluginContext): Iterable[AutoCompletionResult] = {
-
-    val multiSearchWords = extractSearchTerms(searchQuery)
-    algorithms
-      .filter(r => matchesSearchTerm(multiSearchWords, r.toLowerCase))
-      .map(r => AutoCompletionResult(r, None))
-  }
-
-  override def valueToLabel(value: String, dependOnParameterValues: Seq[ParamValue],
-                            workspace: WorkspaceReadTrait)
-                           (implicit context: PluginContext): Option[String] = {
-    None
-  }
-}