Add UTF-8 validation documentation and tests for proto2, proto3, and editions for Java Protobuf

PiperOrigin-RevId: 931326656

Author: protobuf-github-bot

content/reference/java/java-utf8-validation.md

@@ -0,0 +1,85 @@
++++
+title = "UTF-8 Validation in Java"
+weight = 650
+linkTitle = "UTF-8 Validation in Java"
+description = "Describes the UTF-8 string behavior in Java protobuf across all syntaxes, editions, and option/feature settings."
+type = "docs"
++++
+
+.
+
+Proto2 has both string and bytes, and from the beginning it was documented
+that strings must always contain UTF-8 encoded text.
+Enforcement of this requirement varies from language to language (details later
+in this topic), which means that in practice, many string fields contain
+non-UTF-8 data despite what the spec says.
+
+For proto3 it was decided that all parsers must validate UTF-8. This
+decision came almost a year after proto3 had debuted in
+google3, so this change required
+a project to set
+`enforce_utf8=false` on all existing string fields before flipping the default
+to `enforce_utf8=true`. It was a goal of the LSC to burn down all uses of
+`enforce_utf8=false` and ultimately remove the
+option.
+
+Here is a summary of UTF-8 validation behavior for Java Protobuf across syntaxes
+and editions. This behavior is consistent for both standard string fields and
+string-typed extensions:
+
+Syntax / Edition        | `features.utf8_validation`  | `features.(pb.java).utf8_validation`    | `java_string_check_utf8`   | `enforce_utf8`            | Java UTF-8 Validation Behavior
+:---------------------- | :-------------------------- | :-------------------------------------- | :------------------------- | :------------------------ | :-----------------------------
+**proto2**              | *Disallowed*                | *Disallowed*                            | unset or `false` (default) | *Ignored*                 | **No validation** (proto2 default)
+**proto2**              | *Disallowed*                | *Disallowed*                            | `true`                     | *Ignored*                 | **Validation enabled***
+**proto3**              | *Disallowed*                | *Disallowed*                            | *Ignored*                  | unset or `true` (default) | **Validation enabled** (proto3 default)
+**proto3**              | *Disallowed*                | *Disallowed*                            | *Ignored*                  | `false`                   | **No validation**
+**Edition 2023 / 2024** | unset or `VERIFY` (default) | unset, `DEFAULT` (default), or `VERIFY` | *Disallowed*               | *Disallowed*              | **Validation enabled**** (editions default)
+**Edition 2023 / 2024** | `NONE`                      | unset or `DEFAULT` (default)            | *Disallowed*               | *Disallowed*              | **No validation**
+**Edition 2023 / 2024** | `NONE`                      | `VERIFY`                                | *Disallowed*               | *Disallowed*              | **Validation enabled** (validation in Java only)
+
+\* In Java Lite, `proto2` string-typed extensions are **not** validated when
+`java_string_check_utf8=true`.
+
+\** In Java Lite, under edition 2023 and 2024, string-typed extensions are
+**not** validated under any combination of options. This is an oversight that
+should be corrected in a future
+edition.
+
+### Unvalidated Behavior
+
+JavaLite and Java have the same API for string fields, but use different object
+representations:
+
+*   **String-Only:** JavaLite always stores strings as `java.lang.String`. This
+    means that the field cannot preserve invalid UTF-8. If invalid UTF-8 is
+    parsed or assigned via `setFoo(ByteString)`, it immediately undergoes a
+    lossy conversion to replacement characters.
+*   **Dual Mode:** Java "full" supports a dual representation, where the
+    `java.lang.Object` field can be either `java.lang.String` or
+    `java.lang.ByteString`. This means that invalid UTF-8 can be preserved when
+    we parse proto2, and `ByteString getFooBytes()` will return whatever was
+    parsed off the wire.
+
+### Legacy Options
+
+-   **`java_string_check_utf8`** is a **file-level option** applicable only to
+    `proto2`. This option can be set in a proto3 file, but it will have no
+    effect. Java Lite does **not** validate string-typed extensions.
+-   **`enforce_utf8`** is a **field-level option** applicable only to `proto3`.
+    Setting it to `false` is the only way to disable validation in proto3, but
+    it is deprecated and subject to removal (and requires being on an
+    allowlist). This option can be set in a proto2 file, but it will have no
+    effect.
+-   Both legacy options are **disallowed under Editions**, where you must use
+    the features framework instead.
+
+### Editions Features
+
+-   **`features.utf8_validation`** is a file- or field-level feature to control
+    UTF-8 validation for **all** languages.
+-   **`features.(pb.java).utf8_validation`** is a file- or field-level feature
+    to control UTF-8 validation for **Java Only**. The default is to use
+    `features.utf8_validation` for Java. This feature should be phased out in
+    favor of `features.utf8_validation`.
+-   Both features are **disallowed under proto2 and proto3** (as are all
+    editions features).