diff --git a/.agents/skills/fory-code-review/SKILL.md b/.agents/skills/fory-code-review/SKILL.md index 8e5896e974..5085c57b9e 100644 --- a/.agents/skills/fory-code-review/SKILL.md +++ b/.agents/skills/fory-code-review/SKILL.md @@ -11,21 +11,34 @@ Find the highest-value bugs, regressions, and missing verification in Apache For ## Start Here -1. If the target is a GitHub PR, create a new local git worktree for the review before checking out or fetching the PR branch. -2. Do not switch the current branch or reuse the current worktree for PR review unless the user explicitly asks for that. -3. If reviewing against main, run `git fetch apache main` before diffing. -4. Inspect the changed files first and cluster them by subsystem. -5. Load only the references needed for the touched areas: +1. Always perform the code review in a subagent. The main agent coordinates scope, gathers the subagent's findings, sanity-checks them, and reports the final review. +2. Reuse the same review subagent for later review passes on the same feature unless the user explicitly asks to review that feature in a new subagent. +3. Do not reuse a review subagent across different features; start a new subagent for each distinct feature or review topic. +4. The review subagent must be read-only: it must not write code, apply patches, create commits, push branches, fix tests, or update docs. It sends review findings or an explicit no-findings result back to the caller. +5. If the target is a GitHub PR, create a new local git worktree for the review before checking out or fetching the PR branch. +6. Do not switch the current branch or reuse the current worktree for PR review unless the user explicitly asks for that. +7. If reviewing against main, run `git fetch apache main` before diffing. +8. Inspect the changed files first and cluster them by subsystem. +9. Load only the references needed for the touched areas: - `references/review-checklist.md` - `references/lesson-derived-red-flags.md` - `references/validation-command-matrix.md` - matching runtime docs under `../../languages/*.md` when the patch is language-specific +## Subagent Reuse + +- Treat each feature under review as the reuse key. A feature may be a PR, issue, branch, commit range, local diff topic, or clearly named subsystem change. +- When the user asks for another pass on the same feature, send the updated context to the existing review subagent and ask it to continue from its prior review state. +- When the user asks to review a different feature, spawn a fresh review subagent even if the same files, language, or subsystem are involved. +- If the user explicitly requests a new subagent for the same feature, honor that request and do not reuse the prior subagent. +- Keep implementation work, CI fixing, and non-review exploration out of the review subagent. The review subagent reports comments to the caller; the caller decides whether any separate implementation task should happen. + ## Review Workflow 1. Define the review target. - Determine whether the user wants a review of a PR, branch, commit range, or local diff. +- Assign the target to the correct review subagent using the Subagent Reuse rules before inspecting code. - For a GitHub PR, create and use a dedicated local worktree for the review. Keep the current worktree and branch unchanged unless the user explicitly requests otherwise. - In that worktree, fetch the PR head and review there instead of checking out the PR branch in the current workspace. - Prefer `git diff --stat` first, then inspect the full patch only for touched subsystems. @@ -63,6 +76,7 @@ Find the highest-value bugs, regressions, and missing verification in Apache For ## Hard Rules - Do not lead with style nits when there are correctness or verification risks. +- Do not write code, edit files, apply patches, commit, push, or fix tests from the review subagent. - Treat benchmark-shape tricks, payload-specific caches, and methodology changes as real findings. - Treat undocumented public API additions, compatibility shims, and one-line wrapper growth as findings when they increase maintenance surface without clear need. - Treat protocol or performance claims without verification evidence as incomplete. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fa0acf2118..243cffa288 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -999,6 +999,10 @@ jobs: run: | cd dart/packages/fory-test dart test + - name: Run web tests + run: | + cd dart/packages/fory + dart test -p chrome - name: Run code analysis run: | cd dart/packages/fory-test diff --git a/dart/packages/fory-test/test/cross_lang_test/xlang_test_main.dart b/dart/packages/fory-test/test/cross_lang_test/xlang_test_main.dart index 0f66eadbd6..2509f84660 100644 --- a/dart/packages/fory-test/test/cross_lang_test/xlang_test_main.dart +++ b/dart/packages/fory-test/test/cross_lang_test/xlang_test_main.dart @@ -244,8 +244,8 @@ void _verifyDecimalCase() { Uint8List _hashBytes(int low, int high) { final buffer = Buffer(); - buffer.writeInt64(low); - buffer.writeInt64(high); + buffer.writeInt64(Int64(low)); + buffer.writeInt64(Int64(high)); return buffer.toBytes(); } diff --git a/dart/packages/fory/README.md b/dart/packages/fory/README.md index ee4f453c7b..a44cbdd234 100644 --- a/dart/packages/fory/README.md +++ b/dart/packages/fory/README.md @@ -9,12 +9,13 @@ cases. ## Features - Cross-language serialization with the Fory xlang format +- Dart VM/AOT, Flutter, and web platform support - Generated serializers for annotated structs and enums - Compatible mode for schema evolution - Optional reference tracking for shared and circular object graphs - Manual serializers for external types, custom payloads, and unions -- Explicit xlang value wrappers such as `Int32`, `Uint32`, `Float16`, - `Bfloat16`, `Float32`, `LocalDate`, and `Timestamp`, plus `Duration` +- Explicit xlang value class such as `Int32`, `Int64`, `Uint32`, `Uint64`, + `Float16`, `Bfloat16`, `Float32`, `LocalDate`, and `Timestamp`, plus `Duration` support ## Getting Started @@ -193,13 +194,13 @@ final class PersonSerializer extends Serializer { void write(WriteContext context, Person value) { final buffer = context.buffer; buffer.writeUtf8(value.name); - buffer.writeInt64(value.age); + buffer.writeInt64FromInt(value.age); } @override Person read(ReadContext context) { final buffer = context.buffer; - return Person(buffer.readUtf8(), buffer.readInt64()); + return Person(buffer.readUtf8(), buffer.readInt64AsInt()); } } @@ -222,9 +223,9 @@ void main() { Dart has no native fixed-width 8/16/32-bit integer, unsigned 64-bit integer, or single-precision float types. Fory Dart provides thin wrapper types -(`Int8`, `Int16`, `Int32`, `Uint8`, `Uint16`, `Uint32`, `Uint64`, `Float16`, -`Bfloat16`, `Float32`) imported from `package:fory/fory.dart` to represent -these xlang wire types. For 16-bit floating-point arrays, Dart exposes +(`Int8`, `Int16`, `Int32`, `Int64`, `Uint8`, `Uint16`, `Uint32`, `Uint64`, +`Float16`, `Bfloat16`, `Float32`) imported from `package:fory/fory.dart` to +represent these xlang wire types. For 16-bit floating-point arrays, Dart exposes `Float16List` and `Bfloat16List` as contiguous fixed-length buffers. | Fory xlang type | Dart type | @@ -233,7 +234,7 @@ these xlang wire types. For 16-bit floating-point arrays, Dart exposes | int8 | `fory.Int8` (wrapper) | | int16 | `fory.Int16` (wrapper) | | int32 | `fory.Int32` (wrapper) | -| int64 | `int` | +| int64 | `int` or `fory.Int64` | | uint8 | `fory.Uint8` (wrapper) | | uint16 | `fory.Uint16` (wrapper) | | uint32 | `fory.Uint32` (wrapper) | @@ -257,6 +258,9 @@ these xlang wire types. For 16-bit floating-point arrays, Dart exposes | int16_array | `Int16List` | | int32_array | `Int32List` | | int64_array | `Int64List` | +| uint16_array | `Uint16List` | +| uint32_array | `Uint32List` | +| uint64_array | `Uint64List` | | float16_array | `Float16List` | | bfloat16_array | `Bfloat16List` | | float32_array | `Float32List` | @@ -276,8 +280,8 @@ The main exported API includes: annotations - `Int32Type`, `Int64Type`, `Uint32Type`, `Uint64Type` — numeric encoding overrides -- Numeric wrappers: `Int8`, `Int16`, `Int32`, `Uint8`, `Uint16`, `Uint32`, - `Uint64`, `Float16`, `Bfloat16`, `Float32` +- Numeric wrappers: `Int8`, `Int16`, `Int32`, `Int64`, `Uint8`, `Uint16`, + `Uint32`, `Uint64`, `Float16`, `Bfloat16`, `Float32` - Temporal types: `LocalDate`, `Timestamp`, `Duration` ## Cross-Language Notes diff --git a/dart/packages/fory/example/manual_serializer.dart b/dart/packages/fory/example/manual_serializer.dart index fd016882d9..d269c3128e 100644 --- a/dart/packages/fory/example/manual_serializer.dart +++ b/dart/packages/fory/example/manual_serializer.dart @@ -23,7 +23,7 @@ final class Person { Person(this.name, this.age); final String name; - final int age; + final Int64 age; } final class PersonSerializer extends Serializer { @@ -52,7 +52,7 @@ void main() { typeName: 'Person', ); - final person = Person('Ada', 36); + final person = Person('Ada', Int64(36)); final bytes = fory.serialize(person); final roundTrip = fory.deserialize(bytes); print(roundTrip.name); diff --git a/dart/packages/fory/lib/fory.dart b/dart/packages/fory/lib/fory.dart index 86232c609a..5bc276e9be 100644 --- a/dart/packages/fory/lib/fory.dart +++ b/dart/packages/fory/lib/fory.dart @@ -30,7 +30,7 @@ export 'src/annotation/fory_struct.dart'; export 'src/annotation/fory_union.dart'; export 'src/annotation/type_spec.dart'; export 'src/annotation/numeric_types.dart'; -export 'src/buffer.dart' +export 'src/memory/buffer.dart' hide bufferByteData, bufferBytes, @@ -55,6 +55,7 @@ export 'src/types/float16.dart'; export 'src/types/float32.dart'; export 'src/types/int16.dart'; export 'src/types/int32.dart'; +export 'src/types/int64.dart'; export 'src/types/int8.dart'; export 'src/types/local_date.dart'; export 'src/types/timestamp.dart'; diff --git a/dart/packages/fory/lib/src/codegen/fory_generator.dart b/dart/packages/fory/lib/src/codegen/fory_generator.dart index 3697cefd5b..c6e9ad5810 100644 --- a/dart/packages/fory/lib/src/codegen/fory_generator.dart +++ b/dart/packages/fory/lib/src/codegen/fory_generator.dart @@ -33,10 +33,12 @@ class DebugGeneratedFieldTypeSpec { required this.nullable, required this.ref, required this.dynamic, + this.declaredTypeName, this.arguments = const [], }); final String typeLiteral; + final String? declaredTypeName; final int typeId; final bool nullable; final bool ref; @@ -93,7 +95,7 @@ final class ForyGenerator extends Generator { annotatedClasses.map(_analyzeStruct).toList(growable: false); final output = StringBuffer() ..writeln( - '// ignore_for_file: implementation_imports, invalid_use_of_internal_member, no_leading_underscores_for_local_identifiers, unused_element, unused_element_parameter, unnecessary_null_comparison', + '// ignore_for_file: implementation_imports, invalid_use_of_internal_member, no_leading_underscores_for_local_identifiers, unreachable_switch_case, unused_element, unused_element_parameter, unnecessary_null_comparison', ) ..writeln(); @@ -218,6 +220,7 @@ final class ForyGenerator extends Generator { typeSpec is _ListTypeSpecInfo ? typeSpec.element : null; return _GeneratedFieldTypeSpec( typeLiteral: _typeReferenceLiteral(type), + declaredTypeName: _typeReferenceLiteral(type), typeId: _typeIdFor(type), nullable: nullable, ref: ref, @@ -239,6 +242,7 @@ final class ForyGenerator extends Generator { final valueSpec = typeSpec is _MapTypeSpecInfo ? typeSpec.value : null; return _GeneratedFieldTypeSpec( typeLiteral: _typeReferenceLiteral(type), + declaredTypeName: _typeReferenceLiteral(type), typeId: _typeIdFor(type), nullable: nullable, ref: ref, @@ -263,6 +267,7 @@ final class ForyGenerator extends Generator { } return _GeneratedFieldTypeSpec( typeLiteral: _typeReferenceLiteral(type), + declaredTypeName: _typeReferenceLiteral(type), typeId: _typeIdFor(type, integerAnnotation: integerAnnotation), nullable: nullable, ref: ref, @@ -421,12 +426,14 @@ final class ForyGenerator extends Generator { output.writeln(); for (var index = 0; index < structSpec.fields.length; index += 1) { final field = structSpec.fields[index]; + final fieldValue = + _generatedFieldInfoWriteValueExpression(field, 'value.${field.name}'); output ..writeln( 'void _write${structSpec.name}Field$index(WriteContext context, GeneratedStructFieldInfo field, ${structSpec.name} value) {', ) ..writeln( - ' writeGeneratedStructFieldInfoValue(context, field, value.${field.name});', + ' writeGeneratedStructFieldInfoValue(context, field, $fieldValue);', ) ..writeln('}') ..writeln(); @@ -536,8 +543,12 @@ final class ForyGenerator extends Generator { ' ${_directGeneratedWriteStatement(field, 'value.${field.name}')};', ); } else { + final fieldValue = _generatedFieldInfoWriteValueExpression( + field, + 'value.${field.name}', + ); output.writeln( - ' writeGeneratedStructFieldInfoValue(context, fields[$index], value.${field.name});', + ' writeGeneratedStructFieldInfoValue(context, fields[$index], $fieldValue);', ); } final directCursorEndRun = directCursorRunByEnd[index]; @@ -885,6 +896,7 @@ final class ForyGenerator extends Generator { return ''' GeneratedFieldType( type: ${fieldType.typeLiteral}, + declaredTypeName: '${fieldType.typeLiteral}', typeId: ${fieldType.typeId}, nullable: ${fieldType.nullable}, ref: ${fieldType.ref}, @@ -912,6 +924,7 @@ GeneratedFieldType( ) { return _GeneratedFieldTypeSpec( typeLiteral: fieldType.typeLiteral, + declaredTypeName: fieldType.declaredTypeName, typeId: fieldType.typeId, nullable: fieldType.nullable, ref: fieldType.ref, @@ -1035,6 +1048,10 @@ GeneratedFieldType( case TypeIds.int32: case TypeIds.varInt32: return 'switch ($valueExpression) { int typed => typed, Int32 typed => typed.value, _ => throw StateError(\'Expected int or Int32.\') }'; + case TypeIds.int64: + case TypeIds.varInt64: + case TypeIds.taggedInt64: + return 'switch ($valueExpression) { Int64 typed => typed.toInt(), int typed => typed, _ => throw StateError(\'Expected int or Int64.\') }'; case TypeIds.uint8: return 'switch ($valueExpression) { int typed => typed, Uint8 typed => typed.value, _ => throw StateError(\'Expected int or Uint8.\') }'; case TypeIds.uint16: @@ -1045,7 +1062,7 @@ GeneratedFieldType( case TypeIds.uint64: case TypeIds.varUint64: case TypeIds.taggedUint64: - return 'switch ($valueExpression) { int typed => typed, Uint64 typed => typed.value, _ => throw StateError(\'Expected int or Uint64.\') }'; + return 'switch ($valueExpression) { Uint64 typed => typed.toInt(), int typed => typed, _ => throw StateError(\'Expected int or Uint64.\') }'; default: return '$valueExpression as int'; } @@ -1223,10 +1240,19 @@ GeneratedFieldType( case TypeIds.varInt32: return 'buffer.writeVarInt32(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.int64: + if (field.type.isDartCoreInt) { + return 'buffer.writeInt64FromInt($valueExpression)'; + } return 'buffer.writeInt64(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.varInt64: + if (field.type.isDartCoreInt) { + return 'buffer.writeVarInt64FromInt($valueExpression)'; + } return 'buffer.writeVarInt64(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.taggedInt64: + if (field.type.isDartCoreInt) { + return 'buffer.writeTaggedInt64FromInt($valueExpression)'; + } return 'buffer.writeTaggedInt64(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.uint8: return 'buffer.writeUint8(${_directGeneratedScalarExpression(field, valueExpression)})'; @@ -1305,10 +1331,19 @@ GeneratedFieldType( case TypeIds.varInt32: return '$cursorExpression.writeVarInt32(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.int64: + if (field.type.isDartCoreInt) { + return '$cursorExpression.writeInt64FromInt($valueExpression)'; + } return '$cursorExpression.writeInt64(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.varInt64: + if (field.type.isDartCoreInt) { + return '$cursorExpression.writeVarInt64FromInt($valueExpression)'; + } return '$cursorExpression.writeVarInt64(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.taggedInt64: + if (field.type.isDartCoreInt) { + return '$cursorExpression.writeTaggedInt64FromInt($valueExpression)'; + } return '$cursorExpression.writeTaggedInt64(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.uint8: return '$cursorExpression.writeUint8(${_directGeneratedScalarExpression(field, valueExpression)})'; @@ -1319,10 +1354,19 @@ GeneratedFieldType( case TypeIds.varUint32: return '$cursorExpression.writeVarUint32(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.uint64: + if (field.type.isDartCoreInt) { + return '$cursorExpression.writeUint64FromInt($valueExpression)'; + } return '$cursorExpression.writeUint64(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.varUint64: + if (field.type.isDartCoreInt) { + return '$cursorExpression.writeVarUint64FromInt($valueExpression)'; + } return '$cursorExpression.writeVarUint64(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.taggedUint64: + if (field.type.isDartCoreInt) { + return '$cursorExpression.writeTaggedUint64FromInt($valueExpression)'; + } return '$cursorExpression.writeTaggedUint64(${_directGeneratedScalarExpression(field, valueExpression)})'; case TypeIds.float16: return '$cursorExpression.writeFloat16($valueExpression)'; @@ -1370,11 +1414,17 @@ GeneratedFieldType( ? 'buffer.readVarInt32()' : 'Int32(buffer.readVarInt32())'; case TypeIds.int64: - return 'buffer.readInt64()'; + return field.type.isDartCoreInt + ? 'buffer.readInt64AsInt()' + : 'buffer.readInt64()'; case TypeIds.varInt64: - return 'buffer.readVarInt64()'; + return field.type.isDartCoreInt + ? 'buffer.readVarInt64AsInt()' + : 'buffer.readVarInt64()'; case TypeIds.taggedInt64: - return 'buffer.readTaggedInt64()'; + return field.type.isDartCoreInt + ? 'buffer.readTaggedInt64AsInt()' + : 'buffer.readTaggedInt64()'; case TypeIds.uint8: return field.type.isDartCoreInt ? 'buffer.readUint8()' @@ -1393,16 +1443,16 @@ GeneratedFieldType( : 'Uint32(buffer.readVarUint32())'; case TypeIds.uint64: return field.type.isDartCoreInt - ? 'buffer.readUint64()' - : 'Uint64(buffer.readUint64())'; + ? 'buffer.readUint64().toInt()' + : 'buffer.readUint64()'; case TypeIds.varUint64: return field.type.isDartCoreInt - ? 'buffer.readVarUint64()' - : 'Uint64(buffer.readVarUint64())'; + ? 'buffer.readVarUint64().toInt()' + : 'buffer.readVarUint64()'; case TypeIds.taggedUint64: return field.type.isDartCoreInt - ? 'buffer.readTaggedUint64()' - : 'Uint64(buffer.readTaggedUint64())'; + ? 'buffer.readTaggedUint64().toInt()' + : 'buffer.readTaggedUint64()'; case TypeIds.float16: return 'buffer.readFloat16()'; case TypeIds.bfloat16: @@ -1436,7 +1486,7 @@ GeneratedFieldType( case TypeIds.int32Array: return 'readGeneratedTypedArrayValue(context, 4, (bytes) => bytes.buffer.asInt32List(bytes.offsetInBytes, bytes.lengthInBytes ~/ 4))'; case TypeIds.int64Array: - return 'readGeneratedTypedArrayValue(context, 8, (bytes) => bytes.buffer.asInt64List(bytes.offsetInBytes, bytes.lengthInBytes ~/ 8))'; + return 'readGeneratedTypedArrayValue(context, 8, (bytes) => Int64List.view(bytes.buffer, bytes.offsetInBytes, bytes.lengthInBytes ~/ 8))'; case TypeIds.uint8Array: return 'readGeneratedBinaryValue(context)'; case TypeIds.uint16Array: @@ -1448,7 +1498,7 @@ GeneratedFieldType( case TypeIds.uint32Array: return 'readGeneratedTypedArrayValue(context, 4, (bytes) => bytes.buffer.asUint32List(bytes.offsetInBytes, bytes.lengthInBytes ~/ 4))'; case TypeIds.uint64Array: - return 'readGeneratedTypedArrayValue(context, 8, (bytes) => bytes.buffer.asUint64List(bytes.offsetInBytes, bytes.lengthInBytes ~/ 8))'; + return 'readGeneratedTypedArrayValue(context, 8, (bytes) => Uint64List.view(bytes.buffer, bytes.offsetInBytes, bytes.lengthInBytes ~/ 8))'; case TypeIds.float32Array: return 'readGeneratedTypedArrayValue(context, 4, (bytes) => bytes.buffer.asFloat32List(bytes.offsetInBytes, bytes.lengthInBytes ~/ 4))'; case TypeIds.float64Array: @@ -1482,11 +1532,17 @@ GeneratedFieldType( ? '$cursorExpression.readVarInt32()' : 'Int32($cursorExpression.readVarInt32())'; case TypeIds.int64: - return '$cursorExpression.readInt64()'; + return field.type.isDartCoreInt + ? '$cursorExpression.readInt64AsInt()' + : '$cursorExpression.readInt64()'; case TypeIds.varInt64: - return '$cursorExpression.readVarInt64()'; + return field.type.isDartCoreInt + ? '$cursorExpression.readVarInt64AsInt()' + : '$cursorExpression.readVarInt64()'; case TypeIds.taggedInt64: - return '$cursorExpression.readTaggedInt64()'; + return field.type.isDartCoreInt + ? '$cursorExpression.readTaggedInt64AsInt()' + : '$cursorExpression.readTaggedInt64()'; case TypeIds.uint8: return field.type.isDartCoreInt ? '$cursorExpression.readUint8()' @@ -1505,16 +1561,16 @@ GeneratedFieldType( : 'Uint32($cursorExpression.readVarUint32())'; case TypeIds.uint64: return field.type.isDartCoreInt - ? '$cursorExpression.readUint64()' - : 'Uint64($cursorExpression.readUint64())'; + ? '$cursorExpression.readUint64AsInt()' + : '$cursorExpression.readUint64()'; case TypeIds.varUint64: return field.type.isDartCoreInt - ? '$cursorExpression.readVarUint64()' - : 'Uint64($cursorExpression.readVarUint64())'; + ? '$cursorExpression.readVarUint64AsInt()' + : '$cursorExpression.readVarUint64()'; case TypeIds.taggedUint64: return field.type.isDartCoreInt - ? '$cursorExpression.readTaggedUint64()' - : 'Uint64($cursorExpression.readTaggedUint64())'; + ? '$cursorExpression.readTaggedUint64AsInt()' + : '$cursorExpression.readTaggedUint64()'; case TypeIds.float16: return '$cursorExpression.readFloat16()'; case TypeIds.bfloat16: @@ -1568,15 +1624,33 @@ GeneratedFieldType( _GeneratedFieldSpec field, String valueExpression, ) { - if (field.type.isDartCoreInt || - field.type.isDartCoreDouble || + if (field.type.isDartCoreInt) { + switch (field.fieldType.typeId) { + case TypeIds.int64: + case TypeIds.varInt64: + case TypeIds.taggedInt64: + return 'Int64($valueExpression)'; + case TypeIds.uint64: + case TypeIds.varUint64: + case TypeIds.taggedUint64: + return 'Uint64($valueExpression)'; + default: + return valueExpression; + } + } + if (field.type.isDartCoreDouble || field.type.isDartCoreBool || field.type.isDartCoreString) { return valueExpression; } switch (field.fieldType.typeId) { + case TypeIds.int64: + case TypeIds.varInt64: + case TypeIds.taggedInt64: + case TypeIds.uint64: + case TypeIds.varUint64: + case TypeIds.taggedUint64: case TypeIds.float16: - return valueExpression; case TypeIds.bfloat16: return valueExpression; default: @@ -1584,6 +1658,28 @@ GeneratedFieldType( } } + String _generatedFieldInfoWriteValueExpression( + _GeneratedFieldSpec field, + String valueExpression, + ) { + if (!_withoutNullability(field.type).isDartCoreInt) { + return valueExpression; + } + final wrapper = switch (field.fieldType.typeId) { + TypeIds.int8 => 'Int8', + TypeIds.int16 => 'Int16', + TypeIds.int32 || TypeIds.varInt32 => 'Int32', + _ => null, + }; + if (wrapper == null) { + return valueExpression; + } + if (_isNullable(field.type)) { + return '$valueExpression == null ? null : $wrapper($valueExpression!)'; + } + return '$wrapper($valueExpression)'; + } + String _nullExpression( DartType type, { required String errorTarget, @@ -1607,6 +1703,7 @@ GeneratedFieldType( } return _GeneratedFieldTypeSpec( typeLiteral: fieldType.typeLiteral, + declaredTypeName: fieldType.declaredTypeName, typeId: fieldType.typeId, nullable: false, ref: fieldType.ref, @@ -2098,9 +2195,11 @@ GeneratedFieldType( case 'Uint16': return TypeIds.uint16; case 'Uint32': - return TypeIds.uint32; + return TypeIds.varUint32; case 'Uint64': - return TypeIds.uint64; + return TypeIds.varUint64; + case 'Int64': + return TypeIds.varInt64; case 'Float16': return TypeIds.float16; case 'Bfloat16': @@ -2381,6 +2480,7 @@ final class _DirectGeneratedWriteReservationRun { final class _GeneratedFieldTypeSpec { final String typeLiteral; + final String? declaredTypeName; final int typeId; final bool nullable; final bool ref; @@ -2389,6 +2489,7 @@ final class _GeneratedFieldTypeSpec { const _GeneratedFieldTypeSpec({ required this.typeLiteral, + this.declaredTypeName, required this.typeId, required this.nullable, required this.ref, diff --git a/dart/packages/fory/lib/src/codegen/generated_cursor.dart b/dart/packages/fory/lib/src/codegen/generated_cursor.dart new file mode 100644 index 0000000000..44816f1189 --- /dev/null +++ b/dart/packages/fory/lib/src/codegen/generated_cursor.dart @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +export 'generated_cursor_native.dart' + if (dart.library.js_interop) 'generated_cursor_web.dart'; diff --git a/dart/packages/fory/lib/src/codegen/generated_cursor_mixin.dart b/dart/packages/fory/lib/src/codegen/generated_cursor_mixin.dart new file mode 100644 index 0000000000..d8aa225336 --- /dev/null +++ b/dart/packages/fory/lib/src/codegen/generated_cursor_mixin.dart @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// ignore_for_file: use_string_in_part_of_directives + +part of fory.src.codegen.generated_cursor; + +mixin _GeneratedWriteCursorMixin { + late final Buffer _buffer; + late final Uint8List _bytes; + late final ByteData _view; + int _offset = 0; + + void _initWriteCursor(Buffer buffer, int maxBytes) { + final start = bufferReserveBytes(buffer, maxBytes); + _buffer = buffer; + _bytes = bufferBytes(buffer); + _view = bufferByteData(buffer); + _offset = start; + } + + void finish() { + bufferSetWriterIndex(_buffer, _offset); + } + + @pragma('vm:prefer-inline') + void writeBool(bool value) { + _bytes[_offset] = value ? 1 : 0; + _offset += 1; + } + + @pragma('vm:prefer-inline') + void writeByte(int value) { + _view.setInt8(_offset, value); + _offset += 1; + } + + @pragma('vm:prefer-inline') + void writeUint8(int value) { + _view.setUint8(_offset, value); + _offset += 1; + } + + @pragma('vm:prefer-inline') + void writeInt16(int value) { + _view.setInt16(_offset, value, Endian.little); + _offset += 2; + } + + @pragma('vm:prefer-inline') + void writeUint16(int value) { + _view.setUint16(_offset, value, Endian.little); + _offset += 2; + } + + @pragma('vm:prefer-inline') + void writeInt32(int value) { + _view.setInt32(_offset, value, Endian.little); + _offset += 4; + } + + @pragma('vm:prefer-inline') + void writeUint32(int value) { + _view.setUint32(_offset, value, Endian.little); + _offset += 4; + } + + @pragma('vm:prefer-inline') + void writeFloat16(Float16 value) { + writeUint16(value.toBits()); + } + + @pragma('vm:prefer-inline') + void writeBfloat16(Bfloat16 value) { + writeUint16(value.toBits()); + } + + @pragma('vm:prefer-inline') + void writeFloat32(double value) { + _view.setFloat32(_offset, value, Endian.little); + _offset += 4; + } + + @pragma('vm:prefer-inline') + void writeFloat64(double value) { + _view.setFloat64(_offset, value, Endian.little); + _offset += 8; + } + + @pragma('vm:prefer-inline') + void writeVarUint32(int value) { + var remaining = value; + while (remaining >= 0x80) { + _bytes[_offset] = (remaining & 0x7f) | 0x80; + _offset += 1; + remaining >>>= 7; + } + _bytes[_offset] = remaining; + _offset += 1; + } + + @pragma('vm:prefer-inline') + void writeVarInt32(int value) { + writeVarUint32(((value << 1) ^ (value >> 31)).toUnsigned(32)); + } + + void writeVarUint64(Uint64 value); +} + +mixin _GeneratedReadCursorMixin { + late final Buffer _buffer; + late final ByteData _view; + int _offset = 0; + + void _initReadCursor(Buffer buffer) { + _buffer = buffer; + _view = bufferByteData(buffer); + _offset = bufferReaderIndex(buffer); + } + + void finish() { + bufferSetReaderIndex(_buffer, _offset); + } + + @pragma('vm:prefer-inline') + bool readBool() => readUint8() != 0; + + @pragma('vm:prefer-inline') + int readByte() { + final value = _view.getInt8(_offset); + _offset += 1; + return value; + } + + @pragma('vm:prefer-inline') + int readUint8() { + final value = _view.getUint8(_offset); + _offset += 1; + return value; + } + + @pragma('vm:prefer-inline') + int readInt16() { + final value = _view.getInt16(_offset, Endian.little); + _offset += 2; + return value; + } + + @pragma('vm:prefer-inline') + int readUint16() { + final value = _view.getUint16(_offset, Endian.little); + _offset += 2; + return value; + } + + @pragma('vm:prefer-inline') + int readInt32() { + final value = _view.getInt32(_offset, Endian.little); + _offset += 4; + return value; + } + + @pragma('vm:prefer-inline') + int readUint32() { + final value = _view.getUint32(_offset, Endian.little); + _offset += 4; + return value; + } + + @pragma('vm:prefer-inline') + Float16 readFloat16() => Float16.fromBits(readUint16()); + + @pragma('vm:prefer-inline') + Bfloat16 readBfloat16() => Bfloat16.fromBits(readUint16()); + + @pragma('vm:prefer-inline') + double readFloat32() { + final value = _view.getFloat32(_offset, Endian.little); + _offset += 4; + return value; + } + + @pragma('vm:prefer-inline') + double readFloat64() { + final value = _view.getFloat64(_offset, Endian.little); + _offset += 8; + return value; + } + + @pragma('vm:prefer-inline') + int readVarUint32() { + var shift = 0; + var result = 0; + while (true) { + final byte = readUint8(); + result |= (byte & 0x7f) << shift; + if ((byte & 0x80) == 0) { + return result; + } + shift += 7; + } + } + + @pragma('vm:prefer-inline') + int readVarInt32() { + final value = readVarUint32(); + return ((value >>> 1) ^ -(value & 1)).toSigned(32); + } + + Uint64 readVarUint64(); +} diff --git a/dart/packages/fory/lib/src/codegen/generated_cursor_native.dart b/dart/packages/fory/lib/src/codegen/generated_cursor_native.dart new file mode 100644 index 0000000000..482d31d15c --- /dev/null +++ b/dart/packages/fory/lib/src/codegen/generated_cursor_native.dart @@ -0,0 +1,305 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// ignore_for_file: unnecessary_library_name + +library fory.src.codegen.generated_cursor; + +import 'dart:typed_data'; + +import 'package:meta/meta.dart'; + +import 'package:fory/src/memory/buffer.dart'; +import 'package:fory/src/types/bfloat16.dart'; +import 'package:fory/src/types/float16.dart'; +import 'package:fory/src/types/int64.dart'; +import 'package:fory/src/types/uint64.dart'; + +part 'generated_cursor_mixin.dart'; + +@internal +final class GeneratedWriteCursor with _GeneratedWriteCursorMixin { + GeneratedWriteCursor._(); + + factory GeneratedWriteCursor.reserve(Buffer buffer, int maxBytes) { + return GeneratedWriteCursor._().._initWriteCursor(buffer, maxBytes); + } + + @pragma('vm:prefer-inline') + void writeInt64(Int64 value) { + _view.setInt64(_offset, value.toInt(), Endian.little); + _offset += 8; + } + + @pragma('vm:prefer-inline') + void writeInt64FromInt(int value) { + _view.setInt64(_offset, value, Endian.little); + _offset += 8; + } + + @pragma('vm:prefer-inline') + void writeUint64(Uint64 value) { + _view.setInt64(_offset, value.value, Endian.little); + _offset += 8; + } + + @pragma('vm:prefer-inline') + void writeUint64FromInt(int value) { + _view.setInt64(_offset, value.toSigned(64), Endian.little); + _offset += 8; + } + + @pragma('vm:prefer-inline') + @override + void writeVarUint64(Uint64 value) { + _writeVarUint64Int(value.value); + } + + @pragma('vm:prefer-inline') + void writeVarUint64FromInt(int value) { + _writeVarUint64Int(value.toSigned(64)); + } + + @pragma('vm:prefer-inline') + void writeVarInt64(Int64 value) { + _writeVarUint64Int((value << 1) ^ (value >> 63)); + } + + @pragma('vm:prefer-inline') + void writeVarInt64FromInt(int value) { + _writeVarUint64Int((value << 1) ^ (value >> 63)); + } + + @pragma('vm:prefer-inline') + void writeTaggedInt64(Int64 value) { + if (value >= -0x40000000 && value <= 0x3fffffff) { + writeInt32((value.toInt() << 1).toSigned(32)); + return; + } + writeUint8(0x01); + writeInt64(value); + } + + @pragma('vm:prefer-inline') + void writeTaggedInt64FromInt(int value) { + if (value >= -0x40000000 && value <= 0x3fffffff) { + writeInt32((value << 1).toSigned(32)); + return; + } + writeUint8(0x01); + writeInt64FromInt(value); + } + + @pragma('vm:prefer-inline') + void writeTaggedUint64(Uint64 value) { + if (value >= 0 && value <= 0x7fffffff) { + writeInt32(value.toInt() << 1); + return; + } + writeUint8(0x01); + writeUint64(value); + } + + @pragma('vm:prefer-inline') + void writeTaggedUint64FromInt(int value) { + if (value >= 0 && value <= 0x7fffffff) { + writeInt32(value << 1); + return; + } + writeUint8(0x01); + writeUint64FromInt(value); + } + + @pragma('vm:prefer-inline') + void _writeVarUint64Int(int value) { + var remaining = value; + for (var index = 0; index < 8; index += 1) { + final chunk = remaining & 0x7f; + remaining >>>= 7; + if (remaining == 0) { + _bytes[_offset] = chunk; + _offset += 1; + return; + } + _bytes[_offset] = chunk | 0x80; + _offset += 1; + } + _bytes[_offset] = remaining & 0xff; + _offset += 1; + } +} + +@internal +final class GeneratedReadCursor with _GeneratedReadCursorMixin { + GeneratedReadCursor._(); + + factory GeneratedReadCursor.start(Buffer buffer) { + return GeneratedReadCursor._().._initReadCursor(buffer); + } + + @pragma('vm:prefer-inline') + Int64 readInt64() { + final value = Int64(_view.getInt64(_offset, Endian.little)); + _offset += 8; + return value; + } + + @pragma('vm:prefer-inline') + int readInt64AsInt() { + final value = _view.getInt64(_offset, Endian.little); + _offset += 8; + return value; + } + + @pragma('vm:prefer-inline') + Uint64 readUint64() { + final value = Uint64(_view.getInt64(_offset, Endian.little)); + _offset += 8; + return value; + } + + @pragma('vm:prefer-inline') + int readUint64AsInt() { + final value = _view.getUint64(_offset, Endian.little); + if ((_view.getUint32(_offset + 4, Endian.little) & 0x80000000) != 0) { + throw StateError( + 'Uint64 value $value is not representable as a native int.', + ); + } + _offset += 8; + return value; + } + + @pragma('vm:prefer-inline') + @override + Uint64 readVarUint64() { + var shift = 0; + var result = Uint64(0); + while (shift < 56) { + final byte = readUint8(); + result = result | (Uint64(byte & 0x7f) << shift); + if ((byte & 0x80) == 0) { + return result; + } + shift += 7; + } + return result | (Uint64(readUint8()) << 56); + } + + @pragma('vm:prefer-inline') + int readVarUint64AsInt() { + var shift = 0; + var result = 0; + while (shift < 56) { + final byte = readUint8(); + result |= (byte & 0x7f) << shift; + if ((byte & 0x80) == 0) { + return result; + } + shift += 7; + } + final byte = readUint8(); + final value = result | (byte << 56); + if ((byte & 0x80) != 0) { + throw StateError( + 'Uint64 value $value is not representable as a native int.', + ); + } + return value; + } + + @pragma('vm:prefer-inline') + Int64 readVarInt64() { + final encoded = readVarUint64(); + return Int64((encoded >>> 1) ^ -(encoded & 1)); + } + + @pragma('vm:prefer-inline') + int readVarInt64AsInt() { + var shift = 0; + var encoded = 0; + while (shift < 56) { + final byte = readUint8(); + encoded |= (byte & 0x7f) << shift; + if ((byte & 0x80) == 0) { + return (encoded >>> 1) ^ -(encoded & 1); + } + shift += 7; + } + encoded |= readUint8() << 56; + return (encoded >>> 1) ^ -(encoded & 1); + } + + @pragma('vm:prefer-inline') + Int64 readTaggedInt64() { + final readIndex = _offset; + final first = _view.getInt32(readIndex, Endian.little); + if ((first & 1) == 0) { + _offset = readIndex + 4; + return Int64(first.toSigned(32) ~/ 2); + } + final value = Int64(_view.getInt64(readIndex + 1, Endian.little)); + _offset = readIndex + 9; + return value; + } + + @pragma('vm:prefer-inline') + int readTaggedInt64AsInt() { + final readIndex = _offset; + final first = _view.getInt32(readIndex, Endian.little); + if ((first & 1) == 0) { + _offset = readIndex + 4; + return first.toSigned(32) ~/ 2; + } + final value = _view.getInt64(readIndex + 1, Endian.little); + _offset = readIndex + 9; + return value; + } + + @pragma('vm:prefer-inline') + Uint64 readTaggedUint64() { + final readIndex = _offset; + final first = _view.getUint32(readIndex, Endian.little); + if ((first & 1) == 0) { + _offset = readIndex + 4; + return Uint64(first >>> 1); + } + final value = Uint64(_view.getInt64(readIndex + 1, Endian.little)); + _offset = readIndex + 9; + return value; + } + + @pragma('vm:prefer-inline') + int readTaggedUint64AsInt() { + final readIndex = _offset; + final first = _view.getUint32(readIndex, Endian.little); + if ((first & 1) == 0) { + _offset = readIndex + 4; + return first >>> 1; + } + final value = _view.getUint64(readIndex + 1, Endian.little); + if ((_view.getUint32(readIndex + 5, Endian.little) & 0x80000000) != 0) { + throw StateError( + 'Uint64 value $value is not representable as a native int.', + ); + } + _offset = readIndex + 9; + return value; + } +} diff --git a/dart/packages/fory/lib/src/codegen/generated_cursor_web.dart b/dart/packages/fory/lib/src/codegen/generated_cursor_web.dart new file mode 100644 index 0000000000..d58a518769 --- /dev/null +++ b/dart/packages/fory/lib/src/codegen/generated_cursor_web.dart @@ -0,0 +1,337 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// ignore_for_file: unnecessary_library_name + +library fory.src.codegen.generated_cursor; + +import 'dart:typed_data'; + +import 'package:meta/meta.dart'; + +import 'package:fory/src/memory/buffer.dart'; +import 'package:fory/src/types/bfloat16.dart'; +import 'package:fory/src/types/float16.dart'; +import 'package:fory/src/types/int64.dart'; +import 'package:fory/src/types/uint64.dart'; + +part 'generated_cursor_mixin.dart'; + +const int _jsSafeIntMax = 9007199254740991; +const int _jsSafeIntMin = -9007199254740991; + +@internal +final class GeneratedWriteCursor with _GeneratedWriteCursorMixin { + GeneratedWriteCursor._(); + + factory GeneratedWriteCursor.reserve(Buffer buffer, int maxBytes) { + return GeneratedWriteCursor._().._initWriteCursor(buffer, maxBytes); + } + + @pragma('vm:prefer-inline') + void writeInt64(Int64 value) { + _writeInt64Words(_offset, value); + _offset += 8; + } + + @pragma('vm:prefer-inline') + void writeInt64FromInt(int value) { + _writeInt64Words(_offset, _int64FromInt(value)); + _offset += 8; + } + + @pragma('vm:prefer-inline') + void writeUint64(Uint64 value) { + _writeUint64Words(_offset, value); + _offset += 8; + } + + @pragma('vm:prefer-inline') + void writeUint64FromInt(int value) { + _checkUint64IntRange(value); + _writeUint64Words(_offset, Uint64(value)); + _offset += 8; + } + + @pragma('vm:prefer-inline') + @override + void writeVarUint64(Uint64 value) { + var remaining = value; + for (var shift = 0; shift < 56 && remaining > 0x7f; shift += 7) { + _bytes[_offset] = (remaining.low32 & 0x7f) | 0x80; + _offset += 1; + remaining = remaining >> 7; + } + _bytes[_offset] = remaining.toInt(); + _offset += 1; + } + + @pragma('vm:prefer-inline') + void writeVarUint64FromInt(int value) { + _checkUint64IntRange(value); + writeVarUint64(Uint64(value)); + } + + @pragma('vm:prefer-inline') + void writeVarInt64(Int64 value) { + writeVarUint64(_zigZagEncodeInt64(value)); + } + + @pragma('vm:prefer-inline') + void writeVarInt64FromInt(int value) { + writeVarInt64(_int64FromInt(value)); + } + + @pragma('vm:prefer-inline') + void writeTaggedInt64(Int64 value) { + if (value >= -0x40000000 && value <= 0x3fffffff) { + writeInt32((value.toInt() << 1).toSigned(32)); + return; + } + writeUint8(0x01); + writeInt64(value); + } + + @pragma('vm:prefer-inline') + void writeTaggedInt64FromInt(int value) { + if (value >= -0x40000000 && value <= 0x3fffffff) { + writeInt32((value << 1).toSigned(32)); + return; + } + _checkInt64IntRange(value); + writeUint8(0x01); + writeInt64FromInt(value); + } + + @pragma('vm:prefer-inline') + void writeTaggedUint64(Uint64 value) { + if (value >= 0 && value <= 0x7fffffff) { + writeInt32(value.toInt() << 1); + return; + } + writeUint8(0x01); + writeUint64(value); + } + + @pragma('vm:prefer-inline') + void writeTaggedUint64FromInt(int value) { + if (value >= 0 && value <= 0x7fffffff) { + writeInt32(value << 1); + return; + } + _checkUint64IntRange(value); + writeUint8(0x01); + writeUint64FromInt(value); + } + + @pragma('vm:prefer-inline') + void _writeInt64Words(int offset, Int64 value) { + _view.setUint32(offset, value.low32, Endian.little); + _view.setUint32(offset + 4, value.high32Unsigned, Endian.little); + } + + @pragma('vm:prefer-inline') + void _writeUint64Words(int offset, Uint64 value) { + _view.setUint32(offset, value.low32, Endian.little); + _view.setUint32(offset + 4, value.high32Unsigned, Endian.little); + } +} + +@internal +final class GeneratedReadCursor with _GeneratedReadCursorMixin { + GeneratedReadCursor._(); + + factory GeneratedReadCursor.start(Buffer buffer) { + return GeneratedReadCursor._().._initReadCursor(buffer); + } + + @pragma('vm:prefer-inline') + Int64 readInt64() { + final value = _readInt64Words(_offset); + _offset += 8; + return value; + } + + @pragma('vm:prefer-inline') + int readInt64AsInt() { + final value = _int64ToInt(_readInt64Words(_offset)); + _offset += 8; + return value; + } + + @pragma('vm:prefer-inline') + Uint64 readUint64() { + final value = _readUint64Words(_offset); + _offset += 8; + return value; + } + + @pragma('vm:prefer-inline') + int readUint64AsInt() { + final value = _readUint64Words(_offset).toInt(); + _offset += 8; + return value; + } + + @pragma('vm:prefer-inline') + @override + Uint64 readVarUint64() { + var shift = 0; + var result = Uint64(0); + while (shift < 56) { + final byte = readUint8(); + result = result | (Uint64(byte & 0x7f) << shift); + if ((byte & 0x80) == 0) { + return result; + } + shift += 7; + } + return result | (Uint64(readUint8()) << 56); + } + + @pragma('vm:prefer-inline') + int readVarUint64AsInt() { + return readVarUint64().toInt(); + } + + @pragma('vm:prefer-inline') + Int64 readVarInt64() { + return _zigZagDecodeInt64(readVarUint64()); + } + + @pragma('vm:prefer-inline') + int readVarInt64AsInt() { + return _int64ToInt(readVarInt64()); + } + + @pragma('vm:prefer-inline') + Int64 readTaggedInt64() { + final readIndex = _offset; + final first = _view.getInt32(readIndex, Endian.little); + if ((first & 1) == 0) { + _offset = readIndex + 4; + return Int64(first.toSigned(32) ~/ 2); + } + final value = _readInt64Words(readIndex + 1); + _offset = readIndex + 9; + return value; + } + + @pragma('vm:prefer-inline') + int readTaggedInt64AsInt() { + final readIndex = _offset; + final first = _view.getInt32(readIndex, Endian.little); + if ((first & 1) == 0) { + _offset = readIndex + 4; + return first.toSigned(32) ~/ 2; + } + final value = _int64ToInt(_readInt64Words(readIndex + 1)); + _offset = readIndex + 9; + return value; + } + + @pragma('vm:prefer-inline') + Uint64 readTaggedUint64() { + final readIndex = _offset; + final first = _view.getUint32(readIndex, Endian.little); + if ((first & 1) == 0) { + _offset = readIndex + 4; + return Uint64(first >>> 1); + } + final value = _readUint64Words(readIndex + 1); + _offset = readIndex + 9; + return value; + } + + @pragma('vm:prefer-inline') + int readTaggedUint64AsInt() { + final readIndex = _offset; + final first = _view.getUint32(readIndex, Endian.little); + if ((first & 1) == 0) { + _offset = readIndex + 4; + return first >>> 1; + } + final value = _readUint64Words(readIndex + 1).toInt(); + _offset = readIndex + 9; + return value; + } + + @pragma('vm:prefer-inline') + Int64 _readInt64Words(int offset) { + return Int64.fromWords( + _view.getUint32(offset, Endian.little), + _view.getInt32(offset + 4, Endian.little), + ); + } + + @pragma('vm:prefer-inline') + Uint64 _readUint64Words(int offset) { + return Uint64.fromWords( + _view.getUint32(offset, Endian.little), + _view.getUint32(offset + 4, Endian.little), + ); + } +} + +@pragma('vm:prefer-inline') +void _checkInt64IntRange(int value) { + if (value < _jsSafeIntMin || value > _jsSafeIntMax) { + throw StateError( + 'Dart int value $value is outside the JS-safe signed int64 range ' + '[$_jsSafeIntMin, $_jsSafeIntMax]. Use Int64 for full 64-bit values ' + 'on web.', + ); + } +} + +@pragma('vm:prefer-inline') +Int64 _int64FromInt(int value) { + _checkInt64IntRange(value); + return Int64(value); +} + +@pragma('vm:prefer-inline') +void _checkUint64IntRange(int value) { + if (value < 0 || value > _jsSafeIntMax) { + throw StateError( + 'Dart int value $value is outside the JS-safe unsigned uint64 int ' + 'field range [0, $_jsSafeIntMax]. Use Uint64 for full unsigned ' + '64-bit values on web.', + ); + } +} + +@pragma('vm:prefer-inline') +int _int64ToInt(Int64 value) => value.toInt(); + +@pragma('vm:prefer-inline') +Uint64 _zigZagEncodeInt64(Int64 value) { + final encoded = (value << 1) ^ (value >> 63); + return Uint64.fromWords(encoded.low32, encoded.high32Unsigned); +} + +@pragma('vm:prefer-inline') +Int64 _zigZagDecodeInt64(Uint64 encoded) { + final magnitude = encoded >> 1; + final decoded = Int64.fromWords(magnitude.low32, magnitude.high32Unsigned); + if ((encoded.low32 & 1) == 0) { + return decoded; + } + return -(decoded + 1); +} diff --git a/dart/packages/fory/lib/src/codegen/generated_support.dart b/dart/packages/fory/lib/src/codegen/generated_support.dart index 7fbc1de1bb..645a0b86f2 100644 --- a/dart/packages/fory/lib/src/codegen/generated_support.dart +++ b/dart/packages/fory/lib/src/codegen/generated_support.dart @@ -22,7 +22,8 @@ import 'dart:typed_data'; import 'package:meta/meta.dart'; import 'package:fory/fory.dart'; -import 'package:fory/src/buffer.dart'; +export 'package:fory/src/codegen/generated_cursor.dart'; + import 'package:fory/src/codegen/generated_registry.dart'; import 'package:fory/src/meta/field_info.dart' as meta; import 'package:fory/src/meta/field_type.dart' as meta_types; @@ -37,368 +38,10 @@ import 'package:fory/src/serializer/struct_slots.dart'; import 'package:fory/src/serializer/time_serializers.dart'; import 'package:fory/src/serializer/typed_array_serializers.dart'; -final BigInt _generatedCursorMask64Big = (BigInt.one << 64) - BigInt.one; -final BigInt _generatedCursorSevenBitMaskBig = BigInt.from(0x7f); -final BigInt _generatedCursorByteMaskBig = BigInt.from(0xff); -const bool _generatedCursorUseBigIntVarint64 = - bool.fromEnvironment('dart.library.js_interop') || - bool.fromEnvironment('dart.library.js_util'); - -@internal -final class GeneratedWriteCursor { - final Buffer _buffer; - final Uint8List _bytes; - final ByteData _view; - int _offset; - - GeneratedWriteCursor._( - this._buffer, - this._bytes, - this._view, - this._offset, - ); - - factory GeneratedWriteCursor.reserve(Buffer buffer, int maxBytes) { - final start = bufferReserveBytes(buffer, maxBytes); - return GeneratedWriteCursor._( - buffer, - bufferBytes(buffer), - bufferByteData(buffer), - start, - ); - } - - void finish() { - bufferSetWriterIndex(_buffer, _offset); - } - - void writeBool(bool value) { - _bytes[_offset] = value ? 1 : 0; - _offset += 1; - } - - void writeByte(int value) { - _view.setInt8(_offset, value); - _offset += 1; - } - - void writeUint8(int value) { - _view.setUint8(_offset, value); - _offset += 1; - } - - void writeInt16(int value) { - _view.setInt16(_offset, value, Endian.little); - _offset += 2; - } - - void writeUint16(int value) { - _view.setUint16(_offset, value, Endian.little); - _offset += 2; - } - - void writeInt32(int value) { - _view.setInt32(_offset, value, Endian.little); - _offset += 4; - } - - void writeUint32(int value) { - _view.setUint32(_offset, value, Endian.little); - _offset += 4; - } - - void writeInt64(int value) { - _view.setInt64(_offset, value, Endian.little); - _offset += 8; - } - - void writeUint64(int value) { - _view.setUint64(_offset, value, Endian.little); - _offset += 8; - } - - void writeFloat16(Float16 value) { - writeUint16(value.toBits()); - } - - void writeBfloat16(Bfloat16 value) { - writeUint16(value.toBits()); - } - - void writeFloat32(double value) { - _view.setFloat32(_offset, value, Endian.little); - _offset += 4; - } - - void writeFloat64(double value) { - _view.setFloat64(_offset, value, Endian.little); - _offset += 8; - } - - void writeVarUint32(int value) { - var remaining = value; - while (remaining >= 0x80) { - _bytes[_offset] = (remaining & 0x7f) | 0x80; - _offset += 1; - remaining >>>= 7; - } - _bytes[_offset] = remaining; - _offset += 1; - } - - void writeVarInt32(int value) { - writeVarUint32((value << 1) ^ (value >> 31)); - } - - void writeVarUint64(int value) { - if (!_generatedCursorUseBigIntVarint64) { - var remaining = value; - for (var index = 0; index < 8; index += 1) { - final chunk = remaining & 0x7f; - remaining = remaining >>> 7; - if (remaining == 0) { - _bytes[_offset] = chunk; - _offset += 1; - return; - } - _bytes[_offset] = chunk | 0x80; - _offset += 1; - } - _bytes[_offset] = remaining & 0xff; - _offset += 1; - return; - } - _writeVarUint64BigInt(BigInt.from(value) & _generatedCursorMask64Big); - } - - void writeVarInt64(int value) { - if (!_generatedCursorUseBigIntVarint64) { - writeVarUint64((value << 1) ^ (value >> 63)); - return; - } - final signed = BigInt.from(value); - final zigZag = - ((signed << 1) ^ BigInt.from(value >> 63)) & _generatedCursorMask64Big; - _writeVarUint64BigInt(zigZag); - } - - void writeTaggedInt64(int value) { - if (value >= -0x40000000 && value <= 0x3fffffff) { - writeInt32(value << 1); - return; - } - writeUint8(0x01); - writeInt64(value); - } - - void writeTaggedUint64(int value) { - if (value >= 0 && value <= 0x7fffffff) { - writeInt32(value << 1); - return; - } - writeUint8(0x01); - writeUint64(value); - } - - void _writeVarUint64BigInt(BigInt value) { - var remaining = value & _generatedCursorMask64Big; - for (var index = 0; index < 8; index += 1) { - final chunk = (remaining & _generatedCursorSevenBitMaskBig).toInt(); - remaining >>= 7; - if (remaining == BigInt.zero) { - _bytes[_offset] = chunk; - _offset += 1; - return; - } - _bytes[_offset] = chunk | 0x80; - _offset += 1; - } - _bytes[_offset] = (remaining & _generatedCursorByteMaskBig).toInt(); - _offset += 1; - } -} - -@internal -final class GeneratedReadCursor { - final Buffer _buffer; - final ByteData _view; - int _offset; - - GeneratedReadCursor._( - this._buffer, - this._view, - this._offset, - ); - - factory GeneratedReadCursor.start(Buffer buffer) { - return GeneratedReadCursor._( - buffer, - bufferByteData(buffer), - bufferReaderIndex(buffer), - ); - } - - void finish() { - bufferSetReaderIndex(_buffer, _offset); - } - - bool readBool() => readUint8() != 0; - - int readByte() { - final value = _view.getInt8(_offset); - _offset += 1; - return value; - } - - int readUint8() { - final value = _view.getUint8(_offset); - _offset += 1; - return value; - } - - int readInt16() { - final value = _view.getInt16(_offset, Endian.little); - _offset += 2; - return value; - } - - int readUint16() { - final value = _view.getUint16(_offset, Endian.little); - _offset += 2; - return value; - } - - int readInt32() { - final value = _view.getInt32(_offset, Endian.little); - _offset += 4; - return value; - } - - int readUint32() { - final value = _view.getUint32(_offset, Endian.little); - _offset += 4; - return value; - } - - int readInt64() { - final value = _view.getInt64(_offset, Endian.little); - _offset += 8; - return value; - } - - int readUint64() { - final value = _view.getUint64(_offset, Endian.little); - _offset += 8; - return value; - } - - Float16 readFloat16() => Float16.fromBits(readUint16()); - - Bfloat16 readBfloat16() => Bfloat16.fromBits(readUint16()); - - double readFloat32() { - final value = _view.getFloat32(_offset, Endian.little); - _offset += 4; - return value; - } - - double readFloat64() { - final value = _view.getFloat64(_offset, Endian.little); - _offset += 8; - return value; - } - - int readVarUint32() { - var shift = 0; - var result = 0; - while (true) { - final byte = readUint8(); - result |= (byte & 0x7f) << shift; - if ((byte & 0x80) == 0) { - return result; - } - shift += 7; - } - } - - int readVarInt32() { - final value = readVarUint32(); - return (value >>> 1) ^ -(value & 1); - } - - int readVarUint64() { - if (!_generatedCursorUseBigIntVarint64) { - var shift = 0; - var result = 0; - while (shift < 56) { - final byte = readUint8(); - result |= (byte & 0x7f) << shift; - if ((byte & 0x80) == 0) { - return result; - } - shift += 7; - } - return result | (readUint8() << 56); - } - return _readVarUint64BigInt().toInt(); - } - - int readVarInt64() { - if (!_generatedCursorUseBigIntVarint64) { - final encoded = readVarUint64(); - return (encoded >>> 1) ^ -(encoded & 1); - } - final encoded = _readVarUint64BigInt(); - final magnitude = (encoded >> 1).toInt(); - if ((encoded & BigInt.one) == BigInt.zero) { - return magnitude; - } - return -magnitude - 1; - } - - int readTaggedInt64() { - final readIndex = _offset; - final first = _view.getInt32(readIndex, Endian.little); - if ((first & 1) == 0) { - _offset = readIndex + 4; - return first >> 1; - } - final value = _view.getInt64(readIndex + 1, Endian.little); - _offset = readIndex + 9; - return value; - } - - int readTaggedUint64() { - final readIndex = _offset; - final first = _view.getUint32(readIndex, Endian.little); - if ((first & 1) == 0) { - _offset = readIndex + 4; - return first >>> 1; - } - final value = _view.getUint64(readIndex + 1, Endian.little); - _offset = readIndex + 9; - return value; - } - - BigInt _readVarUint64BigInt() { - var shift = 0; - var result = BigInt.zero; - while (shift < 56) { - final byte = readUint8(); - result |= BigInt.from(byte & 0x7f) << shift; - if ((byte & 0x80) == 0) { - return result; - } - shift += 7; - } - return result | - ((BigInt.from(readUint8()) & _generatedCursorByteMaskBig) << 56); - } -} - @internal final class GeneratedFieldType { final Type type; + final String? declaredTypeName; final int typeId; final bool nullable; final bool ref; @@ -407,6 +50,7 @@ final class GeneratedFieldType { const GeneratedFieldType({ required this.type, + this.declaredTypeName, required this.typeId, required this.nullable, required this.ref, @@ -417,6 +61,7 @@ final class GeneratedFieldType { meta_types.FieldType toFieldType() { return meta_types.FieldType( type: type, + declaredTypeName: declaredTypeName, typeId: typeId, nullable: nullable, ref: ref, @@ -633,7 +278,7 @@ Decimal readGeneratedDecimalValue(ReadContext context) { } @internal -int generatedDurationWireSeconds(Duration value) { +Int64 generatedDurationWireSeconds(Duration value) { return durationWireSeconds(value); } @@ -643,7 +288,7 @@ int generatedDurationWireNanoseconds(Duration value) { } @internal -Duration readGeneratedDurationFromWire(int seconds, int nanoseconds) { +Duration readGeneratedDurationFromWire(Int64 seconds, int nanoseconds) { return durationFromWire(seconds, nanoseconds); } @@ -663,7 +308,7 @@ int generatedTimestampWireNanoseconds(Timestamp value) { } @internal -int generatedDateTimeWireSeconds(DateTime value) { +Int64 generatedDateTimeWireSeconds(DateTime value) { return dateTimeWireSeconds(value); } @@ -673,12 +318,12 @@ int generatedDateTimeWireNanoseconds(DateTime value) { } @internal -Timestamp readGeneratedTimestampFromWire(int seconds, int nanoseconds) { +Timestamp readGeneratedTimestampFromWire(Int64 seconds, int nanoseconds) { return timestampFromWire(seconds, nanoseconds); } @internal -DateTime readGeneratedDateTimeFromWire(int seconds, int nanoseconds) { +DateTime readGeneratedDateTimeFromWire(Int64 seconds, int nanoseconds) { return dateTimeFromWire(seconds, nanoseconds); } @@ -753,6 +398,7 @@ void writeGeneratedStructFieldInfoValue( } @internal +@pragma('vm:prefer-inline') Object? readGeneratedStructFieldInfoValue( ReadContext context, GeneratedStructFieldInfo field, [ @@ -773,15 +419,14 @@ Object? readGeneratedStructFieldInfoValue( if (fieldUsesDeclaredType(context.typeResolver, field)) { return context.readResolvedValue(resolved, fieldType); } - final actualResolved = context.readTypeMetaValue( - resolved.isNamed ? resolved : null, - ); + final actualResolved = context.readTypeMetaValue(resolved); return context.readResolvedValue(actualResolved, fieldType); } return readFieldValue(context, field, fallback); } @internal +@pragma('vm:prefer-inline') Object? readGeneratedStructDeclaredValue( ReadContext context, GeneratedStructFieldInfo field, @@ -790,13 +435,12 @@ Object? readGeneratedStructDeclaredValue( if (fieldUsesDeclaredType(context.typeResolver, field)) { return context.readResolvedValue(resolved, field.fieldType); } - final actualResolved = context.readTypeMetaValue( - resolved.isNamed ? resolved : null, - ); + final actualResolved = context.readTypeMetaValue(resolved); return context.readResolvedValue(actualResolved, field.fieldType); } @internal +@pragma('vm:prefer-inline') Object readGeneratedStructDirectValue( ReadContext context, GeneratedStructFieldInfo field, @@ -806,7 +450,7 @@ Object readGeneratedStructDirectValue( if (fieldUsesDeclaredType(context.typeResolver, field)) { resolved = declared; } else { - resolved = context.readTypeMetaValue(declared.isNamed ? declared : null); + resolved = context.readTypeMetaValue(declared); } context.increaseDepth(); final value = resolved.structSerializer!.readValue(context, resolved); @@ -815,6 +459,7 @@ Object readGeneratedStructDirectValue( } @internal +@pragma('vm:prefer-inline') List readGeneratedDirectListValue( ReadContext context, GeneratedStructFieldInfo field, @@ -835,6 +480,7 @@ List readGeneratedDirectListValue( } @internal +@pragma('vm:prefer-inline') Set readGeneratedDirectSetValue( ReadContext context, GeneratedStructFieldInfo field, @@ -855,6 +501,7 @@ Set readGeneratedDirectSetValue( } @internal +@pragma('vm:prefer-inline') Map readGeneratedDirectMapValue( ReadContext context, GeneratedStructFieldInfo field, diff --git a/dart/packages/fory/lib/src/context/meta_string_reader.dart b/dart/packages/fory/lib/src/context/meta_string_reader.dart index ba44a68c85..40d6d3aedb 100644 --- a/dart/packages/fory/lib/src/context/meta_string_reader.dart +++ b/dart/packages/fory/lib/src/context/meta_string_reader.dart @@ -19,9 +19,10 @@ import 'dart:typed_data'; -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; import 'package:fory/src/meta/meta_string.dart'; import 'package:fory/src/resolver/type_resolver.dart'; +import 'package:fory/src/types/int64.dart'; typedef _MetaStringWords = ({int length, int word0, int word1, int word2, int word3}); @@ -30,8 +31,8 @@ typedef _MetaStringWords = final class MetaStringReader { final TypeResolver _typeResolver; final List _dynamicReadMetaStrings = []; - final Map _bigMetaStrings = - {}; + final Map _bigMetaStrings = + {}; final Map> _smallMetaStrings = >{}; @@ -79,7 +80,7 @@ final class MetaStringReader { } final encoded = _typeResolver.internEncodedMetaString( buffer.copyBytes(length), - encoding: hash & 0xff, + encoding: (hash & 0xff).toInt(), ); _bigMetaStrings[hash] = encoded; return encoded; diff --git a/dart/packages/fory/lib/src/context/meta_string_writer.dart b/dart/packages/fory/lib/src/context/meta_string_writer.dart index b662d62182..b994f5dc89 100644 --- a/dart/packages/fory/lib/src/context/meta_string_writer.dart +++ b/dart/packages/fory/lib/src/context/meta_string_writer.dart @@ -19,7 +19,7 @@ import 'dart:collection'; -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; import 'package:fory/src/meta/meta_string.dart'; /// Write-side state for meta-string references in one serialization stream. diff --git a/dart/packages/fory/lib/src/context/read_context.dart b/dart/packages/fory/lib/src/context/read_context.dart index 1945e5ae0c..ebec1f320c 100644 --- a/dart/packages/fory/lib/src/context/read_context.dart +++ b/dart/packages/fory/lib/src/context/read_context.dart @@ -19,7 +19,7 @@ import 'package:meta/meta.dart'; -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; import 'package:fory/src/config.dart'; import 'package:fory/src/context/meta_string_reader.dart'; import 'package:fory/src/context/ref_reader.dart'; @@ -32,12 +32,15 @@ import 'package:fory/src/serializer/map_serializers.dart'; import 'package:fory/src/serializer/primitive_serializers.dart'; import 'package:fory/src/serializer/scalar_serializers.dart'; import 'package:fory/src/serializer/serializer.dart'; +import 'package:fory/src/serializer/serializer_support.dart'; import 'package:fory/src/serializer/struct_slots.dart'; import 'package:fory/src/serializer/time_serializers.dart'; import 'package:fory/src/serializer/typed_array_serializers.dart'; import 'package:fory/src/types/bfloat16.dart'; import 'package:fory/src/types/float16.dart'; +import 'package:fory/src/types/int64.dart'; import 'package:fory/src/types/timestamp.dart'; +import 'package:fory/src/types/uint64.dart'; /// Read-side serializer context. /// @@ -96,12 +99,14 @@ final class ReadContext { } @internal + @pragma('vm:prefer-inline') TypeInfo readTypeMetaValue([ TypeInfo? expectedNamedType, ]) => _readTypeMeta(expectedNamedType); @internal + @pragma('vm:prefer-inline') Object? readSerializerPayload( Serializer serializer, TypeInfo resolved, {required bool hasCurrentPreservedRef}) { @@ -124,10 +129,11 @@ final class ReadContext { } return value; } - + int get depth => _depth; /// Records entry into one more nested read frame. + @pragma('vm:prefer-inline') void increaseDepth() { _depth += 1; if (_depth > config.maxDepth) { @@ -136,6 +142,7 @@ final class ReadContext { } /// Records exit from a nested read frame. + @pragma('vm:prefer-inline') void decreaseDepth() { _depth -= 1; } @@ -156,7 +163,7 @@ final class ReadContext { int readInt32() => _buffer.readInt32(); /// Reads a signed little-endian 64-bit integer. - int readInt64() => _buffer.readInt64(); + Int64 readInt64() => _buffer.readInt64(); /// Reads a half-precision floating-point value. Float16 readFloat16() => _buffer.readFloat16(); @@ -177,23 +184,25 @@ final class ReadContext { int readVarUint32() => _buffer.readVarUint32(); /// Reads a zig-zag encoded signed 64-bit varint. - int readVarInt64() => _buffer.readVarInt64(); + Int64 readVarInt64() => _buffer.readVarInt64(); /// Reads a tagged signed 64-bit integer. - int readTaggedInt64() => _buffer.readTaggedInt64(); + Int64 readTaggedInt64() => _buffer.readTaggedInt64(); /// Reads an unsigned 64-bit varint. - int readVarUint64() => _buffer.readVarUint64(); + Uint64 readVarUint64() => _buffer.readVarUint64(); /// Reads a tagged unsigned 64-bit integer. - int readTaggedUint64() => _buffer.readTaggedUint64(); + Uint64 readTaggedUint64() => _buffer.readTaggedUint64(); /// Binds [value] to the most recently preserved Ref slot. + @pragma('vm:prefer-inline') void reference(Object? value) { _refReader.reference(value); } /// Reads a non-null string payload without ref/null handling. + @pragma('vm:prefer-inline') String readString() => StringSerializer.readPayload(this); /// Reads a ref-or-null header and resolves back-references immediately. @@ -219,6 +228,17 @@ final class ReadContext { /// Reads a nullable value using Ref semantics and wire type metadata. Object? readRef() { + return _readRefWithResolved((resolved) => resolved); + } + + /// Reads a root value using Ref semantics and expected root type [T]. + Object? readRefAs() { + return _readRefWithResolved( + (resolved) => _typeResolver.resolveExpectedRootWireType(resolved), + ); + } + + Object? _readRefWithResolved(TypeInfo Function(TypeInfo) resolveRootType) { final flag = _refReader.tryPreserveRefId(_buffer); final preservedRefId = flag >= RefWriter.refValueFlag ? flag : null; if (flag == RefWriter.nullFlag) { @@ -227,7 +247,7 @@ final class ReadContext { if (flag == RefWriter.refFlag) { return _refReader.getReadRef(); } - final resolved = _readTypeMeta(); + final resolved = resolveRootType(_readTypeMeta()); final rootPreservedRefId = preservedRefId == null && flag == RefWriter.notNullValueFlag && _depth == 0 && @@ -274,6 +294,7 @@ final class ReadContext { PrimitiveSerializer.readPayload(this, typeId); @internal + @pragma('vm:prefer-inline') Object? readResolvedValue(TypeInfo resolved, FieldType? declaredFieldType, {bool hasPreservedRef = false}) { if (!_tracksDepth(resolved)) { @@ -299,7 +320,11 @@ final class ReadContext { required bool hasPreservedRef, }) { if (TypeIds.isPrimitive(resolved.typeId)) { - return PrimitiveSerializer.readPayload(this, resolved.typeId); + return convertResolvedPrimitiveValue( + PrimitiveSerializer.readPayload(this, resolved.typeId), + resolved, + declaredFieldType, + ); } switch (resolved.typeId) { case TypeIds.none: @@ -387,6 +412,7 @@ final class ReadContext { } } + @pragma('vm:prefer-inline') TypeInfo _readTypeMeta([ TypeInfo? expectedNamedType, ]) { @@ -398,6 +424,7 @@ final class ReadContext { ); } + @pragma('vm:prefer-inline') bool _tracksDepth(TypeInfo resolved) { if (TypeIds.isContainer(resolved.typeId)) { return true; diff --git a/dart/packages/fory/lib/src/context/ref_reader.dart b/dart/packages/fory/lib/src/context/ref_reader.dart index 50896cd38d..09aaaf7feb 100644 --- a/dart/packages/fory/lib/src/context/ref_reader.dart +++ b/dart/packages/fory/lib/src/context/ref_reader.dart @@ -17,7 +17,7 @@ * under the License. */ -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; import 'package:fory/src/context/ref_writer.dart'; final class RefReader { diff --git a/dart/packages/fory/lib/src/context/ref_writer.dart b/dart/packages/fory/lib/src/context/ref_writer.dart index a9a14ce2a5..864c4f293a 100644 --- a/dart/packages/fory/lib/src/context/ref_writer.dart +++ b/dart/packages/fory/lib/src/context/ref_writer.dart @@ -19,7 +19,7 @@ import 'dart:collection'; -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; final class RefWriter { static const int nullFlag = -3; diff --git a/dart/packages/fory/lib/src/context/write_context.dart b/dart/packages/fory/lib/src/context/write_context.dart index f5fec0e484..a9399818cf 100644 --- a/dart/packages/fory/lib/src/context/write_context.dart +++ b/dart/packages/fory/lib/src/context/write_context.dart @@ -23,7 +23,7 @@ import 'dart:collection'; import 'package:meta/meta.dart'; -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; import 'package:fory/src/config.dart'; import 'package:fory/src/context/meta_string_writer.dart'; import 'package:fory/src/context/ref_writer.dart'; @@ -40,8 +40,10 @@ import 'package:fory/src/serializer/time_serializers.dart'; import 'package:fory/src/serializer/typed_array_serializers.dart'; import 'package:fory/src/types/bfloat16.dart'; import 'package:fory/src/types/float16.dart'; +import 'package:fory/src/types/int64.dart'; import 'package:fory/src/types/local_date.dart'; import 'package:fory/src/types/timestamp.dart'; +import 'package:fory/src/types/uint64.dart'; /// Write-side serializer context. /// @@ -105,7 +107,7 @@ final class WriteContext { set structWriteSlots(StructWriteSlots? value) { _structWriteSlots = value; } - + int get depth => _depth; /// Records entry into one more nested write frame. @@ -137,7 +139,7 @@ final class WriteContext { void writeInt32(int value) => _buffer.writeInt32(value); /// Writes a signed little-endian 64-bit integer. - void writeInt64(int value) => _buffer.writeInt64(value); + void writeInt64(Int64 value) => _buffer.writeInt64(value); /// Writes a half-precision floating-point value. void writeFloat16(Float16 value) => _buffer.writeFloat16(value); @@ -158,16 +160,16 @@ final class WriteContext { void writeVarUint32(int value) => _buffer.writeVarUint32(value); /// Writes a zig-zag encoded signed 64-bit varint. - void writeVarInt64(int value) => _buffer.writeVarInt64(value); + void writeVarInt64(Int64 value) => _buffer.writeVarInt64(value); /// Writes a tagged signed 64-bit integer. - void writeTaggedInt64(int value) => _buffer.writeTaggedInt64(value); + void writeTaggedInt64(Int64 value) => _buffer.writeTaggedInt64(value); /// Writes an unsigned 64-bit varint. - void writeVarUint64(int value) => _buffer.writeVarUint64(value); + void writeVarUint64(Uint64 value) => _buffer.writeVarUint64(value); /// Writes a tagged unsigned 64-bit integer. - void writeTaggedUint64(int value) => _buffer.writeTaggedUint64(value); + void writeTaggedUint64(Uint64 value) => _buffer.writeTaggedUint64(value); /// Writes a non-null string payload without adding type metadata. void writeString(String value) => StringSerializer.writePayload(this, value); diff --git a/dart/packages/fory/lib/src/fory.dart b/dart/packages/fory/lib/src/fory.dart index ca5cffa5ca..72b2e64b97 100644 --- a/dart/packages/fory/lib/src/fory.dart +++ b/dart/packages/fory/lib/src/fory.dart @@ -19,7 +19,7 @@ import 'dart:typed_data'; -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; import 'package:fory/src/config.dart'; import 'package:fory/src/context/meta_string_reader.dart'; import 'package:fory/src/context/meta_string_writer.dart'; @@ -124,6 +124,11 @@ final class Fory { /// Deserializes a value from [buffer] and checks that it is assignable to /// [T]. /// + /// The wire metadata normally determines the decoded value type. Supplying + /// [T] also preserves typed root `Int64` values for varint64 payloads so + /// callers can distinguish them from plain `int` roots where the platform + /// representation supports that distinction. + /// /// Only xlang payloads are supported. This method consumes bytes from the /// current reader position of [buffer]. T deserializeFrom(Buffer buffer) { @@ -143,7 +148,7 @@ final class Fory { 'Only xlang payloads are supported by the Dart runtime.', ); } - final value = _readContext.readRef(); + final value = _readContext.readRefAs(); if (value is T) { return value; } diff --git a/dart/packages/fory/lib/src/memory/buffer.dart b/dart/packages/fory/lib/src/memory/buffer.dart new file mode 100644 index 0000000000..6c0b3fb20a --- /dev/null +++ b/dart/packages/fory/lib/src/memory/buffer.dart @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +export 'buffer_native.dart' if (dart.library.js_interop) 'buffer_web.dart'; diff --git a/dart/packages/fory/lib/src/buffer.dart b/dart/packages/fory/lib/src/memory/buffer_mixin.dart similarity index 58% rename from dart/packages/fory/lib/src/buffer.dart rename to dart/packages/fory/lib/src/memory/buffer_mixin.dart index ee1cdb49a7..8466c9f8cc 100644 --- a/dart/packages/fory/lib/src/buffer.dart +++ b/dart/packages/fory/lib/src/memory/buffer_mixin.dart @@ -17,49 +17,27 @@ * under the License. */ -import 'dart:convert'; -import 'dart:typed_data'; - -import 'package:meta/meta.dart'; - -import 'package:fory/src/types/bfloat16.dart'; -import 'package:fory/src/types/float16.dart'; - -final BigInt _mask64Big = (BigInt.one << 64) - BigInt.one; -final BigInt _sevenBitMaskBig = BigInt.from(0x7f); -final BigInt _byteMaskBig = BigInt.from(0xff); -const bool _useBigIntVarint64 = - bool.fromEnvironment('dart.library.js_interop') || - bool.fromEnvironment('dart.library.js_util'); - -/// A reusable byte buffer with explicit reader and writer indices. -/// -/// Fory uses little-endian fixed-width encodings and varint helpers on top of -/// this buffer. The same buffer can be reused across many operations by calling -/// [clear]. -final class Buffer { - Uint8List _bytes; +// ignore_for_file: use_string_in_part_of_directives + +part of fory.src.memory.buffer; + +mixin _BufferMixin { + late Uint8List _bytes; late ByteData _view; - int _readerIndex; - int _writerIndex; - - /// Creates an empty buffer with [initialCapacity] bytes of storage. - Buffer([int initialCapacity = 256]) - : _bytes = Uint8List(initialCapacity), - _readerIndex = 0, - _writerIndex = 0 { + int _readerIndex = 0; + int _writerIndex = 0; + + void _initBuffer(int initialCapacity) { + _bytes = Uint8List(initialCapacity); _view = ByteData.sublistView(_bytes); } - /// Creates a buffer that reads from and writes into [bytes]. - /// - /// The writer index starts at `bytes.length`, so the wrapped bytes are - /// immediately readable. - Buffer.wrap(Uint8List bytes) - : _bytes = bytes, - _view = ByteData.sublistView(bytes), - _readerIndex = 0, - _writerIndex = bytes.length; + void _wrapBuffer(Uint8List bytes) { + _bytes = bytes; + _view = ByteData.sublistView(bytes); + _readerIndex = 0; + _writerIndex = bytes.length; + } /// Number of unread bytes between the reader and writer indices. int get readableBytes => _writerIndex - _readerIndex; @@ -77,10 +55,7 @@ final class Buffer { /// Replaces the underlying storage with [bytes] and resets both indices. void wrap(Uint8List bytes) { - _bytes = bytes; - _view = ByteData.sublistView(bytes); - _readerIndex = 0; - _writerIndex = bytes.length; + _wrapBuffer(bytes); } /// Ensures there is room for [additionalBytes] bytes past the writer index. @@ -89,7 +64,7 @@ final class Buffer { if (required <= _bytes.length) { return; } - var newLength = _bytes.length; + var newLength = _bytes.isEmpty ? 1 : _bytes.length; while (newLength < required) { newLength *= 2; } @@ -194,34 +169,6 @@ final class Buffer { return value; } - /// Writes a signed little-endian 64-bit integer. - void writeInt64(int value) { - ensureWritable(8); - _view.setInt64(_writerIndex, value, Endian.little); - _writerIndex += 8; - } - - /// Reads a signed little-endian 64-bit integer. - int readInt64() { - final value = _view.getInt64(_readerIndex, Endian.little); - _readerIndex += 8; - return value; - } - - /// Writes an unsigned little-endian 64-bit integer. - void writeUint64(int value) { - ensureWritable(8); - _view.setUint64(_writerIndex, value, Endian.little); - _writerIndex += 8; - } - - /// Reads an unsigned little-endian 64-bit integer. - int readUint64() { - final value = _view.getUint64(_readerIndex, Endian.little); - _readerIndex += 8; - return value; - } - /// Writes a single-precision floating-point value. void writeFloat32(double value) { ensureWritable(4); @@ -318,124 +265,18 @@ final class Buffer { } /// Writes a zig-zag encoded signed 32-bit varint. - void writeVarInt32(int value) => writeVarUint32((value << 1) ^ (value >> 31)); + void writeVarInt32(int value) => + writeVarUint32(((value << 1) ^ (value >> 31)).toUnsigned(32)); /// Reads a zig-zag encoded signed 32-bit varint. int readVarInt32() { final value = readVarUint32(); - return (value >>> 1) ^ -(value & 1); + return ((value >>> 1) ^ -(value & 1)).toSigned(32); } - /// Writes an unsigned 64-bit varint. - void writeVarUint64(int value) { - if (!_useBigIntVarint64) { - var remaining = value; - for (var index = 0; index < 8; index += 1) { - final chunk = remaining & 0x7f; - remaining = remaining >>> 7; - if (remaining == 0) { - writeUint8(chunk); - return; - } - writeUint8(chunk | 0x80); - } - writeUint8(remaining & 0xff); - return; - } - _writeVarUint64BigInt(BigInt.from(value) & _mask64Big); - } + void writeVarUint64(Uint64 value); - /// Reads an unsigned 64-bit varint. - int readVarUint64() { - if (!_useBigIntVarint64) { - var shift = 0; - var result = 0; - while (shift < 56) { - final byte = readUint8(); - result |= (byte & 0x7f) << shift; - if ((byte & 0x80) == 0) { - return result; - } - shift += 7; - } - return result | (readUint8() << 56); - } - return _readVarUint64BigInt().toInt(); - } - - /// Writes a zig-zag encoded signed 64-bit varint. - void writeVarInt64(int value) { - if (!_useBigIntVarint64) { - writeVarUint64((value << 1) ^ (value >> 63)); - return; - } - final signed = BigInt.from(value); - final zigZag = ((signed << 1) ^ BigInt.from(value >> 63)) & _mask64Big; - _writeVarUint64BigInt(zigZag); - } - - /// Reads a zig-zag encoded signed 64-bit varint. - int readVarInt64() { - if (!_useBigIntVarint64) { - final encoded = readVarUint64(); - return (encoded >>> 1) ^ -(encoded & 1); - } - final encoded = _readVarUint64BigInt(); - final magnitude = (encoded >> 1).toInt(); - if ((encoded & BigInt.one) == BigInt.zero) { - return magnitude; - } - return -magnitude - 1; - } - - /// Writes a tagged signed 64-bit integer. - /// - /// Small values use four bytes. Larger values use a tag byte plus eight data - /// bytes. - void writeTaggedInt64(int value) { - if (value >= -0x40000000 && value <= 0x3fffffff) { - writeInt32(value << 1); - return; - } - writeUint8(0x01); - writeInt64(value); - } - - /// Reads a signed 64-bit integer written by [writeTaggedInt64]. - int readTaggedInt64() { - final readIndex = _readerIndex; - final first = _view.getInt32(readIndex, Endian.little); - if ((first & 1) == 0) { - _readerIndex = readIndex + 4; - return first >> 1; - } - final value = _view.getInt64(readIndex + 1, Endian.little); - _readerIndex = readIndex + 9; - return value; - } - - /// Writes a tagged unsigned 64-bit integer. - void writeTaggedUint64(int value) { - if (value >= 0 && value <= 0x7fffffff) { - writeInt32(value << 1); - return; - } - writeUint8(0x01); - writeUint64(value); - } - - /// Reads an unsigned 64-bit integer written by [writeTaggedUint64]. - int readTaggedUint64() { - final readIndex = _readerIndex; - final first = _view.getUint32(readIndex, Endian.little); - if ((first & 1) == 0) { - _readerIndex = readIndex + 4; - return first >>> 1; - } - final value = _view.getUint64(readIndex + 1, Endian.little); - _readerIndex = readIndex + 9; - return value; - } + Uint64 readVarUint64(); /// Writes a small unsigned integer using the same varint path as /// [writeVarUint32]. @@ -452,10 +293,10 @@ final class Buffer { int readVarUint32Small14() => readVarUint32(); /// Writes a small unsigned integer using the 64-bit varint path. - void writeVarUint36Small(int value) => writeVarUint64(value); + void writeVarUint36Small(int value) => writeVarUint64(Uint64(value)); /// Reads a small unsigned integer written by [writeVarUint36Small]. - int readVarUint36Small() => readVarUint64(); + int readVarUint36Small() => readVarUint64().toInt(); } @internal @@ -492,33 +333,3 @@ Uint8List bufferBytes(Buffer buffer) => buffer._bytes; @internal ByteData bufferByteData(Buffer buffer) => buffer._view; - -extension on Buffer { - void _writeVarUint64BigInt(BigInt value) { - var remaining = value & _mask64Big; - for (var index = 0; index < 8; index += 1) { - final chunk = (remaining & _sevenBitMaskBig).toInt(); - remaining >>= 7; - if (remaining == BigInt.zero) { - writeUint8(chunk); - return; - } - writeUint8(chunk | 0x80); - } - writeUint8((remaining & _byteMaskBig).toInt()); - } - - BigInt _readVarUint64BigInt() { - var shift = 0; - var result = BigInt.zero; - while (shift < 56) { - final byte = readUint8(); - result |= BigInt.from(byte & 0x7f) << shift; - if ((byte & 0x80) == 0) { - return result; - } - shift += 7; - } - return result | ((BigInt.from(readUint8()) & _byteMaskBig) << 56); - } -} diff --git a/dart/packages/fory/lib/src/memory/buffer_native.dart b/dart/packages/fory/lib/src/memory/buffer_native.dart new file mode 100644 index 0000000000..31fa8c1d18 --- /dev/null +++ b/dart/packages/fory/lib/src/memory/buffer_native.dart @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// ignore_for_file: unnecessary_library_name + +library fory.src.memory.buffer; + +import 'dart:convert'; +import 'dart:typed_data'; + +import 'package:meta/meta.dart'; + +import 'package:fory/src/types/bfloat16.dart'; +import 'package:fory/src/types/float16.dart'; +import 'package:fory/src/types/int64.dart'; +import 'package:fory/src/types/uint64.dart'; + +part 'buffer_mixin.dart'; + +/// A reusable byte buffer with explicit reader and writer indices. +/// +/// Fory uses little-endian fixed-width encodings and varint helpers on top of +/// this buffer. The same buffer can be reused across many operations by calling +/// [clear]. +final class Buffer with _BufferMixin { + /// Creates an empty buffer with [initialCapacity] bytes of storage. + Buffer([int initialCapacity = 256]) { + _initBuffer(initialCapacity); + } + + /// Creates a buffer that reads from and writes into [bytes]. + /// + /// The writer index starts at `bytes.length`, so the wrapped bytes are + /// immediately readable. + Buffer.wrap(Uint8List bytes) { + _wrapBuffer(bytes); + } + + /// Writes a signed little-endian 64-bit integer. + void writeInt64(Int64 value) { + ensureWritable(8); + _view.setInt64(_writerIndex, value.toInt(), Endian.little); + _writerIndex += 8; + } + + /// Writes a signed little-endian 64-bit integer from a Dart [int]. + void writeInt64FromInt(int value) { + ensureWritable(8); + _view.setInt64(_writerIndex, value, Endian.little); + _writerIndex += 8; + } + + /// Reads a signed little-endian 64-bit integer. + Int64 readInt64() { + final value = Int64(_view.getInt64(_readerIndex, Endian.little)); + _readerIndex += 8; + return value; + } + + /// Reads a signed little-endian 64-bit integer as a Dart [int]. + int readInt64AsInt() { + final value = _view.getInt64(_readerIndex, Endian.little); + _readerIndex += 8; + return value; + } + + /// Writes an unsigned little-endian 64-bit integer. + void writeUint64(Uint64 value) { + ensureWritable(8); + _view.setInt64(_writerIndex, value.value, Endian.little); + _writerIndex += 8; + } + + /// Reads an unsigned little-endian 64-bit integer. + Uint64 readUint64() { + final value = Uint64(_view.getInt64(_readerIndex, Endian.little)); + _readerIndex += 8; + return value; + } + + /// Writes an unsigned 64-bit varint. + @override + void writeVarUint64(Uint64 value) { + ensureWritable(10); + _writeVarUint64Int(value.value); + } + + /// Reads an unsigned 64-bit varint. + @override + Uint64 readVarUint64() { + var shift = 0; + var result = Uint64(0); + while (shift < 56) { + final byte = _view.getUint8(_readerIndex); + _readerIndex += 1; + result = result | (Uint64(byte & 0x7f) << shift); + if ((byte & 0x80) == 0) { + return result; + } + shift += 7; + } + final byte = _view.getUint8(_readerIndex); + _readerIndex += 1; + return result | (Uint64(byte) << 56); + } + + /// Writes a zig-zag encoded signed 64-bit varint. + void writeVarInt64(Int64 value) { + ensureWritable(10); + _writeVarUint64Int((value << 1) ^ (value >> 63)); + } + + /// Writes a zig-zag encoded signed 64-bit varint from a Dart [int]. + void writeVarInt64FromInt(int value) { + ensureWritable(10); + _writeVarUint64Int((value << 1) ^ (value >> 63)); + } + + /// Reads a zig-zag encoded signed 64-bit varint. + Int64 readVarInt64() { + final encoded = readVarUint64(); + return Int64((encoded >>> 1) ^ -(encoded & 1)); + } + + /// Reads a zig-zag encoded signed 64-bit varint as a Dart [int]. + int readVarInt64AsInt() { + final encoded = _readVarUint64AsInt(); + return (encoded >>> 1) ^ -(encoded & 1); + } + + /// Writes a tagged signed 64-bit integer. + /// + /// Small values use four bytes. Larger values use a tag byte plus eight data + /// bytes. + void writeTaggedInt64(Int64 value) { + if (value >= -0x40000000 && value <= 0x3fffffff) { + writeInt32((value.toInt() << 1).toSigned(32)); + return; + } + writeUint8(0x01); + writeInt64(value); + } + + /// Writes a tagged signed 64-bit integer from a Dart [int]. + void writeTaggedInt64FromInt(int value) { + if (value >= -0x40000000 && value <= 0x3fffffff) { + writeInt32((value << 1).toSigned(32)); + return; + } + writeUint8(0x01); + writeInt64FromInt(value); + } + + /// Reads a signed 64-bit integer written by [writeTaggedInt64]. + Int64 readTaggedInt64() { + final readIndex = _readerIndex; + final first = _view.getInt32(readIndex, Endian.little); + if ((first & 1) == 0) { + _readerIndex = readIndex + 4; + return Int64(first.toSigned(32) ~/ 2); + } + final value = Int64(_view.getInt64(readIndex + 1, Endian.little)); + _readerIndex = readIndex + 9; + return value; + } + + /// Reads a signed 64-bit integer written by [writeTaggedInt64] as an [int]. + int readTaggedInt64AsInt() { + final readIndex = _readerIndex; + final first = _view.getInt32(readIndex, Endian.little); + if ((first & 1) == 0) { + _readerIndex = readIndex + 4; + return first.toSigned(32) ~/ 2; + } + final value = _view.getInt64(readIndex + 1, Endian.little); + _readerIndex = readIndex + 9; + return value; + } + + /// Writes a tagged unsigned 64-bit integer. + void writeTaggedUint64(Uint64 value) { + if (value >= 0 && value <= 0x7fffffff) { + writeInt32(value.toInt() << 1); + return; + } + writeUint8(0x01); + writeUint64(value); + } + + /// Reads an unsigned 64-bit integer written by [writeTaggedUint64]. + Uint64 readTaggedUint64() { + final readIndex = _readerIndex; + final first = _view.getUint32(readIndex, Endian.little); + if ((first & 1) == 0) { + _readerIndex = readIndex + 4; + return Uint64(first >>> 1); + } + final value = Uint64(_view.getInt64(readIndex + 1, Endian.little)); + _readerIndex = readIndex + 9; + return value; + } + + @pragma('vm:prefer-inline') + void _writeVarUint64Int(int value) { + var remaining = value; + for (var index = 0; index < 8; index += 1) { + final chunk = remaining & 0x7f; + remaining >>>= 7; + if (remaining == 0) { + _bytes[_writerIndex] = chunk; + _writerIndex += 1; + return; + } + _bytes[_writerIndex] = chunk | 0x80; + _writerIndex += 1; + } + _bytes[_writerIndex] = remaining & 0xff; + _writerIndex += 1; + } + + @pragma('vm:prefer-inline') + int _readVarUint64AsInt() { + var shift = 0; + var result = 0; + while (shift < 56) { + final byte = _view.getUint8(_readerIndex); + _readerIndex += 1; + result |= (byte & 0x7f) << shift; + if ((byte & 0x80) == 0) { + return result; + } + shift += 7; + } + final byte = _view.getUint8(_readerIndex); + _readerIndex += 1; + return result | (byte << 56); + } +} diff --git a/dart/packages/fory/lib/src/memory/buffer_web.dart b/dart/packages/fory/lib/src/memory/buffer_web.dart new file mode 100644 index 0000000000..9a4428355c --- /dev/null +++ b/dart/packages/fory/lib/src/memory/buffer_web.dart @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// ignore_for_file: unnecessary_library_name + +library fory.src.memory.buffer; + +import 'dart:convert'; +import 'dart:typed_data'; + +import 'package:meta/meta.dart'; + +import 'package:fory/src/types/bfloat16.dart'; +import 'package:fory/src/types/float16.dart'; +import 'package:fory/src/types/int64.dart'; +import 'package:fory/src/types/uint64.dart'; + +part 'buffer_mixin.dart'; + +const int _jsSafeIntMax = 9007199254740991; +const int _jsSafeIntMin = -9007199254740991; + +/// A reusable byte buffer with explicit reader and writer indices. +/// +/// Fory uses little-endian fixed-width encodings and varint helpers on top of +/// this buffer. The same buffer can be reused across many operations by calling +/// [clear]. +final class Buffer with _BufferMixin { + /// Creates an empty buffer with [initialCapacity] bytes of storage. + Buffer([int initialCapacity = 256]) { + _initBuffer(initialCapacity); + } + + /// Creates a buffer that reads from and writes into [bytes]. + /// + /// The writer index starts at `bytes.length`, so the wrapped bytes are + /// immediately readable. + Buffer.wrap(Uint8List bytes) { + _wrapBuffer(bytes); + } + + /// Writes a signed little-endian 64-bit integer. + void writeInt64(Int64 value) { + ensureWritable(8); + _writeInt64Words(_writerIndex, value); + _writerIndex += 8; + } + + /// Writes a signed little-endian 64-bit integer from a Dart [int]. + void writeInt64FromInt(int value) { + ensureWritable(8); + _writeInt64Words(_writerIndex, _int64FromInt(value)); + _writerIndex += 8; + } + + /// Reads a signed little-endian 64-bit integer. + Int64 readInt64() { + final value = _readInt64Words(_readerIndex); + _readerIndex += 8; + return value; + } + + /// Reads a signed little-endian 64-bit integer as a Dart [int]. + int readInt64AsInt() { + final value = _int64ToInt(_readInt64Words(_readerIndex)); + _readerIndex += 8; + return value; + } + + /// Writes an unsigned little-endian 64-bit integer. + void writeUint64(Uint64 value) { + ensureWritable(8); + _writeUint64Words(_writerIndex, value); + _writerIndex += 8; + } + + /// Reads an unsigned little-endian 64-bit integer. + Uint64 readUint64() { + final value = _readUint64Words(_readerIndex); + _readerIndex += 8; + return value; + } + + /// Writes an unsigned 64-bit varint. + @override + void writeVarUint64(Uint64 value) { + ensureWritable(10); + var remaining = value; + for (var shift = 0; shift < 56 && remaining > 0x7f; shift += 7) { + _bytes[_writerIndex] = (remaining.low32 & 0x7f) | 0x80; + _writerIndex += 1; + remaining = remaining >> 7; + } + _bytes[_writerIndex] = remaining.toInt(); + _writerIndex += 1; + } + + /// Reads an unsigned 64-bit varint. + @override + Uint64 readVarUint64() { + var shift = 0; + var result = Uint64(0); + while (shift < 56) { + final byte = _view.getUint8(_readerIndex); + _readerIndex += 1; + result = result | (Uint64(byte & 0x7f) << shift); + if ((byte & 0x80) == 0) { + return result; + } + shift += 7; + } + final byte = _view.getUint8(_readerIndex); + _readerIndex += 1; + return result | (Uint64(byte) << 56); + } + + /// Writes a zig-zag encoded signed 64-bit varint. + void writeVarInt64(Int64 value) { + writeVarUint64(_zigZagEncodeInt64(value)); + } + + /// Writes a zig-zag encoded signed 64-bit varint from a Dart [int]. + void writeVarInt64FromInt(int value) { + writeVarInt64(_int64FromInt(value)); + } + + /// Reads a zig-zag encoded signed 64-bit varint. + Int64 readVarInt64() { + return _zigZagDecodeInt64(readVarUint64()); + } + + /// Reads a zig-zag encoded signed 64-bit varint as a Dart [int]. + int readVarInt64AsInt() { + return _int64ToInt(readVarInt64()); + } + + /// Writes a tagged signed 64-bit integer. + /// + /// Small values use four bytes. Larger values use a tag byte plus eight data + /// bytes. + void writeTaggedInt64(Int64 value) { + if (value >= -0x40000000 && value <= 0x3fffffff) { + writeInt32((value.toInt() << 1).toSigned(32)); + return; + } + writeUint8(0x01); + writeInt64(value); + } + + /// Writes a tagged signed 64-bit integer from a Dart [int]. + void writeTaggedInt64FromInt(int value) { + if (value >= -0x40000000 && value <= 0x3fffffff) { + writeInt32((value << 1).toSigned(32)); + return; + } + _checkInt64IntRange(value); + writeUint8(0x01); + writeInt64FromInt(value); + } + + /// Reads a signed 64-bit integer written by [writeTaggedInt64]. + Int64 readTaggedInt64() { + final readIndex = _readerIndex; + final first = _view.getInt32(readIndex, Endian.little); + if ((first & 1) == 0) { + _readerIndex = readIndex + 4; + return Int64(first.toSigned(32) ~/ 2); + } + final value = _readInt64Words(readIndex + 1); + _readerIndex = readIndex + 9; + return value; + } + + /// Reads a signed 64-bit integer written by [writeTaggedInt64] as an [int]. + int readTaggedInt64AsInt() { + final readIndex = _readerIndex; + final first = _view.getInt32(readIndex, Endian.little); + if ((first & 1) == 0) { + _readerIndex = readIndex + 4; + return first.toSigned(32) ~/ 2; + } + final value = _int64ToInt(_readInt64Words(readIndex + 1)); + _readerIndex = readIndex + 9; + return value; + } + + /// Writes a tagged unsigned 64-bit integer. + void writeTaggedUint64(Uint64 value) { + if (value >= 0 && value <= 0x7fffffff) { + writeInt32(value.toInt() << 1); + return; + } + writeUint8(0x01); + writeUint64(value); + } + + /// Reads an unsigned 64-bit integer written by [writeTaggedUint64]. + Uint64 readTaggedUint64() { + final readIndex = _readerIndex; + final first = _view.getUint32(readIndex, Endian.little); + if ((first & 1) == 0) { + _readerIndex = readIndex + 4; + return Uint64(first >>> 1); + } + final value = _readUint64Words(readIndex + 1); + _readerIndex = readIndex + 9; + return value; + } + + @pragma('vm:prefer-inline') + void _writeInt64Words(int offset, Int64 value) { + _view.setUint32(offset, value.low32, Endian.little); + _view.setUint32(offset + 4, value.high32Unsigned, Endian.little); + } + + @pragma('vm:prefer-inline') + Int64 _readInt64Words(int offset) { + return Int64.fromWords( + _view.getUint32(offset, Endian.little), + _view.getInt32(offset + 4, Endian.little), + ); + } + + @pragma('vm:prefer-inline') + void _writeUint64Words(int offset, Uint64 value) { + _view.setUint32(offset, value.low32, Endian.little); + _view.setUint32(offset + 4, value.high32Unsigned, Endian.little); + } + + @pragma('vm:prefer-inline') + Uint64 _readUint64Words(int offset) { + return Uint64.fromWords( + _view.getUint32(offset, Endian.little), + _view.getUint32(offset + 4, Endian.little), + ); + } +} + +@pragma('vm:prefer-inline') +void _checkInt64IntRange(int value) { + if (value < _jsSafeIntMin || value > _jsSafeIntMax) { + throw StateError( + 'Dart int value $value is outside the JS-safe signed int64 range ' + '[$_jsSafeIntMin, $_jsSafeIntMax]. Use Int64 for full 64-bit values ' + 'on web.', + ); + } +} + +@pragma('vm:prefer-inline') +Int64 _int64FromInt(int value) { + _checkInt64IntRange(value); + return Int64(value); +} + +@pragma('vm:prefer-inline') +int _int64ToInt(Int64 value) => value.toInt(); + +@pragma('vm:prefer-inline') +Uint64 _zigZagEncodeInt64(Int64 value) { + final encoded = (value << 1) ^ (value >> 63); + return Uint64.fromWords(encoded.low32, encoded.high32Unsigned); +} + +@pragma('vm:prefer-inline') +Int64 _zigZagDecodeInt64(Uint64 encoded) { + final magnitude = encoded >> 1; + final decoded = Int64.fromWords(magnitude.low32, magnitude.high32Unsigned); + if ((encoded.low32 & 1) == 0) { + return decoded; + } + return -(decoded + 1); +} diff --git a/dart/packages/fory/lib/src/meta/field_type.dart b/dart/packages/fory/lib/src/meta/field_type.dart index 140f086899..c4359856d3 100644 --- a/dart/packages/fory/lib/src/meta/field_type.dart +++ b/dart/packages/fory/lib/src/meta/field_type.dart @@ -21,6 +21,7 @@ import 'package:fory/src/meta/type_ids.dart'; final class FieldType { final Type type; + final String? declaredTypeName; final int typeId; final bool nullable; final bool ref; @@ -29,6 +30,7 @@ final class FieldType { const FieldType({ required this.type, + this.declaredTypeName, required this.typeId, required this.nullable, required this.ref, @@ -51,6 +53,7 @@ final class FieldType { } return FieldType( type: type, + declaredTypeName: declaredTypeName, typeId: typeId, nullable: nullable, ref: ref, diff --git a/dart/packages/fory/lib/src/meta/meta_string.dart b/dart/packages/fory/lib/src/meta/meta_string.dart index 935b4b47fc..abd60d517a 100644 --- a/dart/packages/fory/lib/src/meta/meta_string.dart +++ b/dart/packages/fory/lib/src/meta/meta_string.dart @@ -20,6 +20,7 @@ import 'dart:convert'; import 'dart:typed_data'; +import 'package:fory/src/types/int64.dart'; import 'package:fory/src/util/hash_util.dart'; const int metaStringUtf8Encoding = 0; @@ -54,7 +55,7 @@ const List _fieldNameCompactEncodings = [ final class EncodedMetaString { final Uint8List bytes; final int encoding; - final int hash; + final Int64 hash; final int firstWord0; final int firstWord1; final int secondWord0; diff --git a/dart/packages/fory/lib/src/meta/type_def.dart b/dart/packages/fory/lib/src/meta/type_def.dart index 9344223214..cd84804f44 100644 --- a/dart/packages/fory/lib/src/meta/type_def.dart +++ b/dart/packages/fory/lib/src/meta/type_def.dart @@ -20,11 +20,12 @@ import 'dart:typed_data'; import 'package:fory/src/meta/field_info.dart'; +import 'package:fory/src/types/int64.dart'; final class TypeDef { final bool evolving; final List fields; - final int header; + final Int64 header; final Uint8List encoded; const TypeDef({ diff --git a/dart/packages/fory/lib/src/meta/type_meta.dart b/dart/packages/fory/lib/src/meta/type_meta.dart index 5a40c85060..dc8a2df5e1 100644 --- a/dart/packages/fory/lib/src/meta/type_meta.dart +++ b/dart/packages/fory/lib/src/meta/type_meta.dart @@ -19,11 +19,12 @@ import 'dart:collection'; -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; import 'package:fory/src/config.dart'; import 'package:fory/src/meta/meta_string.dart'; import 'package:fory/src/meta/type_ids.dart'; import 'package:fory/src/resolver/type_resolver.dart'; +import 'package:fory/src/types/int64.dart'; /// Wire-level type metadata for one value. final class WireTypeMeta { @@ -52,18 +53,20 @@ final class WireTypeMeta { } final class TypeHeader { - final int value; + final Int64 value; const TypeHeader(this.value); + @pragma('vm:prefer-inline') int readMetaSize(Buffer buffer) { - final lowBits = value & 0xff; + final lowBits = value.low32 & 0xff; if (lowBits == 0xff) { return 0xff + buffer.readVarUint32Small14(); } return lowBits; } + @pragma('vm:prefer-inline') void skipRemaining(Buffer buffer) { buffer.skip(readMetaSize(buffer)); } @@ -72,10 +75,12 @@ final class TypeHeader { final class ParsedTypeMetaCache { static const int maxEntries = 8192; - final LinkedHashMap _entries = LinkedHashMap(); - int? _lastHeader; + final LinkedHashMap _entries = + LinkedHashMap(); + Int64? _lastHeader; TypeInfo? _lastResolved; + @pragma('vm:prefer-inline') TypeInfo? lookup(TypeHeader header) { if (_lastHeader == header.value) { return _lastResolved; @@ -88,6 +93,7 @@ final class ParsedTypeMetaCache { return resolved; } + @pragma('vm:prefer-inline') void remember(TypeHeader header, TypeInfo resolved) { if (!_entries.containsKey(header.value) && _entries.length >= maxEntries) { _entries.remove(_entries.keys.first); diff --git a/dart/packages/fory/lib/src/resolver/type_resolver.dart b/dart/packages/fory/lib/src/resolver/type_resolver.dart index 3c39a06540..8e826275f0 100644 --- a/dart/packages/fory/lib/src/resolver/type_resolver.dart +++ b/dart/packages/fory/lib/src/resolver/type_resolver.dart @@ -20,7 +20,7 @@ import 'dart:collection'; import 'dart:typed_data'; -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; import 'package:fory/src/codegen/generated_registry.dart'; import 'package:fory/src/config.dart'; import 'package:fory/src/context/meta_string_reader.dart'; @@ -48,6 +48,7 @@ import 'package:fory/src/types/float16.dart'; import 'package:fory/src/types/float32.dart'; import 'package:fory/src/types/int16.dart'; import 'package:fory/src/types/int32.dart'; +import 'package:fory/src/types/int64.dart'; import 'package:fory/src/types/int8.dart'; import 'package:fory/src/types/local_date.dart'; import 'package:fory/src/types/timestamp.dart'; @@ -323,6 +324,13 @@ final class TypeResolver { if (value is Int32) { return _builtin(Int32, TypeIds.varInt32); } + // Native Int64 is represented as `int`; web still needs this branch to + // keep explicit wrapper values on the Int64 API while using varint64 wire + // format by default, the same as plain Dart int. + // ignore: unnecessary_type_check + if (value is Int64 && value is! int) { + return _builtin(Int64, TypeIds.varInt64); + } if (value is int) { return _builtin(int, TypeIds.varInt64); } @@ -333,10 +341,10 @@ final class TypeResolver { return _builtin(Uint16, TypeIds.uint16); } if (value is Uint32) { - return _builtin(Uint32, TypeIds.uint32); + return _builtin(Uint32, TypeIds.varUint32); } if (value is Uint64) { - return _builtin(Uint64, TypeIds.uint64); + return _builtin(Uint64, TypeIds.varUint64); } if (value is Float16) { return _builtin(Float16, TypeIds.float16); @@ -467,7 +475,7 @@ final class TypeResolver { case TypeIds.bfloat16Array: case TypeIds.float32Array: case TypeIds.float64Array: - return _builtin(fieldType.type, fieldType.typeId); + return _builtin(_builtinTypeForFieldType(fieldType), fieldType.typeId); default: return _registeredByType[fieldType.type]; } @@ -622,6 +630,7 @@ final class TypeResolver { ); } + @pragma('vm:prefer-inline') TypeInfo readTypeMeta( Buffer buffer, { TypeInfo? expectedNamedType, @@ -646,6 +655,7 @@ final class TypeResolver { readTypeDef: () => _readTypeDef( buffer, sharedTypes: sharedTypes, + expectedType: expectedNamedType, ), readPackageMetaString: ([expected]) => metaStringReader.readMetaString(buffer, expected), @@ -847,9 +857,11 @@ final class TypeResolver { return fieldType.ref ? TypeIds.unknown : fieldType.typeId; } + @pragma('vm:prefer-inline') WireTypeMeta _readTypeDef( Buffer buffer, { required List sharedTypes, + TypeInfo? expectedType, }) { final marker = buffer.readVarUint32Small14(); final isRef = (marker & 1) == 1; @@ -858,6 +870,12 @@ final class TypeResolver { return wireTypeMetaForResolved(sharedTypes[index]); } final header = TypeHeader(buffer.readInt64()); + final expectedTypeDef = expectedType?.typeDef; + if (expectedTypeDef != null && expectedTypeDef.header == header.value) { + header.skipRemaining(buffer); + sharedTypes.add(expectedType!); + return wireTypeMetaForResolved(expectedType); + } final cached = _parsedTypeMetaCache.lookup(header); if (cached != null) { header.skipRemaining(buffer); @@ -988,6 +1006,7 @@ final class TypeResolver { } return FieldType( type: Object, + declaredTypeName: null, typeId: typeId, nullable: nullable, ref: ref, @@ -1019,11 +1038,11 @@ final class TypeResolver { case TypeIds.varInt32: return _builtin(Int32, TypeIds.varInt32); case TypeIds.int64: - return _builtin(int, TypeIds.int64); + return _builtin(Int64, TypeIds.int64); case TypeIds.varInt64: return _builtin(int, TypeIds.varInt64); case TypeIds.taggedInt64: - return _builtin(int, TypeIds.taggedInt64); + return _builtin(Int64, TypeIds.taggedInt64); case TypeIds.uint8: return _builtin(Uint8, TypeIds.uint8); case TypeIds.uint16: @@ -1097,6 +1116,14 @@ final class TypeResolver { } } + TypeInfo resolveExpectedRootWireType(TypeInfo resolved) { + if ((_isType() || _isType()) && + resolved.typeId == TypeIds.varInt64) { + return _builtin(Int64, TypeIds.varInt64); + } + return resolved; + } + TypeInfo _builtin(Type type, int typeId) { final key = _BuiltinKey(type, typeId); final cached = _builtinByKey[key]; @@ -1238,9 +1265,12 @@ final class TypeResolver { if (type == Int32) { return TypeIds.varInt32; } - if (type == int) { + if (type == Int64) { return TypeIds.varInt64; } + if (type == Int64List) { + return TypeIds.int64Array; + } if (type == Uint8) { return TypeIds.uint8; } @@ -1248,10 +1278,10 @@ final class TypeResolver { return TypeIds.uint16; } if (type == Uint32) { - return TypeIds.uint32; + return TypeIds.varUint32; } if (type == Uint64) { - return TypeIds.uint64; + return TypeIds.varUint64; } if (type == Uint64List) { return TypeIds.uint64Array; @@ -1422,6 +1452,43 @@ final class TypeResolver { return true; } + Type _builtinTypeForFieldType(FieldType fieldType) { + switch (fieldType.typeId) { + case TypeIds.int64: + case TypeIds.varInt64: + case TypeIds.taggedInt64: + case TypeIds.uint64: + case TypeIds.varUint64: + case TypeIds.taggedUint64: + case TypeIds.int64Array: + case TypeIds.uint64Array: + break; + default: + return fieldType.type; + } + final declaredTypeName = fieldType.declaredTypeName; + if (declaredTypeName != null) { + if (_matchesDeclaredTypeName(declaredTypeName, 'Int64')) { + return Int64; + } + if (_matchesDeclaredTypeName(declaredTypeName, 'Uint64')) { + return Uint64; + } + if (_matchesDeclaredTypeName(declaredTypeName, 'Int64List')) { + return Int64List; + } + if (_matchesDeclaredTypeName(declaredTypeName, 'Uint64List')) { + return Uint64List; + } + } + return fieldType.type; + } + + bool _matchesDeclaredTypeName(String declaredTypeName, String typeName) { + return declaredTypeName == typeName || + declaredTypeName.endsWith('.$typeName'); + } + bool _matchesNamedWireType(TypeInfo resolved, int wireTypeId) { if (!resolved.isNamed) { return false; @@ -1441,6 +1508,8 @@ final class TypeResolver { } } +bool _isType() => T == U; + final class _BuiltinKey { final Type type; final int typeId; diff --git a/dart/packages/fory/lib/src/serializer/collection_serializers.dart b/dart/packages/fory/lib/src/serializer/collection_serializers.dart index d4d59a6eaa..f65feb7236 100644 --- a/dart/packages/fory/lib/src/serializer/collection_serializers.dart +++ b/dart/packages/fory/lib/src/serializer/collection_serializers.dart @@ -71,7 +71,11 @@ Object? readTypeInfoValue( bool hasPreservedRef = false, }) { if (TypeIds.isPrimitive(typeInfo.typeId)) { - return PrimitiveSerializer.readPayload(context, typeInfo.typeId); + return convertResolvedPrimitiveValue( + PrimitiveSerializer.readPayload(context, typeInfo.typeId), + typeInfo, + fieldType, + ); } if (typeInfo.typeId == TypeIds.string) { return StringSerializer.readPayload(context); @@ -400,6 +404,7 @@ final class SetSerializer extends Serializer { const ListSerializer listSerializer = ListSerializer(); const SetSerializer setSerializer = SetSerializer(); +@pragma('vm:prefer-inline') List readTypedListPayload( ReadContext context, FieldType? elementFieldType, @@ -412,6 +417,45 @@ List readTypedListPayload( if (state.tracksDepth) { context.increaseDepth(); } + final directTypeInfo = state.declaredTypeInfo ?? state.sameTypeInfo; + if (directTypeInfo != null && !state.trackRef && !state.hasNull) { + final directFieldType = + state.declaredTypeInfo != null ? state.elementFieldType : null; + if (directTypeInfo.type == T && + directTypeInfo.kind == RegistrationKind.struct) { + final structSerializer = directTypeInfo.structSerializer!; + final result = List.generate( + state.size, + (_) => structSerializer.readValue(context, directTypeInfo) as T, + growable: false, + ); + if (state.tracksDepth) { + context.decreaseDepth(); + } + return result; + } + if (directTypeInfo.type == T && directTypeInfo.typeId == TypeIds.string) { + final result = List.generate( + state.size, + (_) => StringSerializer.readPayload(context) as T, + growable: false, + ); + if (state.tracksDepth) { + context.decreaseDepth(); + } + return result; + } + final result = List.generate( + state.size, + (_) => + convert(readTypeInfoValue(context, directTypeInfo, directFieldType)), + growable: false, + ); + if (state.tracksDepth) { + context.decreaseDepth(); + } + return result; + } final result = List.generate( state.size, (_) => convert(_readPreparedListItem(context, state)), @@ -574,6 +618,7 @@ final class _PreparedListRead { }); } +@pragma('vm:prefer-inline') _PreparedListRead _prepareListRead( ReadContext context, FieldType? elementFieldType, @@ -602,13 +647,16 @@ _PreparedListRead _prepareListRead( final usesDeclaredType = (header & CollectionFlags.isDeclaredElementType) != 0; final sameType = (header & CollectionFlags.isSameType) != 0; - final declaredTypeInfo = usesDeclaredType && elementFieldType != null - ? context.typeResolver.resolveFieldType( - elementFieldType.withRootOverrides(nullable: hasNull, ref: trackRef), - ) + final needsExpectedElementType = elementFieldType != null && + (usesDeclaredType || + (sameType && TypeIds.isUserType(elementFieldType.typeId))); + final expectedElementTypeInfo = needsExpectedElementType + ? context.typeResolver.tryResolveFieldType(elementFieldType) + : null; + final declaredTypeInfo = usesDeclaredType ? expectedElementTypeInfo : null; + final sameTypeInfo = (!usesDeclaredType && sameType) + ? context.readTypeMetaValue(expectedElementTypeInfo) : null; - final sameTypeInfo = - (!usesDeclaredType && sameType) ? context.readTypeMetaValue() : null; final tracksDepth = (declaredTypeInfo != null && tracksNestedPayloadDepth(declaredTypeInfo)) || (sameTypeInfo != null && tracksNestedPayloadDepth(sameTypeInfo)); diff --git a/dart/packages/fory/lib/src/serializer/map_serializers.dart b/dart/packages/fory/lib/src/serializer/map_serializers.dart index 57ae790303..4f1a35cb57 100644 --- a/dart/packages/fory/lib/src/serializer/map_serializers.dart +++ b/dart/packages/fory/lib/src/serializer/map_serializers.dart @@ -17,7 +17,7 @@ * under the License. */ -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; import 'package:fory/src/context/read_context.dart'; import 'package:fory/src/context/ref_writer.dart'; import 'package:fory/src/context/write_context.dart'; diff --git a/dart/packages/fory/lib/src/serializer/primitive_serializers.dart b/dart/packages/fory/lib/src/serializer/primitive_serializers.dart index 2b00b68e6d..af8e249ee8 100644 --- a/dart/packages/fory/lib/src/serializer/primitive_serializers.dart +++ b/dart/packages/fory/lib/src/serializer/primitive_serializers.dart @@ -26,12 +26,16 @@ import 'package:fory/src/types/float16.dart'; import 'package:fory/src/types/float32.dart'; import 'package:fory/src/types/int16.dart'; import 'package:fory/src/types/int32.dart'; +import 'package:fory/src/types/int64.dart'; import 'package:fory/src/types/int8.dart'; import 'package:fory/src/types/uint16.dart'; import 'package:fory/src/types/uint32.dart'; import 'package:fory/src/types/uint64.dart'; import 'package:fory/src/types/uint8.dart'; +const int _jsSafeUint64IntMax = 9007199254740991; +const bool _isWeb = bool.fromEnvironment('dart.library.js_interop'); + final class PrimitiveSerializer extends Serializer { final int typeId; final bool _supportsRef; @@ -77,13 +81,25 @@ final class PrimitiveSerializer extends Serializer { buffer.writeVarInt32(value is Int32 ? value.value : value as int); return; case TypeIds.int64: - buffer.writeInt64(value as int); + if (value is Int64) { + buffer.writeInt64(value); + } else { + buffer.writeInt64FromInt(value as int); + } return; case TypeIds.varInt64: - buffer.writeVarInt64(value as int); + if (value is Int64) { + buffer.writeVarInt64(value); + } else { + buffer.writeVarInt64FromInt(value as int); + } return; case TypeIds.taggedInt64: - buffer.writeTaggedInt64(value as int); + if (value is Int64) { + buffer.writeTaggedInt64(value); + } else { + buffer.writeTaggedInt64FromInt(value as int); + } return; case TypeIds.uint8: buffer.writeUint8(value is Uint8 ? value.value : value as int); @@ -98,13 +114,13 @@ final class PrimitiveSerializer extends Serializer { buffer.writeVarUint32(value is Uint32 ? value.value : value as int); return; case TypeIds.uint64: - buffer.writeUint64(value is Uint64 ? value.value : value as int); + buffer.writeUint64(_uint64Value(value)); return; case TypeIds.varUint64: - buffer.writeVarUint64(value is Uint64 ? value.value : value as int); + buffer.writeVarUint64(_uint64Value(value)); return; case TypeIds.taggedUint64: - buffer.writeTaggedUint64(value is Uint64 ? value.value : value as int); + buffer.writeTaggedUint64(_uint64Value(value)); return; case TypeIds.float16: buffer.writeFloat16(value as Float16); @@ -154,11 +170,11 @@ final class PrimitiveSerializer extends Serializer { case TypeIds.varUint32: return Uint32(buffer.readVarUint32()); case TypeIds.uint64: - return Uint64(buffer.readUint64()); + return buffer.readUint64(); case TypeIds.varUint64: - return Uint64(buffer.readVarUint64()); + return buffer.readVarUint64(); case TypeIds.taggedUint64: - return Uint64(buffer.readTaggedUint64()); + return buffer.readTaggedUint64(); case TypeIds.float16: return buffer.readFloat16(); case TypeIds.bfloat16: @@ -173,6 +189,21 @@ final class PrimitiveSerializer extends Serializer { } } +Uint64 _uint64Value(Object value) { + if (value is Uint64) { + return value; + } + final intValue = value as int; + if (_isWeb && (intValue < 0 || intValue > _jsSafeUint64IntMax)) { + throw StateError( + 'Dart int value $intValue is outside the JS-safe unsigned uint64 ' + 'int field range [0, $_jsSafeUint64IntMax]. Use Uint64 for full ' + 'unsigned 64-bit values on web.', + ); + } + return Uint64(intValue); +} + const PrimitiveSerializer boolSerializer = PrimitiveSerializer( TypeIds.boolType, supportsRef: false, @@ -194,15 +225,17 @@ const PrimitiveSerializer varInt32Serializer = TypeIds.varInt32, supportsRef: false, ); -const PrimitiveSerializer int64Serializer = PrimitiveSerializer( +const PrimitiveSerializer int64Serializer = PrimitiveSerializer( TypeIds.int64, supportsRef: false, ); -const PrimitiveSerializer varInt64Serializer = PrimitiveSerializer( +const PrimitiveSerializer varInt64Serializer = + PrimitiveSerializer( TypeIds.varInt64, supportsRef: false, ); -const PrimitiveSerializer taggedInt64Serializer = PrimitiveSerializer( +const PrimitiveSerializer taggedInt64Serializer = + PrimitiveSerializer( TypeIds.taggedInt64, supportsRef: false, ); diff --git a/dart/packages/fory/lib/src/serializer/scalar_serializers.dart b/dart/packages/fory/lib/src/serializer/scalar_serializers.dart index 0fbfee8053..053319e40f 100644 --- a/dart/packages/fory/lib/src/serializer/scalar_serializers.dart +++ b/dart/packages/fory/lib/src/serializer/scalar_serializers.dart @@ -22,6 +22,8 @@ import 'dart:typed_data'; import 'package:fory/src/context/read_context.dart'; import 'package:fory/src/context/write_context.dart'; import 'package:fory/src/serializer/serializer.dart'; +import 'package:fory/src/types/int64.dart'; +import 'package:fory/src/types/uint64.dart'; import 'package:fory/src/util/string_util.dart'; import 'package:fory/src/types/decimal.dart'; @@ -56,6 +58,20 @@ BigInt _decimalMagnitudeFromCanonicalLittleEndian(Uint8List payload) { return magnitude; } +Uint64 _zigZagEncodeInt64(Int64 value) { + final encoded = (value << 1) ^ (value >> 63); + return Uint64.fromWords(encoded.low32, encoded.high32Unsigned); +} + +Int64 _zigZagDecodeInt64(Uint64 encoded) { + final magnitude = encoded >> 1; + final decoded = Int64.fromWords(magnitude.low32, magnitude.high32Unsigned); + if ((encoded.low32 & 1) == 0) { + return decoded; + } + return -(decoded + 1); +} + final class NoneSerializer extends Serializer { const NoneSerializer(); @@ -157,8 +173,7 @@ final class DecimalSerializer extends Serializer { final unscaled = value.unscaledValue; buffer.writeVarInt32(value.scale); if (_canUseSmallDecimalEncoding(unscaled)) { - final smallValue = unscaled.toInt(); - final zigZag = (smallValue << 1) ^ (smallValue >> 63); + final zigZag = _zigZagEncodeInt64(Int64.fromBigInt(unscaled)); buffer.writeVarUint64(zigZag << 1); return; } @@ -166,21 +181,20 @@ final class DecimalSerializer extends Serializer { final payload = _decimalMagnitudeToCanonicalLittleEndian(unscaled.abs()); final sign = unscaled.isNegative ? 1 : 0; final meta = (payload.length << 1) | sign; - buffer.writeVarUint64((meta << 1) | 1); + buffer.writeVarUint64(Uint64((meta << 1) | 1)); buffer.writeBytes(payload); } static Decimal readPayload(ReadContext context) { final scale = context.buffer.readVarInt32(); final header = context.buffer.readVarUint64(); - if ((header & 1) == 0) { + if ((header.low32 & 1) == 0) { final zigZag = header >>> 1; - final unscaled = (zigZag >>> 1) ^ -(zigZag & 1); - return Decimal(BigInt.from(unscaled), scale); + return Decimal(_zigZagDecodeInt64(zigZag).toBigInt(), scale); } final meta = header >>> 1; - final length = meta >>> 1; + final length = (meta >>> 1).toInt(); if (length <= 0) { throw StateError('Invalid decimal magnitude length $length.'); } @@ -194,7 +208,7 @@ final class DecimalSerializer extends Serializer { if (magnitude == BigInt.zero) { throw StateError('Big decimal encoding must not represent zero.'); } - final sign = meta & 1; + final sign = (meta & 1).toInt(); return Decimal(sign == 0 ? magnitude : -magnitude, scale); } } diff --git a/dart/packages/fory/lib/src/serializer/serializer_support.dart b/dart/packages/fory/lib/src/serializer/serializer_support.dart index 3ca7228bcb..32b1dccd74 100644 --- a/dart/packages/fory/lib/src/serializer/serializer_support.dart +++ b/dart/packages/fory/lib/src/serializer/serializer_support.dart @@ -28,6 +28,7 @@ import 'package:fory/src/serializer/serialization_field_info.dart'; import 'package:fory/src/types/float32.dart'; import 'package:fory/src/types/int16.dart'; import 'package:fory/src/types/int32.dart'; +import 'package:fory/src/types/int64.dart'; import 'package:fory/src/types/int8.dart'; import 'package:fory/src/types/uint16.dart'; import 'package:fory/src/types/uint32.dart'; @@ -64,6 +65,12 @@ Object convertPrimitiveFieldValue(Object value, FieldType fieldType) { case TypeIds.int32: case TypeIds.varInt32: return (value as Int32).value; + case TypeIds.int64: + case TypeIds.varInt64: + case TypeIds.taggedInt64: + return _declares64BitWrapper(fieldType) + ? value + : (value as Int64).toInt(); case TypeIds.uint8: return (value as Uint8).value; case TypeIds.uint16: @@ -74,7 +81,9 @@ Object convertPrimitiveFieldValue(Object value, FieldType fieldType) { case TypeIds.uint64: case TypeIds.varUint64: case TypeIds.taggedUint64: - return (value as Uint64).value; + return _declares64BitWrapper(fieldType) + ? value + : (value as Uint64).toInt(); default: return value; } @@ -85,6 +94,69 @@ Object convertPrimitiveFieldValue(Object value, FieldType fieldType) { return value; } +bool _declares64BitWrapper(FieldType fieldType) { + final declaredTypeName = fieldType.declaredTypeName; + if (declaredTypeName == null) { + return false; + } + return declaredTypeName == 'Int64' || + declaredTypeName.endsWith('.Int64') || + declaredTypeName == 'Uint64' || + declaredTypeName.endsWith('.Uint64'); +} + +Object convertResolvedPrimitiveValue( + Object value, + TypeInfo resolved, [ + FieldType? fieldType, +]) { + if (fieldType != null && + _declares64BitWrapper(fieldType) && + (resolved.typeId == TypeIds.int64 || + resolved.typeId == TypeIds.varInt64 || + resolved.typeId == TypeIds.taggedInt64 || + resolved.typeId == TypeIds.uint64 || + resolved.typeId == TypeIds.varUint64 || + resolved.typeId == TypeIds.taggedUint64)) { + return value; + } + if (resolved.type == int) { + switch (resolved.typeId) { + case TypeIds.int8: + return (value as Int8).value; + case TypeIds.int16: + return (value as Int16).value; + case TypeIds.int32: + case TypeIds.varInt32: + return (value as Int32).value; + case TypeIds.int64: + case TypeIds.varInt64: + case TypeIds.taggedInt64: + return (value as Int64).toInt(); + case TypeIds.uint8: + return (value as Uint8).value; + case TypeIds.uint16: + return (value as Uint16).value; + case TypeIds.uint32: + case TypeIds.varUint32: + return (value as Uint32).value; + case TypeIds.uint64: + case TypeIds.varUint64: + case TypeIds.taggedUint64: + return (value as Uint64).toInt(); + default: + break; + } + } + if (resolved.type == double && resolved.typeId == TypeIds.float32) { + return (value as Float32).value; + } + if (fieldType != null) { + return convertPrimitiveFieldValue(value, fieldType); + } + return value; +} + void writeFieldValue( WriteContext context, SerializationFieldInfo field, @@ -284,6 +356,7 @@ FieldInfo mergeCompatibleWriteField( } return FieldType( type: local.type, + declaredTypeName: local.declaredTypeName, typeId: remote.typeId, nullable: remote.nullable, ref: remote.ref, @@ -319,6 +392,7 @@ FieldInfo mergeCompatibleReadField( } return FieldType( type: local.type, + declaredTypeName: local.declaredTypeName, typeId: remote.typeId, nullable: remote.nullable, ref: remote.ref, diff --git a/dart/packages/fory/lib/src/serializer/struct_serializer.dart b/dart/packages/fory/lib/src/serializer/struct_serializer.dart index f0b4df1c34..d7167b57dd 100644 --- a/dart/packages/fory/lib/src/serializer/struct_serializer.dart +++ b/dart/packages/fory/lib/src/serializer/struct_serializer.dart @@ -108,6 +108,7 @@ final class StructSerializer extends Serializer { internal.structWriteSlots = previousCompatibleFields; } + @pragma('vm:prefer-inline') Object readValue( ReadContext context, TypeInfo resolved, { diff --git a/dart/packages/fory/lib/src/serializer/time_serializers.dart b/dart/packages/fory/lib/src/serializer/time_serializers.dart index 985f1e57cd..6b607a455a 100644 --- a/dart/packages/fory/lib/src/serializer/time_serializers.dart +++ b/dart/packages/fory/lib/src/serializer/time_serializers.dart @@ -20,6 +20,7 @@ import 'package:fory/src/context/read_context.dart'; import 'package:fory/src/context/write_context.dart'; import 'package:fory/src/serializer/serializer.dart'; +import 'package:fory/src/types/int64.dart'; import 'package:fory/src/types/local_date.dart'; import 'package:fory/src/types/timestamp.dart'; @@ -60,12 +61,12 @@ void _validateDateTimeNanoseconds(int nanoseconds) { } } -int durationWireSeconds(Duration value) { +Int64 durationWireSeconds(Duration value) { final exact = _exactDurationWire[value]; if (exact != null && exact.microseconds == value.inMicroseconds) { - return exact.seconds; + return Int64(exact.seconds); } - return value.inMicroseconds ~/ Duration.microsecondsPerSecond; + return Int64(value.inMicroseconds ~/ Duration.microsecondsPerSecond); } int durationWireNanoseconds(Duration value) { @@ -76,19 +77,21 @@ int durationWireNanoseconds(Duration value) { return value.inMicroseconds.remainder(Duration.microsecondsPerSecond) * 1000; } -Duration durationFromWire(int seconds, int nanoseconds) { +Duration durationFromWire(Int64 seconds, int nanoseconds) { _validateDurationNanoseconds(nanoseconds); if ((seconds > 0 && nanoseconds < 0) || (seconds < 0 && nanoseconds > 0)) { throw StateError( 'Duration wire value has inconsistent signs: seconds=$seconds, nanoseconds=$nanoseconds.', ); } - final totalNanoseconds = seconds * _nanosecondsPerSecond + nanoseconds; - final microseconds = totalNanoseconds ~/ 1000; + final totalNanoseconds = + seconds.toBigInt() * BigInt.from(_nanosecondsPerSecond) + + BigInt.from(nanoseconds); + final microseconds = (totalNanoseconds ~/ BigInt.from(1000)).toInt(); final value = Duration(microseconds: microseconds); if (nanoseconds.remainder(1000) != 0) { _exactDurationWire[value] = _ExactDurationWire( - seconds, + seconds.toInt(), nanoseconds, microseconds, ); @@ -101,7 +104,7 @@ int timestampWireNanoseconds(Timestamp value) { return value.nanoseconds; } -int dateTimeWireSeconds(DateTime value) { +Int64 dateTimeWireSeconds(DateTime value) { final utcValue = value.toUtc(); final microseconds = utcValue.microsecondsSinceEpoch; var seconds = microseconds ~/ Duration.microsecondsPerSecond; @@ -110,7 +113,7 @@ int dateTimeWireSeconds(DateTime value) { micros += Duration.microsecondsPerSecond; seconds -= 1; } - return seconds; + return Int64(seconds); } int dateTimeWireNanoseconds(DateTime value) { @@ -123,15 +126,18 @@ int dateTimeWireNanoseconds(DateTime value) { return micros * 1000; } -Timestamp timestampFromWire(int seconds, int nanoseconds) { +Timestamp timestampFromWire(Int64 seconds, int nanoseconds) { _validateTimestampNanoseconds(nanoseconds); return Timestamp(seconds, nanoseconds); } -DateTime dateTimeFromWire(int seconds, int nanoseconds) { +DateTime dateTimeFromWire(Int64 seconds, int nanoseconds) { _validateDateTimeNanoseconds(nanoseconds); + final microseconds = + seconds.toBigInt() * BigInt.from(Duration.microsecondsPerSecond) + + BigInt.from(nanoseconds ~/ 1000); return DateTime.fromMicrosecondsSinceEpoch( - seconds * Duration.microsecondsPerSecond + nanoseconds ~/ 1000, + microseconds.toInt(), isUtc: true, ); } diff --git a/dart/packages/fory/lib/src/serializer/typed_array_serializers.dart b/dart/packages/fory/lib/src/serializer/typed_array_serializers.dart index c439a86361..221941ba2e 100644 --- a/dart/packages/fory/lib/src/serializer/typed_array_serializers.dart +++ b/dart/packages/fory/lib/src/serializer/typed_array_serializers.dart @@ -17,7 +17,7 @@ * under the License. */ -import 'dart:typed_data'; +import 'dart:typed_data' as td; import 'package:fory/src/context/read_context.dart'; import 'package:fory/src/context/write_context.dart'; @@ -25,13 +25,19 @@ import 'package:fory/src/meta/type_ids.dart'; import 'package:fory/src/serializer/serializer.dart'; import 'package:fory/src/types/bfloat16.dart'; import 'package:fory/src/types/float16.dart'; +import 'package:fory/src/types/int64.dart'; +import 'package:fory/src/types/uint64.dart'; void writeTypedArrayBytes( WriteContext context, Object values, ) { final bytes = switch (values) { - TypedData typed => typed.buffer.asUint8List( + Int64List typed => typed.buffer.asUint8List( + typed.offsetInBytes, + typed.lengthInBytes, + ), + Uint64List typed => typed.buffer.asUint8List( typed.offsetInBytes, typed.lengthInBytes, ), @@ -43,6 +49,10 @@ void writeTypedArrayBytes( typed.offsetInBytes, typed.lengthInBytes, ), + td.TypedData typed => typed.buffer.asUint8List( + typed.offsetInBytes, + typed.lengthInBytes, + ), _ => throw ArgumentError.value( values, 'values', @@ -56,7 +66,7 @@ void writeTypedArrayBytes( T readTypedArrayBytes( ReadContext context, int elementSize, - T Function(Uint8List bytes) viewBuilder, + T Function(td.Uint8List bytes) viewBuilder, ) { final byteSize = context.buffer.readVarUint32(); if (byteSize % elementSize != 0) { @@ -66,7 +76,7 @@ T readTypedArrayBytes( } var bytes = context.buffer.readBytes(byteSize); if (bytes.offsetInBytes % elementSize != 0) { - bytes = Uint8List.fromList(bytes); + bytes = td.Uint8List.fromList(bytes); } return viewBuilder(bytes); } @@ -99,7 +109,7 @@ final class BoolArraySerializer extends Serializer> { final class TypedArraySerializer extends Serializer { final int typeId; final int elementSize; - final T Function(Uint8List bytes) viewBuilder; + final T Function(td.Uint8List bytes) viewBuilder; const TypedArraySerializer( this.typeId, @@ -113,7 +123,7 @@ final class TypedArraySerializer extends Serializer { @override void write(WriteContext context, T value) { if (typeId == TypeIds.int8Array) { - final bytes = value as Int8List; + final bytes = value as td.Int8List; context.buffer.writeVarUint32(bytes.length); context.buffer.writeBytes(bytes); return; @@ -125,27 +135,27 @@ final class TypedArraySerializer extends Serializer { T read(ReadContext context) { if (typeId == TypeIds.int8Array) { final size = context.buffer.readVarUint32(); - return Int8List.fromList(context.buffer.readBytes(size)) as T; + return td.Int8List.fromList(context.buffer.readBytes(size)) as T; } return readTypedArrayBytes(context, elementSize, viewBuilder); } } const BoolArraySerializer boolArraySerializer = BoolArraySerializer(); -const TypedArraySerializer int8ArraySerializer = - TypedArraySerializer( +const TypedArraySerializer int8ArraySerializer = + TypedArraySerializer( TypeIds.int8Array, 1, - Int8List.fromList, + td.Int8List.fromList, ); -const TypedArraySerializer int16ArraySerializer = - TypedArraySerializer( +const TypedArraySerializer int16ArraySerializer = + TypedArraySerializer( TypeIds.int16Array, 2, _asInt16List, ); -const TypedArraySerializer int32ArraySerializer = - TypedArraySerializer( +const TypedArraySerializer int32ArraySerializer = + TypedArraySerializer( TypeIds.int32Array, 4, _asInt32List, @@ -156,14 +166,14 @@ const TypedArraySerializer int64ArraySerializer = 8, _asInt64List, ); -const TypedArraySerializer uint16ArraySerializer = - TypedArraySerializer( +const TypedArraySerializer uint16ArraySerializer = + TypedArraySerializer( TypeIds.uint16Array, 2, _asUint16List, ); -const TypedArraySerializer uint32ArraySerializer = - TypedArraySerializer( +const TypedArraySerializer uint32ArraySerializer = + TypedArraySerializer( TypeIds.uint32Array, 4, _asUint32List, @@ -186,64 +196,63 @@ const TypedArraySerializer bfloat16ArraySerializer = 2, _asBfloat16List, ); -const TypedArraySerializer float32ArraySerializer = - TypedArraySerializer( +const TypedArraySerializer float32ArraySerializer = + TypedArraySerializer( TypeIds.float32Array, 4, _asFloat32List, ); -const TypedArraySerializer float64ArraySerializer = - TypedArraySerializer( +const TypedArraySerializer float64ArraySerializer = + TypedArraySerializer( TypeIds.float64Array, 8, _asFloat64List, ); -Int16List _asInt16List(Uint8List bytes) => bytes.buffer.asInt16List( +td.Int16List _asInt16List(td.Uint8List bytes) => bytes.buffer.asInt16List( bytes.offsetInBytes, bytes.lengthInBytes ~/ 2, ); -Int32List _asInt32List(Uint8List bytes) => bytes.buffer.asInt32List( +td.Int32List _asInt32List(td.Uint8List bytes) => bytes.buffer.asInt32List( bytes.offsetInBytes, bytes.lengthInBytes ~/ 4, ); -Int64List _asInt64List(Uint8List bytes) => bytes.buffer.asInt64List( - bytes.offsetInBytes, - bytes.lengthInBytes ~/ 8, - ); +Int64List _asInt64List(td.Uint8List bytes) => + Int64List.view(bytes.buffer, bytes.offsetInBytes, bytes.lengthInBytes ~/ 8); -Uint16List _asUint16List(Uint8List bytes) => bytes.buffer.asUint16List( +td.Uint16List _asUint16List(td.Uint8List bytes) => bytes.buffer.asUint16List( bytes.offsetInBytes, bytes.lengthInBytes ~/ 2, ); -Float16List _asFloat16List(Uint8List bytes) => Float16List.view( +Float16List _asFloat16List(td.Uint8List bytes) => Float16List.view( bytes.buffer, bytes.offsetInBytes, bytes.lengthInBytes ~/ 2); -Bfloat16List _asBfloat16List(Uint8List bytes) => Bfloat16List.view( +Bfloat16List _asBfloat16List(td.Uint8List bytes) => Bfloat16List.view( bytes.buffer, bytes.offsetInBytes, bytes.lengthInBytes ~/ 2, ); -Uint32List _asUint32List(Uint8List bytes) => bytes.buffer.asUint32List( +td.Uint32List _asUint32List(td.Uint8List bytes) => bytes.buffer.asUint32List( bytes.offsetInBytes, bytes.lengthInBytes ~/ 4, ); -Uint64List _asUint64List(Uint8List bytes) => bytes.buffer.asUint64List( +Uint64List _asUint64List(td.Uint8List bytes) => Uint64List.view( + bytes.buffer, bytes.offsetInBytes, bytes.lengthInBytes ~/ 8, ); -Float32List _asFloat32List(Uint8List bytes) => bytes.buffer.asFloat32List( +td.Float32List _asFloat32List(td.Uint8List bytes) => bytes.buffer.asFloat32List( bytes.offsetInBytes, bytes.lengthInBytes ~/ 4, ); -Float64List _asFloat64List(Uint8List bytes) => bytes.buffer.asFloat64List( +td.Float64List _asFloat64List(td.Uint8List bytes) => bytes.buffer.asFloat64List( bytes.offsetInBytes, bytes.lengthInBytes ~/ 8, ); diff --git a/dart/packages/fory/lib/src/types/bfloat16.dart b/dart/packages/fory/lib/src/types/bfloat16.dart index 66d152f852..c1267c7d42 100644 --- a/dart/packages/fory/lib/src/types/bfloat16.dart +++ b/dart/packages/fory/lib/src/types/bfloat16.dart @@ -20,6 +20,10 @@ import 'dart:collection'; import 'dart:typed_data'; +const int _bfloat16ImplicitMantissaBit = 0x0010000000000000; +const int _bfloat16RoundIncrement = 0x0000100000000000; +const int _bfloat16MantissaDivisor = 0x0000200000000000; + /// Brain floating-point wrapper used by the xlang type system. /// /// [Bfloat16] stores an IEEE 754 bfloat16 payload exactly. Constructing from a @@ -37,16 +41,17 @@ final class Bfloat16 implements Comparable { /// Converts [value] to the closest representable bfloat16 value. factory Bfloat16.fromDouble(double value) { final data = ByteData(8)..setFloat64(0, value, Endian.little); - final bits = data.getUint64(0, Endian.little); - final sign = (bits >> 63) & 0x1; - final exponent = (bits >> 52) & 0x7ff; - final mantissa = bits & 0x000fffffffffffff; + final low32 = data.getUint32(0, Endian.little); + final high32 = data.getUint32(4, Endian.little); + final sign = (high32 >>> 31) & 0x1; + final exponent = (high32 >>> 20) & 0x7ff; + final mantissa = ((high32 & 0x000fffff) * 0x100000000) + low32; if (exponent == 0x7ff) { if (mantissa == 0) { return Bfloat16.fromBits((sign << 15) | 0x7f80); } - var payload = (mantissa >> 45) & 0x7f; + var payload = (mantissa ~/ _bfloat16MantissaDivisor) & 0x7f; if (payload == 0) { payload = 0x40; } else { @@ -63,14 +68,15 @@ final class Bfloat16 implements Comparable { if (adjustedExponent < -7) { return Bfloat16.fromBits(sign << 15); } - final shifted = (mantissa | (1 << 52)) >> (46 - adjustedExponent); - final rounded = (shifted + 1) >> 1; + final shifted = (mantissa + _bfloat16ImplicitMantissaBit) ~/ + _pow2Int(46 - adjustedExponent); + final rounded = (shifted + 1) ~/ 2; return Bfloat16.fromBits((sign << 15) | rounded); } - var roundedMantissa = mantissa + 0x0000100000000000; + var roundedMantissa = mantissa + _bfloat16RoundIncrement; var roundedExponent = adjustedExponent; - if ((roundedMantissa & 0x0010000000000000) != 0) { + if (roundedMantissa >= _bfloat16ImplicitMantissaBit) { roundedMantissa = 0; roundedExponent += 1; if (roundedExponent >= 0xff) { @@ -78,7 +84,9 @@ final class Bfloat16 implements Comparable { } } return Bfloat16.fromBits( - (sign << 15) | (roundedExponent << 7) | (roundedMantissa >> 45), + (sign << 15) | + (roundedExponent << 7) | + (roundedMantissa ~/ _bfloat16MantissaDivisor), ); } @@ -193,6 +201,14 @@ final class Bfloat16 implements Comparable { String toString() => value.toString(); } +int _pow2Int(int exponent) { + var result = 1; + for (var index = 0; index < exponent; index += 1) { + result *= 2; + } + return result; +} + /// Fixed-length contiguous storage for [Bfloat16] values. /// /// [Bfloat16List] behaves like a typed-data-style list whose elements are diff --git a/dart/packages/fory/lib/src/types/float16.dart b/dart/packages/fory/lib/src/types/float16.dart index 7c3532a9ab..dcb514e76a 100644 --- a/dart/packages/fory/lib/src/types/float16.dart +++ b/dart/packages/fory/lib/src/types/float16.dart @@ -20,6 +20,10 @@ import 'dart:collection'; import 'dart:typed_data'; +const int _float16ImplicitMantissaBit = 0x0010000000000000; +const int _float16RoundIncrement = 0x0000020000000000; +const int _float16MantissaDivisor = 0x0000040000000000; + /// Half-precision floating-point wrapper used by the xlang type system. /// /// [Float16] stores an IEEE 754 binary16 payload exactly. Constructing from a @@ -41,10 +45,11 @@ final class Float16 implements Comparable { return const Float16.fromBits(0x7e00); } final data = ByteData(8)..setFloat64(0, value, Endian.little); - final bits = data.getUint64(0, Endian.little); - final sign = (bits >> 63) & 0x1; - final exponent = (bits >> 52) & 0x7ff; - final mantissa = bits & 0x000fffffffffffff; + final low32 = data.getUint32(0, Endian.little); + final high32 = data.getUint32(4, Endian.little); + final sign = (high32 >>> 31) & 0x1; + final exponent = (high32 >>> 20) & 0x7ff; + final mantissa = ((high32 & 0x000fffff) * 0x100000000) + low32; if (exponent == 0x7ff) { return Float16.fromBits((sign << 15) | 0x7c00); @@ -58,14 +63,15 @@ final class Float16 implements Comparable { if (adjustedExponent < -10) { return Float16.fromBits(sign << 15); } - final shifted = (mantissa | (1 << 52)) >> (43 - adjustedExponent); - final rounded = (shifted + 1) >> 1; + final shifted = (mantissa + _float16ImplicitMantissaBit) ~/ + _pow2Int(43 - adjustedExponent); + final rounded = (shifted + 1) ~/ 2; return Float16.fromBits((sign << 15) | rounded); } - var roundedMantissa = mantissa + 0x0000020000000000; + var roundedMantissa = mantissa + _float16RoundIncrement; var roundedExponent = adjustedExponent; - if ((roundedMantissa & 0x0010000000000000) != 0) { + if (roundedMantissa >= _float16ImplicitMantissaBit) { roundedMantissa = 0; roundedExponent += 1; if (roundedExponent >= 0x1f) { @@ -73,7 +79,9 @@ final class Float16 implements Comparable { } } return Float16.fromBits( - (sign << 15) | (roundedExponent << 10) | (roundedMantissa >> 42), + (sign << 15) | + (roundedExponent << 10) | + (roundedMantissa ~/ _float16MantissaDivisor), ); } @@ -207,6 +215,14 @@ final class Float16 implements Comparable { String toString() => value.toString(); } +int _pow2Int(int exponent) { + var result = 1; + for (var index = 0; index < exponent; index += 1) { + result *= 2; + } + return result; +} + /// Fixed-length contiguous storage for [Float16] values. /// /// [Float16List] behaves like a typed-data-style list whose elements are diff --git a/dart/packages/fory/lib/src/types/int64.dart b/dart/packages/fory/lib/src/types/int64.dart new file mode 100644 index 0000000000..08b3aab19b --- /dev/null +++ b/dart/packages/fory/lib/src/types/int64.dart @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +export 'int64_native.dart' + if (dart.library.js_interop) 'int64_web.dart'; diff --git a/dart/packages/fory/lib/src/types/int64_native.dart b/dart/packages/fory/lib/src/types/int64_native.dart new file mode 100644 index 0000000000..1c7f148c62 --- /dev/null +++ b/dart/packages/fory/lib/src/types/int64_native.dart @@ -0,0 +1,262 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import 'dart:typed_data' as td; + +/// Signed 64-bit integer wrapper used by the xlang type system. +extension type Int64._(int _value) implements int { + /// Creates a signed 64-bit value by truncating [value] to 64 bits. + factory Int64(int value) => Int64._(value.toSigned(64)); + + /// Creates a signed 64-bit value by truncating [value] to 64 bits. + factory Int64.fromBigInt(BigInt value) => Int64._(_normalizeSigned64(value)); + + /// Creates a signed 64-bit value from little-endian 32-bit words. + factory Int64.fromWords(int low32, int high32) => Int64._( + ((high32 & 0xffffffff) << 32 | (low32 & 0xffffffff)).toSigned(64)); + + /// Parses a hexadecimal two's-complement payload. + factory Int64.parseHex(String value) => + Int64.fromBigInt(BigInt.parse(value, radix: 16)); + + /// The normalized signed 64-bit value on native platforms. + int get value => _value; + + /// Returns whether this value is negative. + bool get isNegative => _value < 0; + + /// Returns whether this value is zero. + bool get isZero => _value == 0; + + /// Returns the low 32 bits as an unsigned integer. + int get low32 => _value & 0xffffffff; + + /// Returns the high 32 bits as an unsigned integer. + int get high32Unsigned => (_value >> 32) & 0xffffffff; + + /// Returns the high 32 bits as a signed integer. + int get high32Signed => _value >> 32; + + /// Returns the exact value as a [BigInt]. + BigInt toBigInt() => BigInt.from(_value); + + /// Returns the exact native [int] value. + int toInt() => _value; + + int compareTo(Int64 other) => _value.compareTo(other._value); + + Int64 operator +(Object other) => switch (other) { + int otherValue => Int64(_value + otherValue), + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + Int64 operator -(Object other) => switch (other) { + int otherValue => Int64(_value - otherValue), + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + Int64 operator *(Object other) => switch (other) { + int otherValue => Int64(_value * otherValue), + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + Int64 operator ~/(Object other) => switch (other) { + int otherValue => Int64(_value ~/ otherValue), + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + Int64 operator %(Object other) => switch (other) { + int otherValue => Int64(_value % otherValue), + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + double operator /(Object other) => switch (other) { + int otherValue => _value / otherValue, + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + Int64 operator -() => Int64(-_value); + + Int64 operator ~() => Int64(~_value); + + Int64 operator &(Object other) => switch (other) { + int otherValue => Int64(_value & otherValue), + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + Int64 operator |(Object other) => switch (other) { + int otherValue => Int64(_value | otherValue), + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + Int64 operator ^(Object other) => switch (other) { + int otherValue => Int64(_value ^ otherValue), + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + Int64 operator <<(int shift) => Int64(_value << shift); + + Int64 operator >>(int shift) => Int64(_value >> shift); + + Int64 operator >>>(int shift) => Int64(_value >>> shift); + + bool operator <(Object other) => switch (other) { + int otherValue => _value < otherValue, + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + bool operator <=(Object other) => switch (other) { + int otherValue => _value <= otherValue, + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + bool operator >(Object other) => switch (other) { + int otherValue => _value > otherValue, + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; + + bool operator >=(Object other) => switch (other) { + int otherValue => _value >= otherValue, + _ => throw ArgumentError.value( + other, + 'other', + 'Expected an int or Int64.', + ), + }; +} + +/// Fixed-length contiguous storage for [Int64] values on native platforms. +extension type Int64List._(td.Int64List _storage) implements td.Int64List { + /// The number of bytes used by one [Int64] element. + static const int bytesPerElement = td.Int64List.bytesPerElement; + + /// Creates a zero-initialized list with [length] signed 64-bit elements. + Int64List(int length) : _storage = td.Int64List(length); + + /// Copies [values] into a new contiguous signed 64-bit list. + factory Int64List.fromList(Iterable values) { + final copied = values.toList(growable: false); + final storage = td.Int64List(copied.length); + for (var index = 0; index < copied.length; index += 1) { + storage[index] = switch (copied[index]) { + int value => Int64(value).value, + _ => throw ArgumentError.value( + copied[index], + 'values[$index]', + 'Expected an int or Int64.', + ), + }; + } + return Int64List._(storage); + } + + /// Creates a zero-copy view over [buffer]. + factory Int64List.view( + td.ByteBuffer buffer, [ + int offsetInBytes = 0, + int? length, + ]) { + return Int64List._(td.Int64List.view(buffer, offsetInBytes, length)); + } + + /// Creates a zero-copy element-range view of [data]. + factory Int64List.sublistView(td.TypedData data, [int start = 0, int? end]) { + return Int64List._(td.Int64List.sublistView(data, start, end)); + } + + /// The number of elements in this list. + int get length => _storage.length; + + /// Returns the element at [index]. + Int64 operator [](int index) => Int64(_storage[index]); + + /// Stores [value] at [index]. + void operator []=(int index, Int64 value) { + _storage[index] = value.value; + } + + td.ByteBuffer get buffer => _storage.buffer; + + int get elementSizeInBytes => _storage.elementSizeInBytes; + + int get offsetInBytes => _storage.offsetInBytes; + + int get lengthInBytes => _storage.lengthInBytes; + + Iterator get iterator => + Iterable.generate(length, (index) => this[index]).iterator; +} + +int _normalizeSigned64(BigInt value) { + final mask64 = (BigInt.one << 64) - BigInt.one; + final signBit64 = BigInt.one << 63; + var normalized = value & mask64; + if ((normalized & signBit64) != BigInt.zero) { + normalized -= BigInt.one << 64; + } + return normalized.toInt(); +} diff --git a/dart/packages/fory/lib/src/types/int64_web.dart b/dart/packages/fory/lib/src/types/int64_web.dart new file mode 100644 index 0000000000..ecb53c9934 --- /dev/null +++ b/dart/packages/fory/lib/src/types/int64_web.dart @@ -0,0 +1,515 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import 'dart:collection'; +import 'dart:typed_data' as td; + +const int _int64SegmentBits = 22; +const int _int64SegmentMask = (1 << _int64SegmentBits) - 1; +const int _int64SegmentBase = 1 << _int64SegmentBits; +const int _int64HighBits = 20; +const int _int64HighMask = (1 << _int64HighBits) - 1; +const int _int64HighBase = 1 << _int64HighBits; +const int _int64HighSignBit = 1 << (_int64HighBits - 1); +const int _int64SafeHighMax = 0x1ff; +const int _int64SafeHighMin = -0x200; +const int _int64HighValueBase = 17592186044416; +const int _int64WordBase = 0x100000000; +final BigInt _int64Mask64Big = (BigInt.one << 64) - BigInt.one; +final BigInt _int64Mask32Big = (BigInt.one << 32) - BigInt.one; +final BigInt _int64SignBit64Big = BigInt.one << 63; + +/// Signed 64-bit integer wrapper used by the xlang type system on web builds. +final class Int64 implements Comparable { + final int _l; + final int _m; + final int _h; + + const Int64._parts(this._l, this._m, this._h); + + /// Creates a signed 64-bit value by truncating [value] to 64 bits. + factory Int64(int value) { + final low32 = value.toUnsigned(32); + final high32 = (value - low32) ~/ _int64WordBase; + return Int64.fromWords(low32, high32); + } + + /// Creates a signed 64-bit value by truncating [value] to 64 bits. + factory Int64.fromBigInt(BigInt value) { + var normalized = value & _int64Mask64Big; + if ((normalized & _int64SignBit64Big) != BigInt.zero) { + normalized -= BigInt.one << 64; + } + final unsigned = + normalized.isNegative ? normalized + (BigInt.one << 64) : normalized; + return Int64.fromWords( + (unsigned & _int64Mask32Big).toInt(), + ((unsigned >> 32) & _int64Mask32Big).toInt(), + ); + } + + /// Creates a signed 64-bit value from little-endian 32-bit words. + factory Int64.fromWords(int low32, int high32) { + final normalizedLow32 = low32.toUnsigned(32); + final normalizedHigh32 = high32.toSigned(32); + return Int64._parts( + normalizedLow32 & _int64SegmentMask, + ((normalizedLow32 >>> _int64SegmentBits) | + ((normalizedHigh32 & 0xfff) << 10)) & + _int64SegmentMask, + _normalizeInt64High(normalizedHigh32 >> 12), + ); + } + + /// Parses a hexadecimal two's-complement payload. + factory Int64.parseHex(String value) => + Int64.fromBigInt(BigInt.parse(value, radix: 16)); + + /// Returns whether this value is negative. + bool get isNegative => _h < 0; + + /// Returns whether this value is zero. + bool get isZero => _l == 0 && _m == 0 && _h == 0; + + /// Returns the low 32 bits as an unsigned integer. + int get low32 => _l | ((_m & 0x3ff) << 22); + + /// Returns the high 32 bits as an unsigned integer. + int get high32Unsigned => + ((_m >>> 10) & 0xfff) | ((_h & _int64HighMask) << 12); + + /// Returns the high 32 bits as a signed integer. + int get high32Signed => high32Unsigned.toSigned(32); + + /// Returns the exact value as a [BigInt]. + BigInt toBigInt() { + final unsigned = (BigInt.from(high32Unsigned) << 32) | BigInt.from(low32); + return isNegative ? unsigned - (BigInt.one << 64) : unsigned; + } + + /// Returns this value as a Dart [int] when it is JS-safe. + int toInt() { + if (_h > _int64SafeHighMax || + _h < _int64SafeHighMin || + (_h == _int64SafeHighMin && _m == 0 && _l == 0)) { + throw StateError('Int64 value ${toBigInt()} is not a JS-safe int.'); + } + return _h * _int64HighValueBase + _m * _int64SegmentBase + _l; + } + + @override + int compareTo(Int64 other) { + if (_h != other._h) { + return _h < other._h ? -1 : 1; + } + if (_m != other._m) { + return _m < other._m ? -1 : 1; + } + if (_l != other._l) { + return _l < other._l ? -1 : 1; + } + return 0; + } + + Int64 operator +(Object other) => switch (other) { + int otherValue => _add(Int64(otherValue)), + Int64 otherValue => _add(otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + Int64 operator -(Object other) => switch (other) { + int otherValue => _subtract(Int64(otherValue)), + Int64 otherValue => _subtract(otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + Int64 operator *(Object other) => switch (other) { + int otherValue => + Int64.fromBigInt(toBigInt() * BigInt.from(otherValue)), + Int64 otherValue => + Int64.fromBigInt(toBigInt() * otherValue.toBigInt()), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + Int64 operator ~/(Object other) => switch (other) { + int otherValue => + Int64.fromBigInt(toBigInt() ~/ BigInt.from(otherValue)), + Int64 otherValue => + Int64.fromBigInt(toBigInt() ~/ otherValue.toBigInt()), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + Int64 operator %(Object other) => switch (other) { + int otherValue => + Int64.fromBigInt(toBigInt() % BigInt.from(otherValue)), + Int64 otherValue => + Int64.fromBigInt(toBigInt() % otherValue.toBigInt()), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + double operator /(Object other) => switch (other) { + int otherValue => toBigInt().toDouble() / otherValue, + Int64 otherValue => + toBigInt().toDouble() / otherValue.toBigInt().toDouble(), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + Int64 operator -() => Int64(0) - this; + + Int64 operator ~() => Int64._parts( + _l ^ _int64SegmentMask, _m ^ _int64SegmentMask, _normalizeInt64High(~_h)); + + Int64 operator &(Object other) => switch (other) { + int otherValue => _bitwiseAnd(Int64(otherValue)), + Int64 otherValue => _bitwiseAnd(otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + Int64 operator |(Object other) => switch (other) { + int otherValue => _bitwiseOr(Int64(otherValue)), + Int64 otherValue => _bitwiseOr(otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + Int64 operator ^(Object other) => switch (other) { + int otherValue => _bitwiseXor(Int64(otherValue)), + Int64 otherValue => _bitwiseXor(otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + Int64 operator <<(int shift) { + RangeError.checkNotNegative(shift, 'shift'); + if (shift == 0) { + return this; + } + if (shift >= 64) { + return Int64(0); + } + final highBits = _h & _int64HighMask; + if (shift < 22) { + return Int64._parts( + (_l << shift) & _int64SegmentMask, + ((_m << shift) | (_l >>> (22 - shift))) & _int64SegmentMask, + _normalizeInt64High((highBits << shift) | (_m >>> (22 - shift))), + ); + } + if (shift < 44) { + final segmentShift = shift - 22; + return Int64._parts( + 0, + (_l << segmentShift) & _int64SegmentMask, + _normalizeInt64High( + (_m << segmentShift) | (_l >>> (22 - segmentShift)), + ), + ); + } + final segmentShift = shift - 44; + return Int64._parts( + 0, + 0, + _normalizeInt64High(_l << segmentShift), + ); + } + + Int64 operator >>(int shift) { + RangeError.checkNotNegative(shift, 'shift'); + if (shift == 0) { + return this; + } + if (shift >= 64) { + return _h < 0 ? Int64(-1) : Int64(0); + } + if (shift < 22) { + return Int64._parts( + ((_l >>> shift) | ((_m & _int64LowBitsMask(shift)) << (22 - shift))) & + _int64SegmentMask, + ((_m >>> shift) | ((_h & _int64LowBitsMask(shift)) << (22 - shift))) & + _int64SegmentMask, + _normalizeInt64High(_h >> shift), + ); + } + if (shift < 44) { + final segmentShift = shift - 22; + return Int64._parts( + ((_m >>> segmentShift) | + ((_h & _int64LowBitsMask(segmentShift)) << + (22 - segmentShift))) & + _int64SegmentMask, + (_h >> segmentShift) & _int64SegmentMask, + _h < 0 ? -1 : 0, + ); + } + final segmentShift = shift - 44; + return Int64._parts( + (_h >> segmentShift) & _int64SegmentMask, + _h < 0 ? _int64SegmentMask : 0, + _h < 0 ? -1 : 0, + ); + } + + Int64 operator >>>(int shift) { + RangeError.checkNotNegative(shift, 'shift'); + if (shift == 0) { + return this; + } + if (shift >= 64) { + return Int64(0); + } + return _logicalShiftRight(shift); + } + + bool operator <(Object other) => switch (other) { + int otherValue => compareTo(Int64(otherValue)) < 0, + Int64 otherValue => compareTo(otherValue) < 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + bool operator <=(Object other) => switch (other) { + int otherValue => compareTo(Int64(otherValue)) <= 0, + Int64 otherValue => compareTo(otherValue) <= 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + bool operator >(Object other) => switch (other) { + int otherValue => compareTo(Int64(otherValue)) > 0, + Int64 otherValue => compareTo(otherValue) > 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + bool operator >=(Object other) => switch (other) { + int otherValue => compareTo(Int64(otherValue)) >= 0, + Int64 otherValue => compareTo(otherValue) >= 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Int64.'), + }; + + @override + bool operator ==(Object other) => + identical(this, other) || + other is Int64 && other._l == _l && other._m == _m && other._h == _h; + + @override + int get hashCode => Object.hash(low32, high32Signed); + + @override + String toString() => toBigInt().toString(); + + Int64 _add(Int64 other) { + var low = _l + other._l; + var carry = low >> 22; + low &= _int64SegmentMask; + var mid = _m + other._m + carry; + carry = mid >> 22; + mid &= _int64SegmentMask; + return Int64._parts(low, mid, _normalizeInt64High(_h + other._h + carry)); + } + + Int64 _subtract(Int64 other) { + var low = _l - other._l; + var borrow = 0; + if (low < 0) { + low += _int64SegmentBase; + borrow = 1; + } + var mid = _m - other._m - borrow; + borrow = 0; + if (mid < 0) { + mid += _int64SegmentBase; + borrow = 1; + } + return Int64._parts( + low, + mid, + _normalizeInt64High(_h - other._h - borrow), + ); + } + + Int64 _bitwiseAnd(Int64 other) => Int64._parts( + _l & other._l, + _m & other._m, + _normalizeInt64High(_h & other._h), + ); + + Int64 _bitwiseOr(Int64 other) => Int64._parts( + _l | other._l, + _m | other._m, + _normalizeInt64High(_h | other._h), + ); + + Int64 _bitwiseXor(Int64 other) => Int64._parts( + _l ^ other._l, + _m ^ other._m, + _normalizeInt64High(_h ^ other._h), + ); + + Int64 _logicalShiftRight(int shift) { + final highBits = _h & _int64HighMask; + if (shift < 22) { + return Int64._parts( + ((_l >>> shift) | ((_m & _int64LowBitsMask(shift)) << (22 - shift))) & + _int64SegmentMask, + ((_m >>> shift) | + ((highBits & _int64LowBitsMask(shift)) << (22 - shift))) & + _int64SegmentMask, + highBits >>> shift, + ); + } + if (shift < 44) { + final segmentShift = shift - 22; + return Int64._parts( + ((_m >>> segmentShift) | + ((highBits & _int64LowBitsMask(segmentShift)) << + (22 - segmentShift))) & + _int64SegmentMask, + highBits >>> segmentShift, + 0, + ); + } + final segmentShift = shift - 44; + return Int64._parts(highBits >>> segmentShift, 0, 0); + } +} + +/// Fixed-length contiguous storage for [Int64] values on web builds. +final class Int64List extends IterableBase { + /// The number of bytes used by one [Int64] element. + static const int bytesPerElement = 8; + + final td.Uint8List _bytes; + late final td.ByteData _view = td.ByteData.sublistView(_bytes); + + /// Creates a zero-initialized list with [length] signed 64-bit elements. + Int64List(int length) : _bytes = td.Uint8List(length * bytesPerElement); + + Int64List._(this._bytes); + + /// Copies [values] into a new contiguous signed 64-bit list. + factory Int64List.fromList(Iterable values) { + final copied = values.toList(growable: false); + final result = Int64List(copied.length); + for (var index = 0; index < copied.length; index += 1) { + result[index] = switch (copied[index]) { + int value => Int64(value), + Int64 value => value, + _ => throw ArgumentError.value( + copied[index], + 'values[$index]', + 'Expected an int or Int64.', + ), + }; + } + return result; + } + + /// Creates a zero-copy view over [buffer]. + factory Int64List.view( + td.ByteBuffer buffer, [ + int offsetInBytes = 0, + int? length, + ]) { + final byteLength = length == null + ? buffer.lengthInBytes - offsetInBytes + : length * bytesPerElement; + return Int64List._(td.Uint8List.view(buffer, offsetInBytes, byteLength)); + } + + /// Creates a zero-copy element-range view of [data]. + factory Int64List.sublistView(td.TypedData data, [int start = 0, int? end]) { + final elementCount = data.lengthInBytes ~/ data.elementSizeInBytes; + final endIndex = RangeError.checkValidRange(start, end, elementCount); + final byteOffset = data.offsetInBytes + start * data.elementSizeInBytes; + final byteLength = (endIndex - start) * data.elementSizeInBytes; + if (byteOffset % bytesPerElement != 0 || + byteLength % bytesPerElement != 0) { + throw ArgumentError( + 'The selected byte range must align to $bytesPerElement-byte ' + 'Int64 elements.', + ); + } + return Int64List.view( + data.buffer, + byteOffset, + byteLength ~/ bytesPerElement, + ); + } + + /// The number of elements in this list. + @override + int get length => _bytes.lengthInBytes ~/ bytesPerElement; + + /// Returns the element at [index]. + Int64 operator [](int index) { + RangeError.checkValidIndex(index, this, 'index', length); + final byteOffset = index * bytesPerElement; + return Int64.fromWords( + _view.getUint32(byteOffset, td.Endian.little), + _view.getInt32(byteOffset + 4, td.Endian.little), + ); + } + + /// Stores [value] at [index]. + void operator []=(int index, Int64 value) { + RangeError.checkValidIndex(index, this, 'index', length); + final byteOffset = index * bytesPerElement; + _view.setUint32(byteOffset, value.low32, td.Endian.little); + _view.setUint32(byteOffset + 4, value.high32Unsigned, td.Endian.little); + } + + td.ByteBuffer get buffer => _bytes.buffer; + + int get elementSizeInBytes => bytesPerElement; + + int get offsetInBytes => _bytes.offsetInBytes; + + int get lengthInBytes => _bytes.lengthInBytes; + + @override + Iterator get iterator => + Iterable.generate(length, (index) => this[index]).iterator; + + @override + String toString() => toList().toString(); +} + +int _normalizeInt64High(int value) { + final normalized = value & _int64HighMask; + if ((normalized & _int64HighSignBit) != 0) { + return normalized - _int64HighBase; + } + return normalized; +} + +int _int64LowBitsMask(int bits) { + if (bits <= 0) { + return 0; + } + return (1 << bits) - 1; +} diff --git a/dart/packages/fory/lib/src/types/local_date.dart b/dart/packages/fory/lib/src/types/local_date.dart index acd72fc3c4..6a1c31a322 100644 --- a/dart/packages/fory/lib/src/types/local_date.dart +++ b/dart/packages/fory/lib/src/types/local_date.dart @@ -17,6 +17,8 @@ * under the License. */ +import 'package:fory/src/types/int64.dart'; + /// Calendar date without time-of-day or time-zone information. final class LocalDate implements Comparable { /// Year component. @@ -32,9 +34,9 @@ final class LocalDate implements Comparable { const LocalDate(this.year, this.month, this.day); /// Creates a date from the xlang epoch-day representation. - factory LocalDate.fromEpochDay(int epochDay) { + factory LocalDate.fromEpochDay(Int64 epochDay) { final instant = DateTime.fromMillisecondsSinceEpoch( - epochDay * Duration.millisecondsPerDay, + epochDay.toInt() * Duration.millisecondsPerDay, isUtc: true, ); return LocalDate(instant.year, instant.month, instant.day); @@ -47,9 +49,10 @@ final class LocalDate implements Comparable { } /// Converts this date to xlang epoch-day form. - int toEpochDay() => - DateTime.utc(year, month, day).millisecondsSinceEpoch ~/ - Duration.millisecondsPerDay; + Int64 toEpochDay() => Int64( + DateTime.utc(year, month, day).millisecondsSinceEpoch ~/ + Duration.millisecondsPerDay, + ); /// Converts this date to a UTC [DateTime] at midnight. DateTime toDateTime() => DateTime.utc(year, month, day); diff --git a/dart/packages/fory/lib/src/types/timestamp.dart b/dart/packages/fory/lib/src/types/timestamp.dart index 060b529f29..08cc4f8275 100644 --- a/dart/packages/fory/lib/src/types/timestamp.dart +++ b/dart/packages/fory/lib/src/types/timestamp.dart @@ -17,16 +17,18 @@ * under the License. */ +import 'package:fory/src/types/int64.dart'; + /// Timestamp with second and nanosecond precision in UTC. final class Timestamp implements Comparable { /// Whole seconds since the Unix epoch. - final int seconds; + final Int64 seconds; /// Nanoseconds within the second. final int nanoseconds; /// Creates a timestamp from epoch seconds and nanoseconds. - const Timestamp(this.seconds, this.nanoseconds); + Timestamp(this.seconds, this.nanoseconds); /// Converts a [DateTime] to UTC timestamp components. factory Timestamp.fromDateTime(DateTime value) { @@ -38,12 +40,14 @@ final class Timestamp implements Comparable { micros += Duration.microsecondsPerSecond; seconds -= 1; } - return Timestamp(seconds, micros * 1000); + return Timestamp(Int64(seconds), micros * 1000); } /// Converts this timestamp to a UTC [DateTime]. DateTime toDateTime() => DateTime.fromMicrosecondsSinceEpoch( - seconds * Duration.microsecondsPerSecond + nanoseconds ~/ 1000, + (seconds.toBigInt() * BigInt.from(Duration.microsecondsPerSecond) + + BigInt.from(nanoseconds ~/ 1000)) + .toInt(), isUtc: true, ); diff --git a/dart/packages/fory/lib/src/types/uint64.dart b/dart/packages/fory/lib/src/types/uint64.dart index 1c335e84f2..b3a9881c5d 100644 --- a/dart/packages/fory/lib/src/types/uint64.dart +++ b/dart/packages/fory/lib/src/types/uint64.dart @@ -17,131 +17,5 @@ * under the License. */ -/// Unsigned 64-bit integer wrapper used by the xlang type system. -/// -/// The constructor truncates [value] to 64 bits using unsigned wrap-around -/// semantics. This wrapper is the default Dart carrier for root and untyped -/// xlang `uint64`, `varuint64`, and `taggeduint64` payloads; declared `int` -/// fields annotated with `@Uint64Type(...)` still read back as plain Dart -/// [int] values through generated conversion paths. -final class Uint64 implements Comparable { - /// The normalized unsigned 64-bit value. - final int value; - - /// Creates an unsigned 64-bit value by truncating [value] to 64 bits. - Uint64(int value) : value = value.toUnsigned(64); - - @override - int compareTo(Uint64 other) => value.compareTo(other.value); - - Uint64 operator +(Object other) => switch (other) { - int otherValue => Uint64(value + otherValue), - Uint64 otherValue => Uint64(value + otherValue.value), - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - Uint64 operator -(Object other) => switch (other) { - int otherValue => Uint64(value - otherValue), - Uint64 otherValue => Uint64(value - otherValue.value), - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - Uint64 operator *(Object other) => switch (other) { - int otherValue => Uint64(value * otherValue), - Uint64 otherValue => Uint64(value * otherValue.value), - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - Uint64 operator ~/(Object other) => switch (other) { - int otherValue => Uint64(value ~/ otherValue), - Uint64 otherValue => Uint64(value ~/ otherValue.value), - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - Uint64 operator %(Object other) => switch (other) { - int otherValue => Uint64(value % otherValue), - Uint64 otherValue => Uint64(value % otherValue.value), - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - double operator /(Object other) => switch (other) { - int otherValue => value / otherValue, - Uint64 otherValue => value / otherValue.value, - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - Uint64 operator -() => Uint64(-value); - - Uint64 operator ~() => Uint64(~value); - - Uint64 operator &(Object other) => switch (other) { - int otherValue => Uint64(value & otherValue), - Uint64 otherValue => Uint64(value & otherValue.value), - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - Uint64 operator |(Object other) => switch (other) { - int otherValue => Uint64(value | otherValue), - Uint64 otherValue => Uint64(value | otherValue.value), - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - Uint64 operator ^(Object other) => switch (other) { - int otherValue => Uint64(value ^ otherValue), - Uint64 otherValue => Uint64(value ^ otherValue.value), - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - Uint64 operator <<(int shift) => Uint64(value << shift); - - Uint64 operator >>(int shift) => Uint64(value >> shift); - - bool operator <(Object other) => switch (other) { - int otherValue => value < otherValue, - Uint64 otherValue => value < otherValue.value, - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - bool operator <=(Object other) => switch (other) { - int otherValue => value <= otherValue, - Uint64 otherValue => value <= otherValue.value, - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - bool operator >(Object other) => switch (other) { - int otherValue => value > otherValue, - Uint64 otherValue => value > otherValue.value, - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - bool operator >=(Object other) => switch (other) { - int otherValue => value >= otherValue, - Uint64 otherValue => value >= otherValue.value, - _ => throw ArgumentError.value( - other, 'other', 'Expected an int or Uint64.'), - }; - - int toInt() => value; - - @override - bool operator ==(Object other) => - identical(this, other) || other is Uint64 && other.value == value; - - @override - int get hashCode => value.hashCode; - - @override - String toString() => value.toString(); -} +export 'uint64_native.dart' + if (dart.library.js_interop) 'uint64_web.dart'; diff --git a/dart/packages/fory/lib/src/types/uint64_native.dart b/dart/packages/fory/lib/src/types/uint64_native.dart new file mode 100644 index 0000000000..09d2da11f2 --- /dev/null +++ b/dart/packages/fory/lib/src/types/uint64_native.dart @@ -0,0 +1,243 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import 'dart:typed_data' as td; + +final BigInt _uint64Mask64Big = (BigInt.one << 64) - BigInt.one; +final BigInt _uint64SignBit64Big = BigInt.one << 63; + +/// Unsigned 64-bit integer wrapper used by the xlang type system. +extension type Uint64._(int _value) implements int { + /// Creates an unsigned 64-bit value by truncating [value] to 64 bits. + factory Uint64(int value) => Uint64._(value.toSigned(64)); + + /// Creates an unsigned 64-bit value by truncating [value] to 64 bits. + factory Uint64.fromBigInt(BigInt value) => + Uint64._(_normalizeUnsigned64(value)); + + /// Creates an unsigned 64-bit value from little-endian 32-bit words. + factory Uint64.fromWords(int low32, int high32) { + final normalizedLow32 = low32 & 0xffffffff; + final normalizedHigh32 = high32 & 0xffffffff; + return Uint64._( + (((normalizedHigh32 << 32) | normalizedLow32).toSigned(64)), + ); + } + + /// Parses a hexadecimal payload. + factory Uint64.parseHex(String value) => + Uint64.fromBigInt(BigInt.parse(value, radix: 16)); + + /// The normalized 64-bit bit pattern stored in a native [int]. + int get value => _value; + + /// Returns whether this value is zero. + bool get isZero => _value == 0; + + /// Returns the low 32 bits as an unsigned integer. + int get low32 => _value & 0xffffffff; + + /// Returns the high 32 bits as an unsigned integer. + int get high32Unsigned => (_value >> 32) & 0xffffffff; + + /// Returns the exact value as a [BigInt]. + BigInt toBigInt() => _value >= 0 + ? BigInt.from(_value) + : BigInt.from(_value) + (BigInt.one << 64); + + /// Returns this value as a native [int] when it is exactly representable. + int toInt() { + if (_value < 0) { + final exact = toBigInt(); + throw StateError( + 'Uint64 value $exact is not representable as a native int.', + ); + } + return _value; + } + + int compareTo(Uint64 other) { + final highCompare = high32Unsigned.compareTo(other.high32Unsigned); + if (highCompare != 0) { + return highCompare; + } + return low32.compareTo(other.low32); + } + + Uint64 operator +(Object other) => switch (other) { + int otherValue => Uint64(_value + otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator -(Object other) => switch (other) { + int otherValue => Uint64(_value - otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator *(Object other) => switch (other) { + int otherValue => Uint64(_value * Uint64(otherValue).value), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator ~/(Object other) => switch (other) { + int otherValue => + Uint64.fromBigInt(toBigInt() ~/ Uint64(otherValue).toBigInt()), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator %(Object other) => switch (other) { + int otherValue => Uint64.fromBigInt( + toBigInt().remainder(Uint64(otherValue).toBigInt())), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + double operator /(Object other) => switch (other) { + int otherValue => + toBigInt().toDouble() / Uint64(otherValue).toBigInt().toDouble(), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator -() => Uint64(-_value); + + Uint64 operator ~() => Uint64(~_value); + + Uint64 operator &(Object other) => switch (other) { + int otherValue => Uint64(_value & otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator |(Object other) => switch (other) { + int otherValue => Uint64(_value | otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator ^(Object other) => switch (other) { + int otherValue => Uint64(_value ^ otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator <<(int shift) => Uint64(_value << shift); + + Uint64 operator >>(int shift) => Uint64(_value >>> shift); + + Uint64 operator >>>(int shift) => Uint64(_value >>> shift); + + bool operator <(Object other) => switch (other) { + int otherValue => compareTo(Uint64(otherValue)) < 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + bool operator <=(Object other) => switch (other) { + int otherValue => compareTo(Uint64(otherValue)) <= 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + bool operator >(Object other) => switch (other) { + int otherValue => compareTo(Uint64(otherValue)) > 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + bool operator >=(Object other) => switch (other) { + int otherValue => compareTo(Uint64(otherValue)) >= 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; +} + +/// Fixed-length contiguous storage for [Uint64] values on native platforms. +extension type Uint64List._(td.Uint64List _storage) implements td.Uint64List { + /// The number of bytes used by one [Uint64] element. + static const int bytesPerElement = td.Uint64List.bytesPerElement; + + /// Creates a zero-initialized list with [length] unsigned 64-bit elements. + Uint64List(int length) : _storage = td.Uint64List(length); + + /// Copies [values] into a new contiguous unsigned 64-bit list. + factory Uint64List.fromList(Iterable values) { + final copied = values.toList(growable: false); + final storage = td.Uint64List(copied.length); + for (var index = 0; index < copied.length; index += 1) { + storage[index] = switch (copied[index]) { + int value => Uint64(value).value, + _ => throw ArgumentError.value( + copied[index], + 'values[$index]', + 'Expected an int or Uint64.', + ), + }; + } + return Uint64List._(storage); + } + + /// Creates a zero-copy view over [buffer]. + factory Uint64List.view( + td.ByteBuffer buffer, [ + int offsetInBytes = 0, + int? length, + ]) { + return Uint64List._(td.Uint64List.view(buffer, offsetInBytes, length)); + } + + /// Creates a zero-copy element-range view of [data]. + factory Uint64List.sublistView(td.TypedData data, [int start = 0, int? end]) { + return Uint64List._(td.Uint64List.sublistView(data, start, end)); + } + + /// The number of elements in this list. + int get length => _storage.length; + + /// Returns the element at [index]. + Uint64 operator [](int index) => Uint64(_storage[index]); + + /// Stores [value] at [index]. + void operator []=(int index, Uint64 value) { + _storage[index] = value.value; + } + + td.ByteBuffer get buffer => _storage.buffer; + + int get elementSizeInBytes => _storage.elementSizeInBytes; + + int get offsetInBytes => _storage.offsetInBytes; + + int get lengthInBytes => _storage.lengthInBytes; + + Iterator get iterator => + Iterable.generate(length, (index) => this[index]).iterator; +} + +int _normalizeUnsigned64(BigInt value) { + var normalized = value & _uint64Mask64Big; + if ((normalized & _uint64SignBit64Big) != BigInt.zero) { + normalized -= BigInt.one << 64; + } + return normalized.toInt(); +} diff --git a/dart/packages/fory/lib/src/types/uint64_web.dart b/dart/packages/fory/lib/src/types/uint64_web.dart new file mode 100644 index 0000000000..b15d7d27b9 --- /dev/null +++ b/dart/packages/fory/lib/src/types/uint64_web.dart @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import 'dart:collection'; +import 'dart:typed_data' as td; + +const int _uint64SegmentBits = 22; +const int _uint64SegmentMask = (1 << _uint64SegmentBits) - 1; +const int _uint64SegmentBase = 1 << _uint64SegmentBits; +const int _uint64HighBits = 20; +const int _uint64HighMask = (1 << _uint64HighBits) - 1; +const int _uint64SafeHighMax = 0x1ff; +const int _uint64HighValueBase = 17592186044416; +const int _uint64WordBase = 0x100000000; +final BigInt _uint64Mask64Big = (BigInt.one << 64) - BigInt.one; +final BigInt _uint64Mask32Big = (BigInt.one << 32) - BigInt.one; + +/// Unsigned 64-bit integer wrapper used by the xlang type system on web builds. +final class Uint64 implements Comparable { + final int _l; + final int _m; + final int _h; + + const Uint64._parts(this._l, this._m, this._h); + + /// Creates an unsigned 64-bit value by truncating [value] to 64 bits. + factory Uint64(int value) { + final low32 = value.toUnsigned(32); + final high32 = (value - low32) ~/ _uint64WordBase; + return Uint64.fromWords(low32, high32); + } + + /// Creates an unsigned 64-bit value by truncating [value] to 64 bits. + factory Uint64.fromBigInt(BigInt value) { + final normalized = value & _uint64Mask64Big; + return Uint64.fromWords( + (normalized & _uint64Mask32Big).toInt(), + ((normalized >> 32) & _uint64Mask32Big).toInt(), + ); + } + + /// Creates an unsigned 64-bit value from little-endian 32-bit words. + factory Uint64.fromWords(int low32, int high32) { + final normalizedLow32 = low32.toUnsigned(32); + final normalizedHigh32 = high32.toUnsigned(32); + return Uint64._parts( + normalizedLow32 & _uint64SegmentMask, + ((normalizedLow32 >>> _uint64SegmentBits) | + ((normalizedHigh32 & 0xfff) << 10)) & + _uint64SegmentMask, + (normalizedHigh32 >>> 12) & _uint64HighMask, + ); + } + + /// Parses a hexadecimal payload. + factory Uint64.parseHex(String value) => + Uint64.fromBigInt(BigInt.parse(value, radix: 16)); + + /// Returns whether this value is zero. + bool get isZero => _l == 0 && _m == 0 && _h == 0; + + /// Returns the low 32 bits as an unsigned integer. + int get low32 => _l | ((_m & 0x3ff) << 22); + + /// Returns the high 32 bits as an unsigned integer. + int get high32Unsigned => ((_m >>> 10) & 0xfff) | (_h << 12); + + /// Returns the exact value as a [BigInt]. + BigInt toBigInt() => (BigInt.from(high32Unsigned) << 32) | BigInt.from(low32); + + /// Returns this value as a Dart [int] when it is JS-safe. + int toInt() { + if (_h > _uint64SafeHighMax) { + throw StateError('Uint64 value ${toBigInt()} is not a JS-safe int.'); + } + return _h * _uint64HighValueBase + _m * _uint64SegmentBase + _l; + } + + @override + int compareTo(Uint64 other) { + if (_h != other._h) { + return _h < other._h ? -1 : 1; + } + if (_m != other._m) { + return _m < other._m ? -1 : 1; + } + if (_l != other._l) { + return _l < other._l ? -1 : 1; + } + return 0; + } + + Uint64 operator +(Object other) => switch (other) { + int otherValue => _add(Uint64(otherValue)), + Uint64 otherValue => _add(otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator -(Object other) => switch (other) { + int otherValue => _subtract(Uint64(otherValue)), + Uint64 otherValue => _subtract(otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator *(Object other) => switch (other) { + int otherValue => + Uint64.fromBigInt(toBigInt() * BigInt.from(otherValue)), + Uint64 otherValue => + Uint64.fromBigInt(toBigInt() * otherValue.toBigInt()), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator ~/(Object other) => switch (other) { + int otherValue => + Uint64.fromBigInt(toBigInt() ~/ BigInt.from(otherValue)), + Uint64 otherValue => + Uint64.fromBigInt(toBigInt() ~/ otherValue.toBigInt()), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator %(Object other) => switch (other) { + int otherValue => + Uint64.fromBigInt(toBigInt().remainder(BigInt.from(otherValue))), + Uint64 otherValue => + Uint64.fromBigInt(toBigInt().remainder(otherValue.toBigInt())), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + double operator /(Object other) => switch (other) { + int otherValue => toBigInt().toDouble() / otherValue, + Uint64 otherValue => + toBigInt().toDouble() / otherValue.toBigInt().toDouble(), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator -() => Uint64(0) - this; + + Uint64 operator ~() => Uint64._parts( + _l ^ _uint64SegmentMask, + _m ^ _uint64SegmentMask, + _h ^ _uint64HighMask, + ); + + Uint64 operator &(Object other) => switch (other) { + int otherValue => _bitwiseAnd(Uint64(otherValue)), + Uint64 otherValue => _bitwiseAnd(otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator |(Object other) => switch (other) { + int otherValue => _bitwiseOr(Uint64(otherValue)), + Uint64 otherValue => _bitwiseOr(otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator ^(Object other) => switch (other) { + int otherValue => _bitwiseXor(Uint64(otherValue)), + Uint64 otherValue => _bitwiseXor(otherValue), + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + Uint64 operator <<(int shift) { + RangeError.checkNotNegative(shift, 'shift'); + if (shift == 0) { + return this; + } + if (shift >= 64) { + return Uint64(0); + } + if (shift < 22) { + return Uint64._parts( + (_l << shift) & _uint64SegmentMask, + ((_m << shift) | (_l >>> (22 - shift))) & _uint64SegmentMask, + (((_h << shift) | (_m >>> (22 - shift))) & _uint64HighMask), + ); + } + if (shift < 44) { + final segmentShift = shift - 22; + return Uint64._parts( + 0, + (_l << segmentShift) & _uint64SegmentMask, + (((_m << segmentShift) | (_l >>> (22 - segmentShift))) & + _uint64HighMask), + ); + } + final segmentShift = shift - 44; + return Uint64._parts(0, 0, (_l << segmentShift) & _uint64HighMask); + } + + Uint64 operator >>(int shift) { + RangeError.checkNotNegative(shift, 'shift'); + if (shift == 0) { + return this; + } + if (shift >= 64) { + return Uint64(0); + } + if (shift < 22) { + return Uint64._parts( + ((_l >>> shift) | ((_m & _uint64LowBitsMask(shift)) << (22 - shift))) & + _uint64SegmentMask, + ((_m >>> shift) | ((_h & _uint64LowBitsMask(shift)) << (22 - shift))) & + _uint64SegmentMask, + _h >>> shift, + ); + } + if (shift < 44) { + final segmentShift = shift - 22; + return Uint64._parts( + ((_m >>> segmentShift) | + ((_h & _uint64LowBitsMask(segmentShift)) << + (22 - segmentShift))) & + _uint64SegmentMask, + _h >>> segmentShift, + 0, + ); + } + final segmentShift = shift - 44; + return Uint64._parts(_h >>> segmentShift, 0, 0); + } + + Uint64 operator >>>(int shift) => this >> shift; + + bool operator <(Object other) => switch (other) { + int otherValue => compareTo(Uint64(otherValue)) < 0, + Uint64 otherValue => compareTo(otherValue) < 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + bool operator <=(Object other) => switch (other) { + int otherValue => compareTo(Uint64(otherValue)) <= 0, + Uint64 otherValue => compareTo(otherValue) <= 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + bool operator >(Object other) => switch (other) { + int otherValue => compareTo(Uint64(otherValue)) > 0, + Uint64 otherValue => compareTo(otherValue) > 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + bool operator >=(Object other) => switch (other) { + int otherValue => compareTo(Uint64(otherValue)) >= 0, + Uint64 otherValue => compareTo(otherValue) >= 0, + _ => throw ArgumentError.value( + other, 'other', 'Expected an int or Uint64.'), + }; + + @override + bool operator ==(Object other) => + identical(this, other) || + other is Uint64 && other._l == _l && other._m == _m && other._h == _h; + + @override + int get hashCode => Object.hash(low32, high32Unsigned); + + @override + String toString() => toBigInt().toString(); + + Uint64 _add(Uint64 other) { + var low = _l + other._l; + var carry = low >> 22; + low &= _uint64SegmentMask; + var mid = _m + other._m + carry; + carry = mid >> 22; + mid &= _uint64SegmentMask; + return Uint64._parts(low, mid, (_h + other._h + carry) & _uint64HighMask); + } + + Uint64 _subtract(Uint64 other) { + var low = _l - other._l; + var borrow = 0; + if (low < 0) { + low += _uint64SegmentBase; + borrow = 1; + } + var mid = _m - other._m - borrow; + borrow = 0; + if (mid < 0) { + mid += _uint64SegmentBase; + borrow = 1; + } + return Uint64._parts(low, mid, (_h - other._h - borrow) & _uint64HighMask); + } + + Uint64 _bitwiseAnd(Uint64 other) => Uint64._parts( + _l & other._l, + _m & other._m, + _h & other._h, + ); + + Uint64 _bitwiseOr(Uint64 other) => Uint64._parts( + _l | other._l, + _m | other._m, + _h | other._h, + ); + + Uint64 _bitwiseXor(Uint64 other) => Uint64._parts( + _l ^ other._l, + _m ^ other._m, + _h ^ other._h, + ); +} + +/// Fixed-length contiguous storage for [Uint64] values on web builds. +final class Uint64List extends IterableBase { + /// The number of bytes used by one [Uint64] element. + static const int bytesPerElement = 8; + + final td.Uint8List _bytes; + late final td.ByteData _view = td.ByteData.sublistView(_bytes); + + /// Creates a zero-initialized list with [length] unsigned 64-bit elements. + Uint64List(int length) : _bytes = td.Uint8List(length * bytesPerElement); + + Uint64List._(this._bytes); + + /// Copies [values] into a new contiguous unsigned 64-bit list. + factory Uint64List.fromList(Iterable values) { + final copied = values.toList(growable: false); + final result = Uint64List(copied.length); + for (var index = 0; index < copied.length; index += 1) { + result[index] = switch (copied[index]) { + int value => Uint64(value), + Uint64 value => value, + _ => throw ArgumentError.value( + copied[index], + 'values[$index]', + 'Expected an int or Uint64.', + ), + }; + } + return result; + } + + /// Creates a zero-copy view over [buffer]. + factory Uint64List.view( + td.ByteBuffer buffer, [ + int offsetInBytes = 0, + int? length, + ]) { + final byteLength = length == null + ? buffer.lengthInBytes - offsetInBytes + : length * bytesPerElement; + return Uint64List._(td.Uint8List.view(buffer, offsetInBytes, byteLength)); + } + + /// Creates a zero-copy element-range view of [data]. + factory Uint64List.sublistView(td.TypedData data, [int start = 0, int? end]) { + final elementCount = data.lengthInBytes ~/ data.elementSizeInBytes; + final endIndex = RangeError.checkValidRange(start, end, elementCount); + final byteOffset = data.offsetInBytes + start * data.elementSizeInBytes; + final byteLength = (endIndex - start) * data.elementSizeInBytes; + if (byteOffset % bytesPerElement != 0 || + byteLength % bytesPerElement != 0) { + throw ArgumentError( + 'The selected byte range must align to $bytesPerElement-byte ' + 'Uint64 elements.', + ); + } + return Uint64List.view( + data.buffer, + byteOffset, + byteLength ~/ bytesPerElement, + ); + } + + /// The number of elements in this list. + @override + int get length => _bytes.lengthInBytes ~/ bytesPerElement; + + /// Returns the element at [index]. + Uint64 operator [](int index) { + RangeError.checkValidIndex(index, this, 'index', length); + final byteOffset = index * bytesPerElement; + return Uint64.fromWords( + _view.getUint32(byteOffset, td.Endian.little), + _view.getUint32(byteOffset + 4, td.Endian.little), + ); + } + + /// Stores [value] at [index]. + void operator []=(int index, Uint64 value) { + RangeError.checkValidIndex(index, this, 'index', length); + final byteOffset = index * bytesPerElement; + _view.setUint32(byteOffset, value.low32, td.Endian.little); + _view.setUint32(byteOffset + 4, value.high32Unsigned, td.Endian.little); + } + + td.ByteBuffer get buffer => _bytes.buffer; + + int get elementSizeInBytes => bytesPerElement; + + int get offsetInBytes => _bytes.offsetInBytes; + + int get lengthInBytes => _bytes.lengthInBytes; + + @override + Iterator get iterator => + Iterable.generate(length, (index) => this[index]).iterator; + + @override + String toString() => toList().toString(); +} + +int _uint64LowBitsMask(int bits) { + if (bits <= 0) { + return 0; + } + return (1 << bits) - 1; +} diff --git a/dart/packages/fory/lib/src/util/hash_util.dart b/dart/packages/fory/lib/src/util/hash_util.dart index b6c211bd5e..c49cdd0181 100644 --- a/dart/packages/fory/lib/src/util/hash_util.dart +++ b/dart/packages/fory/lib/src/util/hash_util.dart @@ -22,145 +22,156 @@ import 'dart:convert'; import 'package:fory/src/meta/field_info.dart'; import 'package:fory/src/meta/type_def.dart'; import 'package:fory/src/meta/type_ids.dart'; +import 'package:fory/src/types/int64.dart'; +import 'package:fory/src/types/uint64.dart'; const int _typeDefCompressMetaFlag = 1 << 9; const int _typeDefHasFieldsMetaFlag = 1 << 8; const int _typeDefMetaSizeMask = 0xff; const int _typeDefHashShift = 14; -final BigInt _mask64Big = (BigInt.one << 64) - BigInt.one; -final BigInt _signBit64Big = BigInt.one << 63; -final BigInt _metaStringHashMaskBig = - BigInt.parse('ffffffffffffff00', radix: 16); -final BigInt _c1Big = BigInt.parse('87c37b91114253d5', radix: 16); -final BigInt _c2Big = BigInt.parse('4cf5ad432745937f', radix: 16); +final Uint64 _metaStringHashMask = Uint64.fromWords(0xffffff00, 0xffffffff); +final Uint64 _c1 = Uint64.fromWords(0x114253d5, 0x87c37b91); +final Uint64 _c2 = Uint64.fromWords(0x2745937f, 0x4cf5ad43); +final Uint64 _fmixC1 = Uint64.fromWords(0xed558ccd, 0xff51afd7); +final Uint64 _fmixC2 = Uint64.fromWords(0x1a85ec53, 0xc4ceb9fe); -(int, int) murmurHash3X64_128(List bytes, {int seed = 47}) { - var h1 = seed & 0x00000000ffffffff; - var h2 = seed & 0x00000000ffffffff; +(Int64, Int64) murmurHash3X64_128(List bytes, {int seed = 47}) { + final hash = _murmurHash3X64_128Bits(bytes, seed: seed); + return (_int64FromUint64(hash.$1), _int64FromUint64(hash.$2)); +} + +(Uint64, Uint64) _murmurHash3X64_128Bits(List bytes, {int seed = 47}) { + var h1 = Uint64(seed & 0x00000000ffffffff); + var h2 = Uint64(seed & 0x00000000ffffffff); final blockCount = bytes.length ~/ 16; for (var index = 0; index < blockCount; index += 1) { var k1 = _readLongLittleEndian(bytes, index * 16); var k2 = _readLongLittleEndian(bytes, index * 16 + 8); - k1 = _mul64(k1, _c1Big); + k1 = k1 * _c1; k1 = _rotateLeft64(k1, 31); - k1 = _mul64(k1, _c2Big); - h1 = _toSigned64(h1 ^ k1); + k1 = k1 * _c2; + h1 = h1 ^ k1; h1 = _rotateLeft64(h1, 27); - h1 = _add64(h1, h2); - h1 = _add64(_mul64(h1, BigInt.from(5)), 0x52dce729); + h1 = h1 + h2; + h1 = (h1 * 5) + 0x52dce729; - k2 = _mul64(k2, _c2Big); + k2 = k2 * _c2; k2 = _rotateLeft64(k2, 33); - k2 = _mul64(k2, _c1Big); - h2 = _toSigned64(h2 ^ k2); + k2 = k2 * _c1; + h2 = h2 ^ k2; h2 = _rotateLeft64(h2, 31); - h2 = _add64(h2, h1); - h2 = _add64(_mul64(h2, BigInt.from(5)), 0x38495ab5); + h2 = h2 + h1; + h2 = (h2 * 5) + 0x38495ab5; } - var k1 = 0; - var k2 = 0; + var k1 = Uint64(0); + var k2 = Uint64(0); final tailOffset = blockCount * 16; final tailLength = bytes.length & 15; if (tailLength >= 15) { - k2 ^= (bytes[tailOffset + 14] & 0xff) << 48; + k2 = k2 ^ (Uint64(bytes[tailOffset + 14] & 0xff) << 48); } if (tailLength >= 14) { - k2 ^= (bytes[tailOffset + 13] & 0xff) << 40; + k2 = k2 ^ (Uint64(bytes[tailOffset + 13] & 0xff) << 40); } if (tailLength >= 13) { - k2 ^= (bytes[tailOffset + 12] & 0xff) << 32; + k2 = k2 ^ (Uint64(bytes[tailOffset + 12] & 0xff) << 32); } if (tailLength >= 12) { - k2 ^= (bytes[tailOffset + 11] & 0xff) << 24; + k2 = k2 ^ (Uint64(bytes[tailOffset + 11] & 0xff) << 24); } if (tailLength >= 11) { - k2 ^= (bytes[tailOffset + 10] & 0xff) << 16; + k2 = k2 ^ (Uint64(bytes[tailOffset + 10] & 0xff) << 16); } if (tailLength >= 10) { - k2 ^= (bytes[tailOffset + 9] & 0xff) << 8; + k2 = k2 ^ (Uint64(bytes[tailOffset + 9] & 0xff) << 8); } if (tailLength >= 9) { - k2 ^= bytes[tailOffset + 8] & 0xff; - k2 = _mul64(k2, _c2Big); + k2 = k2 ^ (bytes[tailOffset + 8] & 0xff); + k2 = k2 * _c2; k2 = _rotateLeft64(k2, 33); - k2 = _mul64(k2, _c1Big); - h2 = _toSigned64(h2 ^ k2); + k2 = k2 * _c1; + h2 = h2 ^ k2; } if (tailLength >= 8) { - k1 ^= _signedByte(bytes[tailOffset + 7]) << 56; + k1 = k1 ^ (Uint64(bytes[tailOffset + 7] & 0xff) << 56); } if (tailLength >= 7) { - k1 ^= (bytes[tailOffset + 6] & 0xff) << 48; + k1 = k1 ^ (Uint64(bytes[tailOffset + 6] & 0xff) << 48); } if (tailLength >= 6) { - k1 ^= (bytes[tailOffset + 5] & 0xff) << 40; + k1 = k1 ^ (Uint64(bytes[tailOffset + 5] & 0xff) << 40); } if (tailLength >= 5) { - k1 ^= (bytes[tailOffset + 4] & 0xff) << 32; + k1 = k1 ^ (Uint64(bytes[tailOffset + 4] & 0xff) << 32); } if (tailLength >= 4) { - k1 ^= (bytes[tailOffset + 3] & 0xff) << 24; + k1 = k1 ^ (Uint64(bytes[tailOffset + 3] & 0xff) << 24); } if (tailLength >= 3) { - k1 ^= (bytes[tailOffset + 2] & 0xff) << 16; + k1 = k1 ^ (Uint64(bytes[tailOffset + 2] & 0xff) << 16); } if (tailLength >= 2) { - k1 ^= (bytes[tailOffset + 1] & 0xff) << 8; + k1 = k1 ^ (Uint64(bytes[tailOffset + 1] & 0xff) << 8); } if (tailLength >= 1) { - k1 ^= bytes[tailOffset] & 0xff; - k1 = _mul64(k1, _c1Big); + k1 = k1 ^ (bytes[tailOffset] & 0xff); + k1 = k1 * _c1; k1 = _rotateLeft64(k1, 31); - k1 = _mul64(k1, _c2Big); - h1 = _toSigned64(h1 ^ k1); + k1 = k1 * _c2; + h1 = h1 ^ k1; } - h1 = _toSigned64(h1 ^ bytes.length); - h2 = _toSigned64(h2 ^ bytes.length); + h1 = h1 ^ bytes.length; + h2 = h2 ^ bytes.length; - h1 = _add64(h1, h2); - h2 = _add64(h2, h1); + h1 = h1 + h2; + h2 = h2 + h1; h1 = _fmix64(h1); h2 = _fmix64(h2); - h1 = _add64(h1, h2); - h2 = _add64(h2, h1); + h1 = h1 + h2; + h2 = h2 + h1; return (h1, h2); } -int metaStringHash(List bytes, {int encoding = 0}) { - var hash = _absSigned64(murmurHash3X64_128(bytes).$1); - if (hash == 0) { - hash += 0x100; +Int64 metaStringHash(List bytes, {int encoding = 0}) { + var hash = + _absSigned64Bits(_int64FromUint64(_murmurHash3X64_128Bits(bytes).$1)); + if (hash.isZero) { + hash = hash + 0x100; } - hash = (BigInt.from(hash) & _metaStringHashMaskBig).toInt(); - return hash | (encoding & 0xff); + hash = (hash & _metaStringHashMask) | (encoding & 0xff); + return _int64FromUint64(hash); } -int typeDefHeader( +Int64 typeDefHeader( List bytes, { required bool hasFieldsMeta, bool compressed = false, }) { - final hash = _toSigned64(murmurHash3X64_128(bytes).$1 << _typeDefHashShift); - var header = _absSigned64(hash); + final hash = _int64FromUint64( + _murmurHash3X64_128Bits(bytes).$1 << _typeDefHashShift, + ); + var header = _absSigned64Bits(hash); if (compressed) { - header |= _typeDefCompressMetaFlag; + header = header | _typeDefCompressMetaFlag; } if (hasFieldsMeta) { - header |= _typeDefHasFieldsMetaFlag; + header = header | _typeDefHasFieldsMetaFlag; } - header |= - bytes.length > _typeDefMetaSizeMask ? _typeDefMetaSizeMask : bytes.length; - return _toSigned64(header); + header = header | + (bytes.length > _typeDefMetaSizeMask + ? _typeDefMetaSizeMask + : bytes.length); + return _int64FromUint64(header); } int schemaHash(TypeDef typeDef) { @@ -179,8 +190,7 @@ int schemaHash(TypeDef typeDef) { .map((buffer) => buffer.toString()) .toList(growable: false) ..sort(); - final hash = murmurHash3X64_128(utf8.encode(parts.join())).$1; - return hash & 0xffffffff; + return _murmurHash3X64_128Bits(utf8.encode(parts.join())).$1.low32; } int _fingerprintTypeId(FieldInfo field) { @@ -206,56 +216,42 @@ int _fingerprintTypeId(FieldInfo field) { } } -int _toSigned64Big(BigInt value) { - final normalized = _u64Big(value); - if ((normalized & _signBit64Big) != BigInt.zero) { - return (normalized - (BigInt.one << 64)).toInt(); - } - return normalized.toInt(); -} - -int _readLongLittleEndian(List bytes, int offset) { - final value = (_signedByte(bytes[offset + 7]) << 56) | - ((bytes[offset + 6] & 0xff) << 48) | - ((bytes[offset + 5] & 0xff) << 40) | - ((bytes[offset + 4] & 0xff) << 32) | - ((bytes[offset + 3] & 0xff) << 24) | - ((bytes[offset + 2] & 0xff) << 16) | - ((bytes[offset + 1] & 0xff) << 8) | - (bytes[offset] & 0xff); - return _toSigned64(value); +Uint64 _readLongLittleEndian(List bytes, int offset) { + final low = (bytes[offset] & 0xff) + + ((bytes[offset + 1] & 0xff) << 8) + + ((bytes[offset + 2] & 0xff) << 16) + + ((bytes[offset + 3] & 0xff) * 0x1000000); + final high = (bytes[offset + 4] & 0xff) + + ((bytes[offset + 5] & 0xff) << 8) + + ((bytes[offset + 6] & 0xff) << 16) + + ((bytes[offset + 7] & 0xff) * 0x1000000); + return Uint64.fromWords(low, high); } -int _signedByte(int value) => value >= 0x80 ? value - 0x100 : value; - -int _rotateLeft64(int value, int shift) { - final normalized = _u64Big(BigInt.from(value)); - return _toSigned64Big((normalized << shift) | (normalized >> (64 - shift))); +Uint64 _rotateLeft64(Uint64 value, int shift) { + return (value << shift) | (value >> (64 - shift)); } -int _unsignedRightShift64(int value, int shift) => - (_u64Big(BigInt.from(value)) >> shift).toInt(); - -int _mul64(int value, BigInt factor) => - _toSigned64Big(BigInt.from(value) * factor); - -int _add64(int left, int right) => - _toSigned64Big(BigInt.from(left) + BigInt.from(right)); - -int _fmix64(int value) { +Uint64 _fmix64(Uint64 value) { var mixed = value; - mixed = _toSigned64(mixed ^ _unsignedRightShift64(mixed, 33)); - mixed = _mul64(mixed, BigInt.parse('ff51afd7ed558ccd', radix: 16)); - mixed = _toSigned64(mixed ^ _unsignedRightShift64(mixed, 33)); - mixed = _mul64(mixed, BigInt.parse('c4ceb9fe1a85ec53', radix: 16)); - mixed = _toSigned64(mixed ^ _unsignedRightShift64(mixed, 33)); + mixed = mixed ^ (mixed >> 33); + mixed = mixed * _fmixC1; + mixed = mixed ^ (mixed >> 33); + mixed = mixed * _fmixC2; + mixed = mixed ^ (mixed >> 33); return mixed; } -BigInt _u64Big(BigInt value) => value & _mask64Big; - -int _toSigned64(int value) { - return _toSigned64Big(BigInt.from(value)); +Uint64 _absSigned64Bits(Int64 value) { + if (!value.isNegative) { + return _uint64FromInt64(value); + } + final negated = -value; + return Uint64.fromWords(negated.low32, negated.high32Unsigned); } -int _absSigned64(int value) => value < 0 ? -value : value; +Int64 _int64FromUint64(Uint64 value) => + Int64.fromWords(value.low32, value.high32Unsigned); + +Uint64 _uint64FromInt64(Int64 value) => + Uint64.fromWords(value.low32, value.high32Unsigned); diff --git a/dart/packages/fory/lib/src/util/string_util.dart b/dart/packages/fory/lib/src/util/string_util.dart index 3a3b5f60dc..b274264b80 100644 --- a/dart/packages/fory/lib/src/util/string_util.dart +++ b/dart/packages/fory/lib/src/util/string_util.dart @@ -20,7 +20,7 @@ import 'dart:convert'; import 'dart:typed_data'; -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; const int stringLatin1Encoding = 0; const int stringUtf16Encoding = 1; diff --git a/dart/packages/fory/test/buffer_test.dart b/dart/packages/fory/test/buffer_test.dart index 4c972e5a9b..2a5ddb1f70 100644 --- a/dart/packages/fory/test/buffer_test.dart +++ b/dart/packages/fory/test/buffer_test.dart @@ -22,6 +22,18 @@ import 'dart:typed_data'; import 'package:fory/fory.dart'; import 'package:test/test.dart'; +Int64 _i64Hex(String value) => Int64.parseHex(value); + +Uint64 _u64Hex(String value) => Uint64.parseHex(value); + +Int64 _i64Pow2(int shift) => Int64(1) << shift; + +Uint64 _u64Pow2(int shift) => Uint64(1) << shift; + +const int _jsSafeIntMax = 9007199254740991; +const int _jsSafeIntMin = -9007199254740991; +const int _jsUnsafeInt = 9007199254740992; + void main() { group('Buffer', () { test('round-trips fixed-width primitives, 16-bit floats, and bytes', () { @@ -37,8 +49,8 @@ void main() { buffer.writeUint16(0xffff); buffer.writeInt32(-0x80000000); buffer.writeUint32(0xffffffff); - buffer.writeInt64(-0x8000000000000000); - buffer.writeUint64(0xffffffffffffffff); + buffer.writeInt64(_i64Hex('8000000000000000')); + buffer.writeUint64(_u64Hex('ffffffffffffffff')); buffer.writeFloat16(half); buffer.writeBfloat16(brain); buffer.writeFloat32(1.5); @@ -53,8 +65,8 @@ void main() { expect(buffer.readUint16(), equals(0xffff)); expect(buffer.readInt32(), equals(-0x80000000)); expect(buffer.readUint32(), equals(0xffffffff)); - expect(buffer.readInt64(), equals(-0x8000000000000000)); - expect(buffer.readUint64(), equals(0xffffffffffffffff)); + expect(buffer.readInt64(), equals(_i64Hex('8000000000000000'))); + expect(buffer.readUint64(), equals(_u64Hex('ffffffffffffffff'))); expect(buffer.readFloat16(), equals(half)); expect(buffer.readBfloat16(), equals(brain)); expect(buffer.readFloat32(), closeTo(1.5, 0.0001)); @@ -79,6 +91,16 @@ void main() { expect(buffer.toBytes().last, equals(7)); }); + test('grows from zero-capacity storage', () { + final buffer = Buffer(0); + buffer.writeBytes([1, 2, 3]); + expect(buffer.toBytes(), orderedEquals([1, 2, 3])); + + final wrapped = Buffer.wrap(Uint8List(0)); + wrapped.writeUint8(4); + expect(wrapped.toBytes(), orderedEquals([4])); + }); + test( 'wrap constructor, wrap method, read views, copyBytes, skip, and clear manage indices', () { @@ -190,33 +212,33 @@ void main() { }); test('round-trips varuint64 boundary values with Java-aligned lengths', () { - const cases = <({int bytes, int value})>[ - (bytes: 1, value: 0), - (bytes: 1, value: 1), - (bytes: 1, value: 1 << 6), - (bytes: 2, value: 1 << 7), - (bytes: 2, value: 1 << 13), - (bytes: 3, value: 1 << 14), - (bytes: 3, value: 1 << 20), - (bytes: 4, value: 1 << 21), - (bytes: 4, value: 1 << 27), - (bytes: 5, value: 1 << 28), - (bytes: 5, value: 1 << 34), - (bytes: 6, value: 1 << 35), - (bytes: 6, value: 1 << 41), - (bytes: 7, value: 1 << 42), - (bytes: 7, value: 1 << 48), - (bytes: 8, value: 1 << 49), - (bytes: 8, value: 1 << 55), - (bytes: 9, value: 1 << 56), - (bytes: 9, value: 1 << 62), - (bytes: 9, value: 0x7fffffffffffffff), - (bytes: 9, value: -0x8000000000000000), - (bytes: 9, value: 0xffffffffffffffff), + final cases = <({int bytes, Uint64 value})>[ + (bytes: 1, value: Uint64(0)), + (bytes: 1, value: Uint64(1)), + (bytes: 1, value: _u64Pow2(6)), + (bytes: 2, value: _u64Pow2(7)), + (bytes: 2, value: _u64Pow2(13)), + (bytes: 3, value: _u64Pow2(14)), + (bytes: 3, value: _u64Pow2(20)), + (bytes: 4, value: _u64Pow2(21)), + (bytes: 4, value: _u64Pow2(27)), + (bytes: 5, value: _u64Pow2(28)), + (bytes: 5, value: _u64Pow2(34)), + (bytes: 6, value: _u64Pow2(35)), + (bytes: 6, value: _u64Pow2(41)), + (bytes: 7, value: _u64Pow2(42)), + (bytes: 7, value: _u64Pow2(48)), + (bytes: 8, value: _u64Pow2(49)), + (bytes: 8, value: _u64Pow2(55)), + (bytes: 9, value: _u64Pow2(56)), + (bytes: 9, value: _u64Pow2(62)), + (bytes: 9, value: _u64Hex('7fffffffffffffff')), + (bytes: 9, value: _u64Hex('8000000000000000')), + (bytes: 9, value: _u64Hex('ffffffffffffffff')), ]; for (final testCase in cases) { - _expectEncodedIntRoundTrip( + _expectEncodedUint64RoundTrip( value: testCase.value, expectedBytes: testCase.bytes, write: (buffer, value) => buffer.writeVarUint64(value), @@ -227,33 +249,33 @@ void main() { }); test('round-trips varint64 boundary values with Java-aligned lengths', () { - const cases = <({int bytes, int value})>[ - (bytes: 1, value: 0), - (bytes: 1, value: 1), - (bytes: 1, value: -1), - (bytes: 1, value: -64), - (bytes: 2, value: 1 << 6), - (bytes: 2, value: -128), - (bytes: 3, value: 1 << 13), - (bytes: 3, value: -16384), - (bytes: 4, value: 1 << 20), - (bytes: 4, value: -2097152), - (bytes: 5, value: 1 << 27), - (bytes: 5, value: -268435456), - (bytes: 6, value: 1 << 34), - (bytes: 6, value: -34359738368), - (bytes: 7, value: 1 << 42), - (bytes: 7, value: -4398046511104), - (bytes: 8, value: 1 << 49), - (bytes: 8, value: -562949953421312), - (bytes: 9, value: 1 << 55), - (bytes: 9, value: -72057594037927936), - (bytes: 9, value: 0x7fffffffffffffff), - (bytes: 9, value: -0x8000000000000000), + final cases = <({int bytes, Int64 value})>[ + (bytes: 1, value: Int64(0)), + (bytes: 1, value: Int64(1)), + (bytes: 1, value: Int64(-1)), + (bytes: 1, value: Int64(-64)), + (bytes: 2, value: _i64Pow2(6)), + (bytes: 2, value: Int64(-128)), + (bytes: 3, value: _i64Pow2(13)), + (bytes: 3, value: Int64(-16384)), + (bytes: 4, value: _i64Pow2(20)), + (bytes: 4, value: Int64(-2097152)), + (bytes: 5, value: _i64Pow2(27)), + (bytes: 5, value: Int64(-268435456)), + (bytes: 6, value: _i64Pow2(34)), + (bytes: 6, value: -_i64Pow2(35)), + (bytes: 7, value: _i64Pow2(42)), + (bytes: 7, value: -_i64Pow2(42)), + (bytes: 8, value: _i64Pow2(49)), + (bytes: 8, value: -_i64Pow2(49)), + (bytes: 9, value: _i64Pow2(55)), + (bytes: 9, value: -_i64Pow2(56)), + (bytes: 9, value: _i64Hex('7fffffffffffffff')), + (bytes: 9, value: _i64Hex('8000000000000000')), ]; for (final testCase in cases) { - _expectEncodedIntRoundTrip( + _expectEncodedInt64RoundTrip( value: testCase.value, expectedBytes: testCase.bytes, write: (buffer, value) => buffer.writeVarInt64(value), @@ -265,21 +287,21 @@ void main() { test('round-trips tagged int64 boundary values with Java-aligned lengths', () { - const cases = <({int bytes, int value})>[ - (bytes: 4, value: -0x40000000), - (bytes: 4, value: -1), - (bytes: 4, value: 0), - (bytes: 4, value: 1), - (bytes: 4, value: 1 << 28), - (bytes: 4, value: 0x3fffffff), - (bytes: 9, value: -0x40000001), - (bytes: 9, value: 0x40000000), - (bytes: 9, value: 0x7fffffffffffffff), - (bytes: 9, value: -0x8000000000000000), + final cases = <({int bytes, Int64 value})>[ + (bytes: 4, value: Int64(-0x40000000)), + (bytes: 4, value: Int64(-1)), + (bytes: 4, value: Int64(0)), + (bytes: 4, value: Int64(1)), + (bytes: 4, value: Int64(1 << 28)), + (bytes: 4, value: Int64(0x3fffffff)), + (bytes: 9, value: Int64(-0x40000001)), + (bytes: 9, value: Int64(0x40000000)), + (bytes: 9, value: _i64Hex('7fffffffffffffff')), + (bytes: 9, value: _i64Hex('8000000000000000')), ]; for (final testCase in cases) { - _expectEncodedIntRoundTrip( + _expectEncodedInt64RoundTrip( value: testCase.value, expectedBytes: testCase.bytes, write: (buffer, value) => buffer.writeTaggedInt64(value), @@ -292,20 +314,20 @@ void main() { test( 'round-trips tagged uint64 boundary values with Java-aligned lengths', () { - const cases = <({int bytes, int value})>[ - (bytes: 4, value: 0), - (bytes: 4, value: 1), - (bytes: 4, value: 1 << 30), - (bytes: 4, value: 0x7fffffff), - (bytes: 9, value: 0x80000000), - (bytes: 9, value: 0x100000000), - (bytes: 9, value: 0x7fffffffffffffff), - (bytes: 9, value: -0x8000000000000000), - (bytes: 9, value: 0xffffffffffffffff), + final cases = <({int bytes, Uint64 value})>[ + (bytes: 4, value: Uint64(0)), + (bytes: 4, value: Uint64(1)), + (bytes: 4, value: Uint64(1 << 30)), + (bytes: 4, value: Uint64(0x7fffffff)), + (bytes: 9, value: Uint64(0x80000000)), + (bytes: 9, value: _u64Pow2(32)), + (bytes: 9, value: _u64Hex('7fffffffffffffff')), + (bytes: 9, value: _u64Hex('8000000000000000')), + (bytes: 9, value: _u64Hex('ffffffffffffffff')), ]; for (final testCase in cases) { - _expectEncodedIntRoundTrip( + _expectEncodedUint64RoundTrip( value: testCase.value, expectedBytes: testCase.bytes, write: (buffer, value) => buffer.writeTaggedUint64(value), @@ -316,6 +338,181 @@ void main() { }, ); + test('int64 int helpers match Int64 wrapper encodings at safe boundaries', + () { + const cases = [ + _jsSafeIntMin, + -0x40000001, + -0x40000000, + -1, + 0, + 1, + 0x3fffffff, + 0x40000000, + _jsSafeIntMax, + ]; + + for (final value in cases) { + _expectInt64IntHelperMatchesWrapper( + value: value, + writeInt: (buffer, value) => buffer.writeInt64FromInt(value), + writeWrapper: (buffer, value) => buffer.writeInt64(Int64(value)), + readInt: (buffer) => buffer.readInt64AsInt(), + cursorWriteInt: (cursor, value) => cursor.writeInt64FromInt(value), + cursorWriteWrapper: (cursor, value) => + cursor.writeInt64(Int64(value)), + cursorReadInt: (cursor) => cursor.readInt64AsInt(), + ); + _expectInt64IntHelperMatchesWrapper( + value: value, + writeInt: (buffer, value) => buffer.writeVarInt64FromInt(value), + writeWrapper: (buffer, value) => buffer.writeVarInt64(Int64(value)), + readInt: (buffer) => buffer.readVarInt64AsInt(), + cursorWriteInt: (cursor, value) => cursor.writeVarInt64FromInt(value), + cursorWriteWrapper: (cursor, value) => + cursor.writeVarInt64(Int64(value)), + cursorReadInt: (cursor) => cursor.readVarInt64AsInt(), + ); + _expectInt64IntHelperMatchesWrapper( + value: value, + writeInt: (buffer, value) => buffer.writeTaggedInt64FromInt(value), + writeWrapper: (buffer, value) => + buffer.writeTaggedInt64(Int64(value)), + readInt: (buffer) => buffer.readTaggedInt64AsInt(), + cursorWriteInt: (cursor, value) => + cursor.writeTaggedInt64FromInt(value), + cursorWriteWrapper: (cursor, value) => + cursor.writeTaggedInt64(Int64(value)), + cursorReadInt: (cursor) => cursor.readTaggedInt64AsInt(), + ); + } + }); + + test( + 'uint64 cursor int helpers match Uint64 wrapper encodings at safe boundaries', + () { + const cases = [ + 0, + 1, + 0x7fffffff, + 0x80000000, + 0xffffffff, + _jsSafeIntMax, + ]; + + for (final value in cases) { + _expectUint64CursorIntHelperMatchesWrapper( + value: value, + cursorWriteInt: (cursor, value) => cursor.writeUint64FromInt(value), + cursorWriteWrapper: (cursor, value) => cursor.writeUint64(value), + ); + _expectUint64CursorIntHelperMatchesWrapper( + value: value, + cursorWriteInt: (cursor, value) => + cursor.writeVarUint64FromInt(value), + cursorWriteWrapper: (cursor, value) => cursor.writeVarUint64(value), + ); + _expectUint64CursorIntHelperMatchesWrapper( + value: value, + cursorWriteInt: (cursor, value) => + cursor.writeTaggedUint64FromInt(value), + cursorWriteWrapper: (cursor, value) => + cursor.writeTaggedUint64(value), + ); + } + }); + + test('web rejects JS-unsafe int64 int helper values', () { + if (!identical(1, 1.0)) { + final buffer = Buffer(); + buffer.writeInt64FromInt(_jsUnsafeInt); + buffer.writeVarInt64FromInt(_jsUnsafeInt); + buffer.writeTaggedInt64FromInt(_jsUnsafeInt); + expect(buffer.readInt64AsInt(), equals(_jsUnsafeInt)); + expect(buffer.readVarInt64AsInt(), equals(_jsUnsafeInt)); + expect(buffer.readTaggedInt64AsInt(), equals(_jsUnsafeInt)); + return; + } + + expect( + () => Buffer().writeInt64FromInt(_jsUnsafeInt), + throwsA(isA()), + ); + expect( + () => Buffer().writeVarInt64FromInt(_jsUnsafeInt), + throwsA(isA()), + ); + expect( + () => Buffer().writeTaggedInt64FromInt(_jsUnsafeInt), + throwsA(isA()), + ); + + final fixed = Buffer()..writeInt64(Int64(_jsUnsafeInt)); + final varint = Buffer()..writeVarInt64(Int64(_jsUnsafeInt)); + final tagged = Buffer()..writeTaggedInt64(Int64(_jsUnsafeInt)); + expect( + () => Buffer.wrap(Uint8List.fromList(fixed.toBytes())).readInt64AsInt(), + throwsA(isA()), + ); + expect( + () => Buffer.wrap(Uint8List.fromList(varint.toBytes())) + .readVarInt64AsInt(), + throwsA(isA()), + ); + expect( + () => Buffer.wrap(Uint8List.fromList(tagged.toBytes())) + .readTaggedInt64AsInt(), + throwsA(isA()), + ); + }); + + test('web rejects unsafe uint64 cursor int helper values', () { + if (!identical(1, 1.0)) { + for (final value in [-1, _jsUnsafeInt]) { + _expectUint64CursorIntHelperMatchesWrapper( + value: value, + cursorWriteInt: (cursor, value) => cursor.writeUint64FromInt(value), + cursorWriteWrapper: (cursor, value) => cursor.writeUint64(value), + ); + _expectUint64CursorIntHelperMatchesWrapper( + value: value, + cursorWriteInt: (cursor, value) => + cursor.writeVarUint64FromInt(value), + cursorWriteWrapper: (cursor, value) => cursor.writeVarUint64(value), + ); + _expectUint64CursorIntHelperMatchesWrapper( + value: value, + cursorWriteInt: (cursor, value) => + cursor.writeTaggedUint64FromInt(value), + cursorWriteWrapper: (cursor, value) => + cursor.writeTaggedUint64(value), + ); + } + return; + } + + for (final value in [-1, _jsUnsafeInt]) { + expect( + () => _writeWithCursor((cursor) => cursor.writeUint64FromInt(value)), + throwsA(isA()), + reason: 'fixed $value', + ); + expect( + () => + _writeWithCursor((cursor) => cursor.writeVarUint64FromInt(value)), + throwsA(isA()), + reason: 'varint $value', + ); + expect( + () => _writeWithCursor( + (cursor) => cursor.writeTaggedUint64FromInt(value), + ), + throwsA(isA()), + reason: 'tagged $value', + ); + } + }); + test('round-trips small varuint helpers', () { const small7Cases = <({int bytes, int value})>[ (bytes: 1, value: 0), @@ -375,18 +572,18 @@ void main() { buffer.writeUint16(65000); buffer.writeInt32(-123456789); buffer.writeUint32(0x89abcdef); - buffer.writeInt64(-0x1234567890abcdef); - buffer.writeUint64(0xfedcba9876543210); + buffer.writeInt64(_i64Hex('-1234567890abcdef')); + buffer.writeUint64(_u64Hex('fedcba9876543210')); buffer.writeFloat16(Float16(1.5)); buffer.writeBfloat16(Bfloat16(2.5)); buffer.writeFloat32(3.25); buffer.writeFloat64(-9.5); buffer.writeVarUint32(0xffffffff); buffer.writeVarInt32(-0x40000000); - buffer.writeVarUint64(0xffffffffffffffff); - buffer.writeVarInt64(-0x4000000000000000); - buffer.writeTaggedInt64(0x40000000); - buffer.writeTaggedUint64(0x80000000); + buffer.writeVarUint64(_u64Hex('ffffffffffffffff')); + buffer.writeVarInt64(_i64Hex('c000000000000000')); + buffer.writeTaggedInt64(Int64(0x40000000)); + buffer.writeTaggedUint64(Uint64(0x80000000)); final cursor = GeneratedWriteCursor.reserve(generated, 128); cursor.writeBool(true); @@ -396,18 +593,18 @@ void main() { cursor.writeUint16(65000); cursor.writeInt32(-123456789); cursor.writeUint32(0x89abcdef); - cursor.writeInt64(-0x1234567890abcdef); - cursor.writeUint64(0xfedcba9876543210); + cursor.writeInt64(_i64Hex('-1234567890abcdef')); + cursor.writeUint64(_u64Hex('fedcba9876543210')); cursor.writeFloat16(Float16(1.5)); cursor.writeBfloat16(Bfloat16(2.5)); cursor.writeFloat32(3.25); cursor.writeFloat64(-9.5); cursor.writeVarUint32(0xffffffff); cursor.writeVarInt32(-0x40000000); - cursor.writeVarUint64(0xffffffffffffffff); - cursor.writeVarInt64(-0x4000000000000000); - cursor.writeTaggedInt64(0x40000000); - cursor.writeTaggedUint64(0x80000000); + cursor.writeVarUint64(_u64Hex('ffffffffffffffff')); + cursor.writeVarInt64(_i64Hex('c000000000000000')); + cursor.writeTaggedInt64(Int64(0x40000000)); + cursor.writeTaggedUint64(Uint64(0x80000000)); cursor.finish(); expect(generated.toBytes(), orderedEquals(buffer.toBytes())); @@ -422,18 +619,18 @@ void main() { expect(readCursor.readUint16(), equals(65000)); expect(readCursor.readInt32(), equals(-123456789)); expect(readCursor.readUint32(), equals(0x89abcdef)); - expect(readCursor.readInt64(), equals(-0x1234567890abcdef)); - expect(readCursor.readUint64(), equals(0xfedcba9876543210)); + expect(readCursor.readInt64(), equals(_i64Hex('-1234567890abcdef'))); + expect(readCursor.readUint64(), equals(_u64Hex('fedcba9876543210'))); expect(readCursor.readFloat16(), equals(Float16(1.5))); expect(readCursor.readBfloat16(), equals(Bfloat16(2.5))); expect(readCursor.readFloat32(), closeTo(3.25, 0.0001)); expect(readCursor.readFloat64(), equals(-9.5)); expect(readCursor.readVarUint32(), equals(0xffffffff)); expect(readCursor.readVarInt32(), equals(-0x40000000)); - expect(readCursor.readVarUint64(), equals(0xffffffffffffffff)); - expect(readCursor.readVarInt64(), equals(-0x4000000000000000)); - expect(readCursor.readTaggedInt64(), equals(0x40000000)); - expect(readCursor.readTaggedUint64(), equals(0x80000000)); + expect(readCursor.readVarUint64(), equals(_u64Hex('ffffffffffffffff'))); + expect(readCursor.readVarInt64(), equals(_i64Hex('c000000000000000'))); + expect(readCursor.readTaggedInt64(), equals(Int64(0x40000000))); + expect(readCursor.readTaggedUint64(), equals(Uint64(0x80000000))); readCursor.finish(); expect(readBuffer.readableBytes, equals(0)); @@ -441,6 +638,72 @@ void main() { }); } +void _expectInt64IntHelperMatchesWrapper({ + required int value, + required void Function(Buffer buffer, int value) writeInt, + required void Function(Buffer buffer, int value) writeWrapper, + required int Function(Buffer buffer) readInt, + required void Function(GeneratedWriteCursor cursor, int value) cursorWriteInt, + required void Function(GeneratedWriteCursor cursor, int value) + cursorWriteWrapper, + required int Function(GeneratedReadCursor cursor) cursorReadInt, +}) { + final intBuffer = Buffer(); + writeInt(intBuffer, value); + + final wrapperBuffer = Buffer(); + writeWrapper(wrapperBuffer, value); + expect(intBuffer.toBytes(), orderedEquals(wrapperBuffer.toBytes())); + + final readBuffer = Buffer.wrap(Uint8List.fromList(intBuffer.toBytes())); + expect(readInt(readBuffer), equals(value)); + expect(readBuffer.readableBytes, equals(0)); + + final cursorIntBuffer = Buffer(); + final cursorInt = GeneratedWriteCursor.reserve(cursorIntBuffer, 10); + cursorWriteInt(cursorInt, value); + cursorInt.finish(); + expect(cursorIntBuffer.toBytes(), orderedEquals(wrapperBuffer.toBytes())); + + final cursorWrapperBuffer = Buffer(); + final cursorWrapper = GeneratedWriteCursor.reserve(cursorWrapperBuffer, 10); + cursorWriteWrapper(cursorWrapper, value); + cursorWrapper.finish(); + expect(cursorWrapperBuffer.toBytes(), orderedEquals(wrapperBuffer.toBytes())); + + final cursorReadBuffer = Buffer.wrap(Uint8List.fromList(intBuffer.toBytes())); + final cursorRead = GeneratedReadCursor.start(cursorReadBuffer); + expect(cursorReadInt(cursorRead), equals(value)); + cursorRead.finish(); + expect(cursorReadBuffer.readableBytes, equals(0)); +} + +void _expectUint64CursorIntHelperMatchesWrapper({ + required int value, + required void Function(GeneratedWriteCursor cursor, int value) cursorWriteInt, + required void Function(GeneratedWriteCursor cursor, Uint64 value) + cursorWriteWrapper, +}) { + final intBuffer = Buffer(); + final intCursor = GeneratedWriteCursor.reserve(intBuffer, 10); + cursorWriteInt(intCursor, value); + intCursor.finish(); + + final wrapperBuffer = Buffer(); + final wrapperCursor = GeneratedWriteCursor.reserve(wrapperBuffer, 10); + cursorWriteWrapper(wrapperCursor, Uint64(value)); + wrapperCursor.finish(); + + expect(intBuffer.toBytes(), orderedEquals(wrapperBuffer.toBytes())); +} + +void _writeWithCursor(void Function(GeneratedWriteCursor cursor) write) { + final buffer = Buffer(); + final cursor = GeneratedWriteCursor.reserve(buffer, 10); + write(cursor); + cursor.finish(); +} + void _expectEncodedIntRoundTrip({ required int value, required int expectedBytes, @@ -466,3 +729,55 @@ void _expectEncodedIntRoundTrip({ expect(cursorBuffer.readableBytes, equals(0)); } } + +void _expectEncodedUint64RoundTrip({ + required Uint64 value, + required int expectedBytes, + required void Function(Buffer buffer, Uint64 value) write, + required Uint64 Function(Buffer buffer) read, + Uint64 Function(GeneratedReadCursor cursor)? cursorRead, +}) { + final buffer = Buffer(); + write(buffer, value); + final bytes = Uint8List.fromList(buffer.toBytes()); + + expect(bytes.length, equals(expectedBytes)); + + final wrapped = Buffer.wrap(Uint8List.fromList(bytes)); + expect(read(wrapped), equals(value)); + expect(wrapped.readableBytes, equals(0)); + + if (cursorRead != null) { + final cursorBuffer = Buffer.wrap(Uint8List.fromList(bytes)); + final cursor = GeneratedReadCursor.start(cursorBuffer); + expect(cursorRead(cursor), equals(value)); + cursor.finish(); + expect(cursorBuffer.readableBytes, equals(0)); + } +} + +void _expectEncodedInt64RoundTrip({ + required Int64 value, + required int expectedBytes, + required void Function(Buffer buffer, Int64 value) write, + required Int64 Function(Buffer buffer) read, + Int64 Function(GeneratedReadCursor cursor)? cursorRead, +}) { + final buffer = Buffer(); + write(buffer, value); + final bytes = Uint8List.fromList(buffer.toBytes()); + + expect(bytes.length, equals(expectedBytes)); + + final wrapped = Buffer.wrap(Uint8List.fromList(bytes)); + expect(read(wrapped), equals(value)); + expect(wrapped.readableBytes, equals(0)); + + if (cursorRead != null) { + final cursorBuffer = Buffer.wrap(Uint8List.fromList(bytes)); + final cursor = GeneratedReadCursor.start(cursorBuffer); + expect(cursorRead(cursor), equals(value)); + cursor.finish(); + expect(cursorBuffer.readableBytes, equals(0)); + } +} diff --git a/dart/packages/fory/test/codegen_conversion_expression_test.dart b/dart/packages/fory/test/codegen_conversion_expression_test.dart index 1da2039cd4..7982787cc7 100644 --- a/dart/packages/fory/test/codegen_conversion_expression_test.dart +++ b/dart/packages/fory/test/codegen_conversion_expression_test.dart @@ -15,6 +15,9 @@ // specific language governing permissions and limitations // under the License. +@TestOn('vm') +library; + import 'package:analyzer/dart/element/nullability_suffix.dart'; import 'package:analyzer/dart/element/type.dart'; import 'package:fory/fory.dart'; diff --git a/dart/packages/fory/test/collection_serializer_test.dart b/dart/packages/fory/test/collection_serializer_test.dart index 4c6fb77390..573bc76b9a 100644 --- a/dart/packages/fory/test/collection_serializer_test.dart +++ b/dart/packages/fory/test/collection_serializer_test.dart @@ -22,6 +22,58 @@ import 'dart:convert'; import 'package:fory/fory.dart'; import 'package:test/test.dart'; +part 'collection_serializer_test.fory.dart'; + +final Int64 _int64Min = Int64.parseHex('8000000000000000'); +final Int64 _int64Max = Int64.parseHex('7fffffffffffffff'); +final Uint64 _uint64HighBit = Uint64.parseHex('8000000000000000'); +final Uint64 _uint64Max = Uint64.parseHex('ffffffffffffffff'); + +@ForyStruct() +class NumericContainerEnvelope { + NumericContainerEnvelope(); + + List int64s = []; + List uint64s = []; + Map int64ByName = {}; + Map uint64ByName = {}; +} + +void _registerNumericContainerEnvelope(Fory fory) { + CollectionSerializerTestFory.register( + fory, + NumericContainerEnvelope, + namespace: 'test', + typeName: 'NumericContainerEnvelope', + ); +} + +NumericContainerEnvelope _numericContainerEnvelope() { + return NumericContainerEnvelope() + ..int64s = [Int64(-1), Int64(0), _int64Min, _int64Max] + ..uint64s = [Uint64(0), Uint64(1), _uint64HighBit, _uint64Max] + ..int64ByName = { + 'negative': Int64(-1), + 'min': _int64Min, + 'max': _int64Max, + } + ..uint64ByName = { + 'zero': Uint64(0), + 'highBit': _uint64HighBit, + 'max': _uint64Max, + }; +} + +void _expectNumericContainerEqual( + NumericContainerEnvelope actual, + NumericContainerEnvelope expected, +) { + expect(actual.int64s, equals(expected.int64s)); + expect(actual.uint64s, equals(expected.uint64s)); + expect(actual.int64ByName, equals(expected.int64ByName)); + expect(actual.uint64ByName, equals(expected.uint64ByName)); +} + void main() { group('collection serializer', () { test('round-trips empty root containers', () { @@ -262,6 +314,19 @@ void main() { } }); + test('round-trips generated Int64 and Uint64 list/map fields', () { + for (final compatible in [false, true]) { + final fory = Fory(compatible: compatible); + _registerNumericContainerEnvelope(fory); + + final value = _numericContainerEnvelope(); + final roundTrip = fory.deserialize( + fory.serialize(value), + ); + _expectNumericContainerEqual(roundTrip, value); + } + }); + test('enforces maxCollectionSize for list set and map', () { final fory = Fory(maxCollectionSize: 2); diff --git a/dart/packages/fory/test/enum_union_serializer_test.dart b/dart/packages/fory/test/enum_union_serializer_test.dart index 1b65d85f09..2d1a822201 100644 --- a/dart/packages/fory/test/enum_union_serializer_test.dart +++ b/dart/packages/fory/test/enum_union_serializer_test.dart @@ -95,7 +95,7 @@ final class TestUnion { factory TestUnion.ofString(String value) => TestUnion._(0, value); - factory TestUnion.ofInt(int value) => TestUnion._(1, value); + factory TestUnion.ofInt(Int64 value) => TestUnion._(1, value); factory TestUnion.ofLeaf(UnionLeaf value) => TestUnion._(2, value); @@ -122,9 +122,12 @@ final class TestUnionSerializer extends UnionSerializer { if (index == 0 && value is String) { return TestUnion.ofString(value); } - if (index == 1 && value is int) { + if (index == 1 && value is Int64) { return TestUnion.ofInt(value); } + if (index == 1 && value is int) { + return TestUnion.ofInt(Int64(value)); + } if (index == 2 && value is UnionLeaf) { return TestUnion.ofLeaf(value); } @@ -230,7 +233,7 @@ void main() { final cases = [ TestUnion.ofString('alpha'), - TestUnion.ofInt(1234), + TestUnion.ofInt(Int64(1234)), TestUnion.ofLeaf(UnionLeaf()..label = 'leaf'), ]; diff --git a/dart/packages/fory/test/int64_uint64_arithmetic_test.dart b/dart/packages/fory/test/int64_uint64_arithmetic_test.dart new file mode 100644 index 0000000000..c75e00e228 --- /dev/null +++ b/dart/packages/fory/test/int64_uint64_arithmetic_test.dart @@ -0,0 +1,583 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import 'dart:typed_data'; + +import 'package:fory/fory.dart'; +import 'package:test/test.dart'; + +const bool _isWeb = bool.fromEnvironment('dart.library.js_util'); + +final BigInt _mask32 = (BigInt.one << 32) - BigInt.one; +final BigInt _mask64 = (BigInt.one << 64) - BigInt.one; +final BigInt _signBit64 = BigInt.one << 63; +final BigInt _jsSafeMax = (BigInt.one << 53) - BigInt.one; +final BigInt _jsSafeMin = -_jsSafeMax; + +BigInt _normalizeSigned64(BigInt value) { + final masked = value & _mask64; + return masked & _signBit64 == BigInt.zero + ? masked + : masked - (BigInt.one << 64); +} + +BigInt _normalizeUnsigned64(BigInt value) => value & _mask64; + +bool _isJsSafeSigned64(BigInt value) => + value >= _jsSafeMin && value <= _jsSafeMax; + +bool _isJsSafeUnsigned64(BigInt value) => + value >= BigInt.zero && value <= _jsSafeMax; + +void _expectInt64State( + Int64 actual, + BigInt expected, { + String? reason, +}) { + final normalized = _normalizeSigned64(expected); + final low32 = (normalized & _mask32).toInt(); + final high32 = ((normalized >> 32) & _mask32).toInt(); + + expect(actual.toBigInt(), equals(normalized), reason: reason); + expect(actual.low32, equals(low32), reason: reason); + expect(actual.high32Unsigned, equals(high32), reason: reason); + expect(actual.high32Signed, equals(high32.toSigned(32)), reason: reason); + expect(actual.isZero, equals(normalized == BigInt.zero), reason: reason); + expect(actual.isNegative, equals(normalized.isNegative), reason: reason); + + if (_isWeb && !_isJsSafeSigned64(normalized)) { + expect(() => actual.toInt(), throwsStateError, reason: reason); + return; + } + expect(BigInt.from(actual.toInt()), equals(normalized), reason: reason); +} + +void _expectUint64State( + Uint64 actual, + BigInt expected, { + String? reason, +}) { + final normalized = _normalizeUnsigned64(expected); + final low32 = (normalized & _mask32).toInt(); + final high32 = ((normalized >> 32) & _mask32).toInt(); + + expect(actual.toBigInt(), equals(normalized), reason: reason); + expect(actual.low32, equals(low32), reason: reason); + expect(actual.high32Unsigned, equals(high32), reason: reason); + expect(actual.isZero, equals(normalized == BigInt.zero), reason: reason); + + if (_isWeb) { + if (!_isJsSafeUnsigned64(normalized)) { + expect(() => actual.toInt(), throwsStateError, reason: reason); + return; + } + } else if (normalized >= _signBit64) { + expect(() => actual.toInt(), throwsStateError, reason: reason); + return; + } + + expect(BigInt.from(actual.toInt()), equals(normalized), reason: reason); +} + +void main() { + group('Int64', () { + test('constructors normalize words and hex edge cases', () { + final cases = <({String name, Int64 value, BigInt expected})>[ + (name: 'zero', value: Int64.fromWords(0, 0), expected: BigInt.zero), + (name: 'one', value: Int64.fromWords(1, 0), expected: BigInt.one), + ( + name: '2^22', + value: Int64.fromWords(0x00400000, 0), + expected: BigInt.one << 22, + ), + ( + name: '2^23', + value: Int64.fromWords(0x00800000, 0), + expected: BigInt.one << 23, + ), + ( + name: '2^31', + value: Int64.fromWords(0x80000000, 0), + expected: BigInt.one << 31, + ), + ( + name: '2^32', + value: Int64.fromWords(0, 1), + expected: BigInt.one << 32, + ), + ( + name: '2^43', + value: Int64.fromWords(0, 0x00000800), + expected: BigInt.one << 43, + ), + ( + name: '2^44', + value: Int64.fromWords(0, 0x00001000), + expected: BigInt.one << 44, + ), + ( + name: '2^45', + value: Int64.fromWords(0, 0x00002000), + expected: BigInt.one << 45, + ), + ( + name: '2^62', + value: Int64.fromWords(0, 0x40000000), + expected: BigInt.one << 62, + ), + ( + name: 'signed min', + value: Int64.fromWords(0, 0x80000000), + expected: -(BigInt.one << 63), + ), + ( + name: 'signed max', + value: Int64.fromWords(0xffffffff, 0x7fffffff), + expected: (BigInt.one << 63) - BigInt.one, + ), + ( + name: 'negative one', + value: Int64.fromWords(0xffffffff, 0xffffffff), + expected: -BigInt.one, + ), + ( + name: 'parseHex signed min', + value: Int64.parseHex('8000000000000000'), + expected: -(BigInt.one << 63), + ), + ( + name: 'parseHex negative', + value: Int64.parseHex('-7'), + expected: BigInt.from(-7), + ), + ( + name: 'parseHex wraparound', + value: Int64.parseHex('ffffffffffffffff'), + expected: -BigInt.one, + ), + ]; + + for (final testCase in cases) { + _expectInt64State( + testCase.value, + testCase.expected, + reason: testCase.name, + ); + } + }); + + test('equality and compare respect signed ordering', () { + final values = [ + Int64.parseHex('8000000000000000'), + Int64.parseHex('ffffffffffffffff'), + Int64(0), + Int64(1), + Int64.parseHex('7fffffffffffffff'), + ]; + + expect(values[0] == values[0], isTrue); + expect(values[0].compareTo(values[1]), lessThan(0)); + expect(values[1].compareTo(values[2]), lessThan(0)); + expect(values[2].compareTo(values[3]), lessThan(0)); + expect(values[3].compareTo(values[4]), lessThan(0)); + expect(values[4].compareTo(values[4]), equals(0)); + expect(values[1], equals(Int64.fromWords(0xffffffff, 0xffffffff))); + }); + + test('arithmetic wraps, borrows, and truncates with sign edge cases', () { + _expectInt64State( + Int64.parseHex('7fffffffffffffff') + 1, + -(BigInt.one << 63), + reason: 'signed overflow add', + ); + _expectInt64State( + Int64.parseHex('8000000000000000') - 1, + (BigInt.one << 63) - BigInt.one, + reason: 'signed underflow subtract', + ); + _expectInt64State( + Int64(-1) + Int64(1), + BigInt.zero, + reason: 'wrapper addition', + ); + _expectInt64State( + Int64(0) - 1, + -BigInt.one, + reason: 'borrow from zero', + ); + _expectInt64State( + Int64.parseHex('4000000000000000') * 2, + -(BigInt.one << 63), + reason: 'multiply overflow', + ); + _expectInt64State( + Int64.parseHex('8000000000000000') * -1, + -(BigInt.one << 63), + reason: 'multiply signed min by -1 wraps', + ); + _expectInt64State( + Int64.parseHex('-7') ~/ 3, + BigInt.from(-2), + reason: 'truncating division toward zero', + ); + _expectInt64State( + Int64.parseHex('-7') % 3, + BigInt.from(2), + reason: 'remainder follows Dart int semantics', + ); + expect(Int64.parseHex('-7') / 2, equals(-3.5)); + expect((-Int64(1)).toBigInt(), equals(BigInt.from(-1))); + expect((-Int64.parseHex('8000000000000000')).toBigInt(), + equals(-(BigInt.one << 63))); + }); + + test('bitwise operators and shifts cross segment boundaries', () { + final left = Int64.parseHex('f23456789abcdef0'); + final right = Int64.parseHex('00ff00ff00ff00ff'); + final leftBig = left.toBigInt(); + final rightBig = right.toBigInt(); + + _expectInt64State(left & right, _normalizeSigned64(leftBig & rightBig)); + _expectInt64State(left | right, _normalizeSigned64(leftBig | rightBig)); + _expectInt64State(left ^ right, _normalizeSigned64(leftBig ^ rightBig)); + _expectInt64State(~left, _normalizeSigned64(~leftBig)); + _expectInt64State(Int64(0x7f) & 0x3, BigInt.from(0x3)); + _expectInt64State(Int64(0x7f) | Int64(0x80), BigInt.from(0xff)); + _expectInt64State(Int64(0x7f) ^ 0x3, BigInt.from(0x7c)); + + const shifts = [ + 0, + 1, + 21, + 22, + 23, + 31, + 32, + 43, + 44, + 45, + 62, + 63, + 64, + 65 + ]; + for (final shift in shifts) { + _expectInt64State( + left << shift, + _normalizeSigned64(leftBig << shift), + reason: 'left shift $shift', + ); + _expectInt64State( + left >> shift, + _normalizeSigned64(leftBig >> shift), + reason: 'right shift $shift', + ); + _expectInt64State( + left >>> shift, + _normalizeSigned64(_normalizeUnsigned64(leftBig) >> shift), + reason: 'logical shift $shift', + ); + } + + expect(() => left << -1, throwsArgumentError); + expect(() => left >> -1, throwsArgumentError); + expect(() => left >>> -1, throwsArgumentError); + }); + + test('toInt respects JS-safe and platform-specific unsafe boundaries', () { + _expectInt64State( + Int64.parseHex('1fffffffffffff'), + _jsSafeMax, + reason: 'safe positive max', + ); + _expectInt64State( + Int64.parseHex('-1fffffffffffff'), + _jsSafeMin, + reason: 'safe negative min', + ); + _expectInt64State( + Int64.parseHex('20000000000000'), + BigInt.one << 53, + reason: 'unsafe positive boundary', + ); + _expectInt64State( + Int64.parseHex('-20000000000000'), + -(BigInt.one << 53), + reason: 'unsafe negative boundary', + ); + }); + + test('list sublistView respects source typed-data element ranges', () { + final bytes = Uint8List(24); + final data = ByteData.sublistView(bytes); + data.setUint32(8, 0x76543210, Endian.little); + data.setUint32(12, 0x01234567, Endian.little); + + final byteView = Int64List.sublistView(bytes, 8, 16); + expect(byteView.length, equals(1)); + expect(byteView[0], equals(Int64.fromWords(0x76543210, 0x01234567))); + + byteView[0] = Int64.fromWords(0x89abcdef, 0x76543210); + expect(data.getUint32(8, Endian.little), equals(0x89abcdef)); + expect(data.getUint32(12, Endian.little), equals(0x76543210)); + + final wordView = + Int64List.sublistView(Uint32List.view(bytes.buffer), 2, 4); + expect(wordView.length, equals(1)); + expect(wordView[0], equals(Int64.fromWords(0x89abcdef, 0x76543210))); + + expect(() => Int64List.sublistView(bytes, 1, 9), throwsArgumentError); + expect(() => Int64List.sublistView(bytes, 8, 15), throwsArgumentError); + }); + }); + + group('Uint64', () { + test('constructors normalize words and hex edge cases', () { + final cases = <({String name, Uint64 value, BigInt expected})>[ + (name: 'zero', value: Uint64.fromWords(0, 0), expected: BigInt.zero), + (name: 'one', value: Uint64.fromWords(1, 0), expected: BigInt.one), + ( + name: '2^22', + value: Uint64.fromWords(0x00400000, 0), + expected: BigInt.one << 22, + ), + ( + name: '2^23', + value: Uint64.fromWords(0x00800000, 0), + expected: BigInt.one << 23, + ), + ( + name: '2^31', + value: Uint64.fromWords(0x80000000, 0), + expected: BigInt.one << 31, + ), + ( + name: '2^32', + value: Uint64.fromWords(0, 1), + expected: BigInt.one << 32, + ), + ( + name: '2^43', + value: Uint64.fromWords(0, 0x00000800), + expected: BigInt.one << 43, + ), + ( + name: '2^44', + value: Uint64.fromWords(0, 0x00001000), + expected: BigInt.one << 44, + ), + ( + name: '2^45', + value: Uint64.fromWords(0, 0x00002000), + expected: BigInt.one << 45, + ), + ( + name: '2^62', + value: Uint64.fromWords(0, 0x40000000), + expected: BigInt.one << 62, + ), + ( + name: '2^63', + value: Uint64.fromWords(0, 0x80000000), + expected: BigInt.one << 63, + ), + ( + name: 'max', + value: Uint64.fromWords(0xffffffff, 0xffffffff), + expected: _mask64, + ), + ( + name: 'parseHex max', + value: Uint64.parseHex('ffffffffffffffff'), + expected: _mask64, + ), + ( + name: 'parseHex negative one wraps', + value: Uint64.parseHex('-1'), + expected: _mask64, + ), + ]; + + for (final testCase in cases) { + _expectUint64State( + testCase.value, + testCase.expected, + reason: testCase.name, + ); + } + }); + + test('equality and compare respect unsigned ordering', () { + final values = [ + Uint64(0), + Uint64(1), + Uint64.parseHex('7fffffffffffffff'), + Uint64.parseHex('8000000000000000'), + Uint64.parseHex('ffffffffffffffff'), + ]; + + expect(values[0] == values[0], isTrue); + expect(values[0].compareTo(values[1]), lessThan(0)); + expect(values[1].compareTo(values[2]), lessThan(0)); + expect(values[2].compareTo(values[3]), lessThan(0)); + expect(values[3].compareTo(values[4]), lessThan(0)); + expect(values[4].compareTo(values[4]), equals(0)); + expect(values[4], equals(Uint64.fromWords(0xffffffff, 0xffffffff))); + }); + + test('arithmetic wraps, borrows, and truncates across the full range', () { + _expectUint64State( + Uint64.parseHex('ffffffffffffffff') + 1, + BigInt.zero, + reason: 'overflow add', + ); + _expectUint64State( + Uint64(0) - 1, + _mask64, + reason: 'borrow from zero', + ); + _expectUint64State( + Uint64.parseHex('8000000000000000') * 2, + BigInt.zero, + reason: 'multiply overflow', + ); + _expectUint64State( + Uint64.parseHex('ffffffffffffffff') * 2, + _mask64 - BigInt.one, + reason: 'multiply max by two', + ); + _expectUint64State( + Uint64.parseHex('ffffffffffffffff') ~/ 3, + _mask64 ~/ BigInt.from(3), + reason: 'truncating division', + ); + _expectUint64State( + Uint64.parseHex('ffffffffffffffff') % 3, + BigInt.from(0), + reason: 'remainder against three', + ); + expect(Uint64.parseHex('7') / 2, equals(3.5)); + expect((-Uint64(1)).toBigInt(), equals(_mask64)); + expect((-Uint64.parseHex('8000000000000000')).toBigInt(), + equals(BigInt.one << 63)); + }); + + test('bitwise operators and shifts cross segment boundaries', () { + final left = Uint64.parseHex('f23456789abcdef0'); + final right = Uint64.parseHex('00ff00ff00ff00ff'); + final leftBig = left.toBigInt(); + final rightBig = right.toBigInt(); + + _expectUint64State( + left & right, _normalizeUnsigned64(leftBig & rightBig)); + _expectUint64State( + left | right, _normalizeUnsigned64(leftBig | rightBig)); + _expectUint64State( + left ^ right, _normalizeUnsigned64(leftBig ^ rightBig)); + _expectUint64State(~left, _normalizeUnsigned64(~leftBig)); + _expectUint64State(Uint64(0x7f) & 0x3, BigInt.from(0x3)); + _expectUint64State(Uint64(0x7f) | Uint64(0x80), BigInt.from(0xff)); + _expectUint64State(Uint64(0x7f) ^ 0x3, BigInt.from(0x7c)); + + const shifts = [ + 0, + 1, + 21, + 22, + 23, + 31, + 32, + 43, + 44, + 45, + 62, + 63, + 64, + 65 + ]; + for (final shift in shifts) { + _expectUint64State( + left << shift, + _normalizeUnsigned64(leftBig << shift), + reason: 'left shift $shift', + ); + _expectUint64State( + left >> shift, + _normalizeUnsigned64(leftBig >> shift), + reason: 'right shift $shift', + ); + _expectUint64State( + left >>> shift, + _normalizeUnsigned64(leftBig >> shift), + reason: 'logical shift $shift', + ); + } + + expect(() => left << -1, throwsArgumentError); + expect(() => left >> -1, throwsArgumentError); + expect(() => left >>> -1, throwsArgumentError); + }); + + test('toInt respects JS-safe and native unsigned boundaries', () { + _expectUint64State( + Uint64.parseHex('1fffffffffffff'), + _jsSafeMax, + reason: 'safe positive max', + ); + _expectUint64State( + Uint64.parseHex('20000000000000'), + BigInt.one << 53, + reason: 'unsafe positive boundary', + ); + _expectUint64State( + Uint64.parseHex('8000000000000000'), + BigInt.one << 63, + reason: 'native-unsafe high-bit boundary', + ); + _expectUint64State( + Uint64.parseHex('ffffffffffffffff'), + _mask64, + reason: 'max boundary', + ); + }); + + test('list sublistView respects source typed-data element ranges', () { + final bytes = Uint8List(24); + final data = ByteData.sublistView(bytes); + data.setUint32(8, 0xfedcba98, Endian.little); + data.setUint32(12, 0x01234567, Endian.little); + + final byteView = Uint64List.sublistView(bytes, 8, 16); + expect(byteView.length, equals(1)); + expect(byteView[0], equals(Uint64.fromWords(0xfedcba98, 0x01234567))); + + byteView[0] = Uint64.fromWords(0x76543210, 0xfedcba98); + expect(data.getUint32(8, Endian.little), equals(0x76543210)); + expect(data.getUint32(12, Endian.little), equals(0xfedcba98)); + + final wordView = + Uint64List.sublistView(Uint32List.view(bytes.buffer), 2, 4); + expect(wordView.length, equals(1)); + expect(wordView[0], equals(Uint64.fromWords(0x76543210, 0xfedcba98))); + + expect(() => Uint64List.sublistView(bytes, 1, 9), throwsArgumentError); + expect(() => Uint64List.sublistView(bytes, 8, 15), throwsArgumentError); + }); + }); +} diff --git a/dart/packages/fory/test/manual_registration_test.dart b/dart/packages/fory/test/manual_registration_test.dart index e6604b2bf1..1558478e37 100644 --- a/dart/packages/fory/test/manual_registration_test.dart +++ b/dart/packages/fory/test/manual_registration_test.dart @@ -24,7 +24,7 @@ final class ManualValue { ManualValue(this.name, this.score); final String name; - final int score; + final Int64 score; } final class ManualValueSerializer extends Serializer { @@ -121,11 +121,11 @@ void main() { typeName: 'ManualValue', ); - final value = ManualValue('alpha', 99); + final value = ManualValue('alpha', Int64(99)); final bytes = fory.serialize(value); final roundTrip = fory.deserialize(bytes); expect(roundTrip.name, equals('alpha')); - expect(roundTrip.score, equals(99)); + expect(roundTrip.score, equals(Int64(99))); }); test('writeNonRef does not seed later back-references', () { diff --git a/dart/packages/fory/test/numeric_wrapper_test.dart b/dart/packages/fory/test/numeric_wrapper_test.dart index a944761f87..969919cdf9 100644 --- a/dart/packages/fory/test/numeric_wrapper_test.dart +++ b/dart/packages/fory/test/numeric_wrapper_test.dart @@ -20,10 +20,21 @@ import 'dart:typed_data'; import 'package:fory/fory.dart'; +import 'package:fory/src/resolver/type_resolver.dart'; +import 'package:fory/src/serializer/compatible_struct_metadata.dart'; import 'package:test/test.dart'; part 'numeric_wrapper_test.fory.dart'; +Uint64 _u64Hex(String value) => Uint64.parseHex(value); + +double _float64FromWords(int low32, int high32) { + final bytes = ByteData(8) + ..setUint32(0, low32, Endian.little) + ..setUint32(4, high32, Endian.little); + return bytes.getFloat64(0, Endian.little); +} + @ForyStruct() class NumericWrappersEnvelope { NumericWrappersEnvelope(); @@ -31,6 +42,7 @@ class NumericWrappersEnvelope { Int8 i8 = Int8(0); Int16 i16 = Int16(0); Int32 i32 = Int32(0); + Int64 i64 = Int64(0); Uint8 u8 = Uint8(0); Uint16 u16 = Uint16(0); Uint32 u32 = Uint32(0); @@ -45,6 +57,13 @@ class NumericWrappersEnvelope { Float32? optionalSingle; } +@ForyStruct() +class NumericWrappersMetadataReader { + NumericWrappersMetadataReader(); + + Int8 i8 = Int8(0); +} + void _registerNumericWrappers(Fory fory) { NumericWrapperTestFory.register( fory, @@ -54,6 +73,15 @@ void _registerNumericWrappers(Fory fory) { ); } +void _registerNumericWrappersMetadataReader(Fory fory) { + NumericWrapperTestFory.register( + fory, + NumericWrappersMetadataReader, + namespace: 'test', + typeName: 'NumericWrappersEnvelope', + ); +} + T _roundTrip(Fory fory, T value) => fory.deserialize(fory.serialize(value)); @@ -62,15 +90,16 @@ NumericWrappersEnvelope _sampleEnvelope() { ..i8 = Int8(-127) ..i16 = Int16(0x7fff) ..i32 = Int32(-2147483648) + ..i64 = Int64.parseHex('8000000000000000') ..u8 = Uint8(0xff) ..u16 = Uint16(0xffff) ..u32 = Uint32(0xffffffff) - ..u64 = Uint64(0xffffffffffffffff) + ..u64 = _u64Hex('ffffffffffffffff') ..half = Float16.fromBits(0x3555) ..brain = Bfloat16.fromBits(0x3eab) ..single = Float32.fromBits(0x40490fdb) ..optionalI8 = Int8(126) - ..optionalU64 = Uint64(0x8000000000000000) + ..optionalU64 = _u64Hex('8000000000000000') ..optionalHalf = Float16.fromBits(0x8000) ..optionalBrain = Bfloat16.fromBits(0x8000) ..optionalSingle = Float32.fromBits(0x80000000); @@ -83,6 +112,7 @@ void _expectEnvelopeEquals( expect(actual.i8, equals(expected.i8)); expect(actual.i16, equals(expected.i16)); expect(actual.i32, equals(expected.i32)); + expect(actual.i64, equals(expected.i64)); expect(actual.u8, equals(expected.u8)); expect(actual.u16, equals(expected.u16)); expect(actual.u32, equals(expected.u32)); @@ -102,6 +132,15 @@ void _expectEnvelopeEquals( ); } +int _remoteFieldTypeId(Object value, String identifier) { + final remoteTypeDef = CompatibleStructMetadata.remoteTypeDefFor(value); + expect(remoteTypeDef, isNotNull); + final field = remoteTypeDef!.fields.firstWhere( + (field) => field.identifier == identifier, + ); + return field.fieldType.typeId; +} + void main() { group('numeric wrappers', () { test('signed integer wrappers normalize arithmetic and bitwise operations', @@ -143,10 +182,12 @@ void main() { expect( Uint32(0xf0f0f0f0) & Uint32(0x0ff00ff0), equals(Uint32(0x00f000f0))); - expect(Uint64(0xffffffffffffffff) + 1, equals(Uint64(0))); - expect(Uint64(0) - 1, equals(Uint64(0xffffffffffffffff))); + expect(_u64Hex('ffffffffffffffff') + 1, equals(Uint64(0))); + expect(Uint64(0) - 1, equals(_u64Hex('ffffffffffffffff'))); expect( - Uint64(0x123456789abcdef0) >> 4, equals(Uint64(0x0123456789abcdef))); + _u64Hex('123456789abcdef0') >> 4, + equals(_u64Hex('0123456789abcdef')), + ); expect(Uint64(0xff).toInt(), equals(0xff)); }); @@ -181,12 +222,8 @@ void main() { test( 'Bfloat16.fromDouble rounds directly from float64 and preserves NaN sign payload bits', () { - final subnormalSource = ByteData(8) - ..setUint64(0, 0x37da834f7e281cc1, Endian.little); - final trickySubnormal = subnormalSource.getFloat64(0, Endian.little); - final nanSource = ByteData(8) - ..setUint64(0, 0xfff123456789abcd, Endian.little); - final payloadNaN = nanSource.getFloat64(0, Endian.little); + final trickySubnormal = _float64FromWords(0x7e281cc1, 0x37da834f); + final payloadNaN = _float64FromWords(0x6789abcd, 0xfff12345); final convertedNaN = Bfloat16.fromDouble(payloadNaN); expect(Bfloat16.fromDouble(trickySubnormal).toBits(), equals(0x0007)); @@ -221,10 +258,104 @@ void main() { expect(_roundTrip(fory, Uint8(-1)), equals(Uint8(0xff))); expect(_roundTrip(fory, Uint16(-1)), equals(Uint16(0xffff))); expect(_roundTrip(fory, Uint32(-1)), equals(Uint32(0xffffffff))); + expect(_roundTrip(fory, Int64(-1)), equals(Int64(-1))); + expect( + _roundTrip(fory, Int64.parseHex('8000000000000000')), + equals(Int64.parseHex('8000000000000000')), + ); + expect( + _roundTrip(fory, Int64.parseHex('7fffffffffffffff')), + equals(Int64.parseHex('7fffffffffffffff')), + ); expect( _roundTrip(fory, Uint64(-1)), - equals(Uint64(0xffffffffffffffff)), + equals(_u64Hex('ffffffffffffffff')), + ); + }); + + test('resolves root numeric wrappers to compact varint wire ids', () { + final resolver = TypeResolver(Config()); + + expect(resolver.resolveValue(1).typeId, equals(TypeIds.varInt64)); + expect(resolver.resolveValue(Int32(1)).typeId, equals(TypeIds.varInt32)); + expect(resolver.resolveValue(Int64(1)).typeId, equals(TypeIds.varInt64)); + expect( + resolver.resolveValue(Uint32(1)).typeId, + equals(TypeIds.varUint32), + ); + if (identical(1, 1.0)) { + expect( + resolver.resolveValue(Uint64(1)).typeId, + equals(TypeIds.varUint64), + ); + } else { + expect( + resolver.resolveValue(Uint64(1)).typeId, + equals(TypeIds.varInt64), + reason: 'Native Uint64 is an extension type represented as int.', + ); + } + }); + + test('typed root Int64 reads preserve wrappers', () { + final fory = Fory(); + + expect( + fory.deserialize(fory.serialize(1)), + equals(1), + reason: 'Plain int roots still decode as Dart int.', + ); + expect( + fory.deserialize(fory.serialize(Int64(-1))), + equals(Int64(-1)), + ); + expect( + fory.deserialize( + fory.serialize(Int64.parseHex('8000000000000000')), + ), + equals(Int64.parseHex('8000000000000000')), + ); + expect( + fory.deserialize( + fory.serialize(Int64.parseHex('7fffffffffffffff')), + ), + equals(Int64.parseHex('7fffffffffffffff')), + ); + expect( + fory.deserialize( + fory.serialize(Int64.parseHex('8000000000000000')), + ), + equals(Int64.parseHex('8000000000000000')), + ); + expect( + fory.deserialize(fory.serialize(Int64(-1))), + equals(-1), + reason: 'Nullable int roots still decode as Dart int.', + ); + }); + + test('web dynamic Uint64 wrappers keep unsigned metadata', () { + if (!identical(1, 1.0)) { + return; + } + + final fory = Fory(); + final value = _u64Hex('ffffffffffffffff'); + + expect( + fory.deserialize(fory.serialize(value)), + equals(value), ); + + final list = fory.deserialize( + fory.serialize([value]), + ) as List; + expect(list.single, equals(value)); + + final map = fory.deserialize( + fory.serialize({'value': value}), + ) as Map; + expect(map['value'], equals(value)); }); test('round-trips root Float16 payloads with exact bits', () { @@ -301,6 +432,23 @@ void main() { _expectEnvelopeEquals(roundTrip, value); }); + test('compatible metadata records numeric wrapper varint defaults', () { + final writer = Fory(compatible: true); + final reader = Fory(compatible: true); + _registerNumericWrappers(writer); + _registerNumericWrappersMetadataReader(reader); + + final roundTrip = reader.deserialize( + writer.serialize(_sampleEnvelope()), + ); + + expect(roundTrip.i8, equals(Int8(-127))); + expect(_remoteFieldTypeId(roundTrip, 'i32'), equals(TypeIds.varInt32)); + expect(_remoteFieldTypeId(roundTrip, 'i64'), equals(TypeIds.varInt64)); + expect(_remoteFieldTypeId(roundTrip, 'u32'), equals(TypeIds.varUint32)); + expect(_remoteFieldTypeId(roundTrip, 'u64'), equals(TypeIds.varUint64)); + }); + test('supports null optional numeric wrapper fields', () { final fory = Fory(); _registerNumericWrappers(fory); diff --git a/dart/packages/fory/test/scalar_and_typed_array_serializer_test.dart b/dart/packages/fory/test/scalar_and_typed_array_serializer_test.dart index 2a03eb0806..7f7d4013c1 100644 --- a/dart/packages/fory/test/scalar_and_typed_array_serializer_test.dart +++ b/dart/packages/fory/test/scalar_and_typed_array_serializer_test.dart @@ -24,6 +24,14 @@ import 'package:test/test.dart'; part 'scalar_and_typed_array_serializer_test.fory.dart'; +Timestamp _timestamp(int seconds, int nanoseconds) => + Timestamp(Int64(seconds), nanoseconds); + +final Int64 _int64Min = Int64.parseHex('8000000000000000'); +final Int64 _int64Max = Int64.parseHex('7fffffffffffffff'); +final Uint64 _uint64HighBit = Uint64.parseHex('8000000000000000'); +final Uint64 _uint64Max = Uint64.parseHex('ffffffffffffffff'); + @ForyStruct() class ScalarAndArrayEnvelope { ScalarAndArrayEnvelope(); @@ -45,7 +53,7 @@ class ScalarAndArrayEnvelope { Bfloat16 brain = Bfloat16(0); Float32 single = Float32(0); LocalDate date = const LocalDate(1970, 1, 1); - Timestamp timestamp = const Timestamp(0, 0); + Timestamp timestamp = _timestamp(0, 0); } void _registerScalarTypes(Fory fory) { @@ -67,10 +75,13 @@ ScalarAndArrayEnvelope _sampleEnvelope() { ..int8s = Int8List.fromList([-128, -1, 0, 127]) ..int16s = Int16List.fromList([-32768, -1, 0, 32767]) ..int32s = Int32List.fromList([-1, 0, 1, 123456789]) - ..int64s = Int64List.fromList([-1, 0, 1, 1 << 40]) + ..int64s = + Int64List.fromList([-1, 0, 1, 1 << 40, _int64Min, _int64Max]) ..uint16s = Uint16List.fromList([0, 1, 65535]) ..uint32s = Uint32List.fromList([0, 1, 0x7fffffff]) - ..uint64s = Uint64List.fromList([0, 1, 1 << 40]) + ..uint64s = Uint64List.fromList( + [0, 1, 1 << 40, _uint64HighBit, _uint64Max], + ) ..float16s = Float16List.fromList([ Float16.fromBits(0x8000), Float16.fromBits(0x3555), @@ -87,8 +98,8 @@ ScalarAndArrayEnvelope _sampleEnvelope() { ..half = const Float16.fromBits(0x8000) ..brain = const Bfloat16.fromBits(0x7fc0) ..single = Float32(3.5) - ..date = LocalDate.fromEpochDay(-1) - ..timestamp = const Timestamp(-123, 456789123); + ..date = LocalDate.fromEpochDay(Int64(-1)) + ..timestamp = _timestamp(-123, 456789123); } void _expectUint8ListEquals(Uint8List actual, Uint8List expected) { @@ -197,7 +208,7 @@ void main() { final beforeEpoch = _roundTripRoot( fory, - LocalDate.fromEpochDay(-1), + LocalDate.fromEpochDay(Int64(-1)), ); final leapDay = _roundTripRoot( fory, @@ -205,18 +216,18 @@ void main() { ); final negativeTimestamp = _roundTripRoot( fory, - const Timestamp(-123, 456789000), + _timestamp(-123, 456789000), ); final fromDateTime = _roundTripRoot( fory, Timestamp.fromDateTime(DateTime.utc(2024, 1, 2, 3, 4, 5, 6, 700)), ); - expect(beforeEpoch, equals(LocalDate.fromEpochDay(-1))); + expect(beforeEpoch, equals(LocalDate.fromEpochDay(Int64(-1)))); expect(leapDay, equals(const LocalDate(2024, 2, 29))); expect( negativeTimestamp, - equals(const Timestamp(-123, 456789000).toDateTime()), + equals(_timestamp(-123, 456789000).toDateTime()), ); expect( fromDateTime, @@ -255,9 +266,23 @@ void main() { _expectInt64ListEquals( _roundTripRoot( fory, - Int64List.fromList([-1, 0, 1, 1 << 40]), + Int64List.fromList([ + -1, + 0, + 1, + 1 << 40, + _int64Min, + _int64Max, + ]), ), - Int64List.fromList([-1, 0, 1, 1 << 40]), + Int64List.fromList([ + -1, + 0, + 1, + 1 << 40, + _int64Min, + _int64Max, + ]), ); _expectUint16ListEquals( _roundTripRoot( @@ -276,9 +301,21 @@ void main() { _expectUint64ListEquals( _roundTripRoot( fory, - Uint64List.fromList([0, 1, 1 << 40]), + Uint64List.fromList([ + 0, + 1, + 1 << 40, + _uint64HighBit, + _uint64Max, + ]), ), - Uint64List.fromList([0, 1, 1 << 40]), + Uint64List.fromList([ + 0, + 1, + 1 << 40, + _uint64HighBit, + _uint64Max, + ]), ); _expectFloat16ListEquals( _roundTripRoot( diff --git a/dart/packages/fory/test/signed_serializer_test.dart b/dart/packages/fory/test/signed_serializer_test.dart new file mode 100644 index 0000000000..176854f368 --- /dev/null +++ b/dart/packages/fory/test/signed_serializer_test.dart @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import 'package:fory/fory.dart'; +import 'package:fory/src/serializer/compatible_struct_metadata.dart'; +import 'package:test/test.dart'; + +part 'signed_serializer_test.fory.dart'; + +const int _jsSafeIntMax = 9007199254740991; +const int _jsSafeIntMin = -9007199254740991; +const int _jsUnsafeInt = 9007199254740992; +final Int64 _int64Min = Int64.parseHex('8000000000000000'); +final Int64 _int64Max = Int64.parseHex('7fffffffffffffff'); + +@ForyStruct() +class SignedFields { + SignedFields(); + + int plainInt = 0; + + @Int32Type(compress: true) + int i32Var = 0; + + @Int32Type(compress: false) + int i32Fixed = 0; + + @Int64Type(encoding: LongEncoding.varint) + int i64VarInt = 0; + + @Int64Type(encoding: LongEncoding.fixed) + int i64FixedInt = 0; + + @Int64Type(encoding: LongEncoding.tagged) + int i64TaggedInt = 0; + + Int64 i64Default = Int64(0); + + @Int64Type(encoding: LongEncoding.varint) + Int64 i64Var = Int64(0); + + @Int64Type(encoding: LongEncoding.fixed) + Int64 i64Fixed = Int64(0); + + @Int64Type(encoding: LongEncoding.tagged) + Int64 i64Tagged = Int64(0); + + @Int64Type(encoding: LongEncoding.varint) + int? optionalI64VarInt; + + @Int32Type(compress: true) + int? optionalI32Var; + + @Int32Type(compress: false) + int? optionalI32Fixed; + + @Int64Type(encoding: LongEncoding.fixed) + Int64? optionalI64Fixed; + + @Int64Type(encoding: LongEncoding.tagged) + Int64? optionalI64Tagged; +} + +@ForyStruct() +class SignedMetadataReader { + SignedMetadataReader(); + + int plainInt = 0; +} + +@ForyStruct() +class SignedIntFieldsReader { + SignedIntFieldsReader(); + + @Int64Type(encoding: LongEncoding.varint) + int i64Var = 0; + + @Int64Type(encoding: LongEncoding.fixed) + int i64Fixed = 0; + + @Int64Type(encoding: LongEncoding.tagged) + int i64Tagged = 0; +} + +void _registerSignedFields(Fory fory) { + SignedSerializerTestFory.register( + fory, + SignedFields, + namespace: 'test', + typeName: 'SignedFields', + ); +} + +void _registerSignedMetadataReader(Fory fory) { + SignedSerializerTestFory.register( + fory, + SignedMetadataReader, + namespace: 'test', + typeName: 'SignedFields', + ); +} + +void _registerSignedIntFieldsReader(Fory fory) { + SignedSerializerTestFory.register( + fory, + SignedIntFieldsReader, + namespace: 'test', + typeName: 'SignedFields', + ); +} + +SignedFields _smallSignedFields() { + return SignedFields() + ..plainInt = -1 + ..i32Var = -64 + ..i32Fixed = 63 + ..i64VarInt = -64 + ..i64FixedInt = 63 + ..i64TaggedInt = 0x3fffffff + ..i64Default = Int64(-1) + ..i64Var = Int64(1) + ..i64Fixed = Int64(-0x40000000) + ..i64Tagged = Int64(0x3fffffff) + ..optionalI32Var = -128 + ..optionalI32Fixed = 0x40000000 + ..optionalI64VarInt = -128 + ..optionalI64Fixed = Int64(0x40000000) + ..optionalI64Tagged = Int64(-0x40000001); +} + +SignedFields _jsSafeBoundarySignedFields() { + return SignedFields() + ..plainInt = _jsSafeIntMin + ..i32Var = -0x80000000 + ..i32Fixed = 0x7fffffff + ..i64VarInt = _jsSafeIntMax + ..i64FixedInt = _jsSafeIntMin + ..i64TaggedInt = _jsSafeIntMax + ..i64Default = Int64(_jsSafeIntMin) + ..i64Var = Int64(_jsSafeIntMax) + ..i64Fixed = Int64(_jsSafeIntMin) + ..i64Tagged = Int64(_jsSafeIntMax) + ..optionalI32Var = -0x80000000 + ..optionalI32Fixed = 0x7fffffff + ..optionalI64VarInt = _jsSafeIntMin + ..optionalI64Fixed = Int64(_jsSafeIntMax) + ..optionalI64Tagged = Int64(_jsSafeIntMin); +} + +SignedFields _fullRangeWrapperSignedFields() { + return SignedFields() + ..plainInt = 0 + ..i32Var = -0x80000000 + ..i32Fixed = 0x7fffffff + ..i64VarInt = -0x40000001 + ..i64FixedInt = 0x40000000 + ..i64TaggedInt = -0x40000001 + ..i64Default = _int64Min + ..i64Var = _int64Max + ..i64Fixed = _int64Min + ..i64Tagged = _int64Max + ..optionalI32Var = -0x80000000 + ..optionalI32Fixed = 0x7fffffff + ..optionalI64VarInt = 0x40000000 + ..optionalI64Fixed = _int64Max + ..optionalI64Tagged = _int64Min; +} + +SignedFields _nullSignedFields() { + return SignedFields() + ..plainInt = 1 + ..i32Var = 2 + ..i32Fixed = 3 + ..i64VarInt = 2 + ..i64FixedInt = 3 + ..i64TaggedInt = 4 + ..i64Default = Int64(5) + ..i64Var = Int64(6) + ..i64Fixed = Int64(7) + ..i64Tagged = Int64(8) + ..optionalI32Var = null + ..optionalI32Fixed = null + ..optionalI64VarInt = null + ..optionalI64Fixed = null + ..optionalI64Tagged = null; +} + +SignedFields _int64ReadMismatchPayload(String encoding) { + final value = _smallSignedFields() + ..i64Var = Int64(1) + ..i64Fixed = Int64(1) + ..i64Tagged = Int64(1); + switch (encoding) { + case 'varint': + value.i64Var = _int64Min; + case 'fixed': + value.i64Fixed = _int64Min; + case 'tagged': + value.i64Tagged = _int64Min; + default: + throw ArgumentError.value(encoding, 'encoding'); + } + return value; +} + +void _expectSignedFieldsEqual(SignedFields actual, SignedFields expected) { + expect(actual.plainInt, equals(expected.plainInt)); + expect(actual.i32Var, equals(expected.i32Var)); + expect(actual.i32Fixed, equals(expected.i32Fixed)); + expect(actual.i64VarInt, equals(expected.i64VarInt)); + expect(actual.i64FixedInt, equals(expected.i64FixedInt)); + expect(actual.i64TaggedInt, equals(expected.i64TaggedInt)); + expect(actual.i64Default, equals(expected.i64Default)); + expect(actual.i64Var, equals(expected.i64Var)); + expect(actual.i64Fixed, equals(expected.i64Fixed)); + expect(actual.i64Tagged, equals(expected.i64Tagged)); + expect(actual.optionalI32Var, equals(expected.optionalI32Var)); + expect(actual.optionalI32Fixed, equals(expected.optionalI32Fixed)); + expect(actual.optionalI64VarInt, equals(expected.optionalI64VarInt)); + expect(actual.optionalI64Fixed, equals(expected.optionalI64Fixed)); + expect(actual.optionalI64Tagged, equals(expected.optionalI64Tagged)); +} + +int _remoteFieldTypeId(Object value, String identifier) { + final remoteTypeDef = CompatibleStructMetadata.remoteTypeDefFor(value); + expect(remoteTypeDef, isNotNull); + final field = remoteTypeDef!.fields.firstWhere( + (field) => field.identifier == identifier, + ); + return field.fieldType.typeId; +} + +bool _remoteFieldNullable(Object value, String identifier) { + final remoteTypeDef = CompatibleStructMetadata.remoteTypeDefFor(value); + expect(remoteTypeDef, isNotNull); + final field = remoteTypeDef!.fields.firstWhere( + (field) => field.identifier == identifier, + ); + return field.fieldType.nullable; +} + +void main() { + group('signed generated fields', () { + test('round trips int and Int64 encoding edge cases', () { + final fory = Fory(); + _registerSignedFields(fory); + + for (final value in [ + _smallSignedFields(), + _jsSafeBoundarySignedFields(), + _fullRangeWrapperSignedFields(), + _nullSignedFields(), + ]) { + final roundTrip = fory.deserialize(fory.serialize(value)); + _expectSignedFieldsEqual(roundTrip, value); + } + }); + + test('compatible mode round trips Int32 and Int64 encoding edge cases', () { + final fory = Fory(compatible: true); + _registerSignedFields(fory); + + for (final value in [ + _smallSignedFields(), + _jsSafeBoundarySignedFields(), + _fullRangeWrapperSignedFields(), + _nullSignedFields(), + ]) { + final roundTrip = fory.deserialize(fory.serialize(value)); + _expectSignedFieldsEqual(roundTrip, value); + } + }); + + test('native schema-consistent Int64 varint int fields decode signed min', + () { + if (identical(1, 1.0)) { + return; + } + + final signedMin = _int64Min.toInt(); + final fory = Fory(); + _registerSignedFields(fory); + + final value = _smallSignedFields() + ..i64VarInt = signedMin + ..optionalI64VarInt = signedMin; + final roundTrip = fory.deserialize(fory.serialize(value)); + expect(roundTrip.i64VarInt, equals(signedMin)); + expect(roundTrip.optionalI64VarInt, equals(signedMin)); + + final buffer = Buffer(); + final writeCursor = GeneratedWriteCursor.reserve(buffer, 10); + writeCursor.writeVarInt64FromInt(signedMin); + writeCursor.finish(); + + final readCursor = + GeneratedReadCursor.start(Buffer.wrap(buffer.toBytes())); + expect(readCursor.readVarInt64AsInt(), equals(signedMin)); + readCursor.finish(); + }); + + test('compatible metadata records signed wire types and nullability', () { + final writer = Fory(compatible: true); + final reader = Fory(compatible: true); + _registerSignedFields(writer); + _registerSignedMetadataReader(reader); + + final roundTrip = reader.deserialize( + writer.serialize(_fullRangeWrapperSignedFields()), + ); + expect(roundTrip.plainInt, equals(0)); + + expect( + _remoteFieldTypeId(roundTrip, 'plain_int'), + equals(TypeIds.varInt64), + ); + expect( + _remoteFieldTypeId(roundTrip, 'i32_var'), + equals(TypeIds.varInt32), + ); + expect( + _remoteFieldTypeId(roundTrip, 'i32_fixed'), + equals(TypeIds.int32), + ); + expect( + _remoteFieldTypeId(roundTrip, 'i64_var_int'), + equals(TypeIds.varInt64), + ); + expect( + _remoteFieldTypeId(roundTrip, 'i64_fixed_int'), + equals(TypeIds.int64), + ); + expect( + _remoteFieldTypeId(roundTrip, 'i64_tagged_int'), + equals(TypeIds.taggedInt64), + ); + expect( + _remoteFieldTypeId(roundTrip, 'i64_default'), + equals(TypeIds.varInt64), + ); + expect( + _remoteFieldTypeId(roundTrip, 'i64_var'), + equals(TypeIds.varInt64), + ); + expect(_remoteFieldTypeId(roundTrip, 'i64_fixed'), equals(TypeIds.int64)); + expect( + _remoteFieldTypeId(roundTrip, 'i64_tagged'), + equals(TypeIds.taggedInt64), + ); + + expect(_remoteFieldNullable(roundTrip, 'plain_int'), isFalse); + expect(_remoteFieldNullable(roundTrip, 'optional_i32_var'), isTrue); + expect(_remoteFieldNullable(roundTrip, 'optional_i32_fixed'), isTrue); + expect(_remoteFieldNullable(roundTrip, 'optional_i64_var_int'), isTrue); + expect(_remoteFieldNullable(roundTrip, 'optional_i64_fixed'), isTrue); + expect(_remoteFieldNullable(roundTrip, 'optional_i64_tagged'), isTrue); + }); + + test('web rejects JS-unsafe Dart int fields instead of corrupting bytes', + () { + final cases = <({String name, SignedFields value})>[ + ( + name: 'plain int', + value: _smallSignedFields()..plainInt = _jsUnsafeInt, + ), + ( + name: 'varint', + value: _smallSignedFields()..i64VarInt = _jsUnsafeInt, + ), + ( + name: 'fixed', + value: _smallSignedFields()..i64FixedInt = _jsUnsafeInt, + ), + ( + name: 'tagged', + value: _smallSignedFields()..i64TaggedInt = _jsUnsafeInt, + ), + ( + name: 'nullable varint', + value: _smallSignedFields()..optionalI64VarInt = _jsUnsafeInt, + ), + ]; + + for (final testCase in cases) { + for (final compatible in [false, true]) { + final fory = Fory(compatible: compatible); + _registerSignedFields(fory); + if (identical(1, 1.0)) { + expect( + () => fory.serialize(testCase.value), + throwsA(isA()), + reason: '${testCase.name}, compatible=$compatible', + ); + } else { + final roundTrip = fory.deserialize( + fory.serialize(testCase.value), + ); + _expectSignedFieldsEqual(roundTrip, testCase.value); + } + } + } + }); + + test('web rejects JS-unsafe root and dynamic Dart ints', () { + final fory = Fory(); + + if (identical(1, 1.0)) { + expect(() => fory.serialize(_jsUnsafeInt), throwsA(isA())); + expect( + () => fory.serialize([_jsUnsafeInt]), + throwsA(isA()), + ); + } else { + expect( + fory.deserialize(fory.serialize(_jsUnsafeInt)), + equals(_jsUnsafeInt), + ); + expect( + fory.deserialize>( + fory.serialize([_jsUnsafeInt]), + ), + equals([_jsUnsafeInt]), + ); + } + }); + + test('web rejects full-range Int64 wrapper payloads read as Dart int', () { + final writer = Fory(compatible: true); + final reader = Fory(compatible: true); + _registerSignedFields(writer); + _registerSignedIntFieldsReader(reader); + + for (final encoding in ['varint', 'fixed', 'tagged']) { + if (identical(1, 1.0)) { + expect( + () => reader.deserialize( + writer.serialize(_int64ReadMismatchPayload(encoding)), + ), + throwsA(isA()), + reason: encoding, + ); + } else { + final roundTrip = reader.deserialize( + writer.serialize(_int64ReadMismatchPayload(encoding)), + ); + switch (encoding) { + case 'varint': + expect(roundTrip.i64Var, equals(_int64Min.toInt())); + case 'fixed': + expect(roundTrip.i64Fixed, equals(_int64Min.toInt())); + case 'tagged': + expect(roundTrip.i64Tagged, equals(_int64Min.toInt())); + } + } + } + }); + }); +} diff --git a/dart/packages/fory/test/string_encoding_test.dart b/dart/packages/fory/test/string_encoding_test.dart index 4a5011cc02..e33d08d4ae 100644 --- a/dart/packages/fory/test/string_encoding_test.dart +++ b/dart/packages/fory/test/string_encoding_test.dart @@ -17,7 +17,7 @@ * under the License. */ -import 'package:fory/src/buffer.dart'; +import 'package:fory/src/memory/buffer.dart'; import 'package:fory/src/util/string_util.dart'; import 'package:test/test.dart'; diff --git a/dart/packages/fory/test/time_serializer_test.dart b/dart/packages/fory/test/time_serializer_test.dart index 9ca5745e39..8715f845df 100644 --- a/dart/packages/fory/test/time_serializer_test.dart +++ b/dart/packages/fory/test/time_serializer_test.dart @@ -20,17 +20,19 @@ import 'dart:typed_data'; import 'package:fory/fory.dart'; -import 'package:fory/src/meta/type_ids.dart'; import 'package:test/test.dart'; part 'time_serializer_test.fory.dart'; +Timestamp _timestamp(int seconds, int nanoseconds) => + Timestamp(Int64(seconds), nanoseconds); + @ForyStruct() class TimeEnvelope { TimeEnvelope(); LocalDate date = const LocalDate(1970, 1, 1); - Timestamp timestamp = const Timestamp(0, 0); + Timestamp timestamp = _timestamp(0, 0); DateTime instant = DateTime.fromMicrosecondsSinceEpoch(0, isUtc: true); Duration duration = Duration.zero; LocalDate? optionalDate; @@ -62,10 +64,10 @@ void _expectTimeEnvelope(TimeEnvelope actual, TimeEnvelope expected) { TimeEnvelope _sampleTimeEnvelope() { return TimeEnvelope() ..date = const LocalDate(2024, 2, 29) - ..timestamp = const Timestamp(-123456789, 987654321) + ..timestamp = _timestamp(-123456789, 987654321) ..instant = DateTime.fromMicrosecondsSinceEpoch(-1, isUtc: true) ..duration = const Duration(days: 2, seconds: 3, microseconds: 456789) - ..optionalDate = LocalDate.fromEpochDay(-1) + ..optionalDate = LocalDate.fromEpochDay(Int64(-1)) ..optionalTimestamp = Timestamp.fromDateTime( DateTime.utc(2024, 1, 2, 3, 4, 5, 6, 700), ) @@ -78,8 +80,8 @@ void main() { test('round-trips LocalDate edge cases', () { final fory = Fory(); final cases = [ - LocalDate.fromEpochDay(-1), - LocalDate.fromEpochDay(0), + LocalDate.fromEpochDay(Int64(-1)), + LocalDate.fromEpochDay(Int64(0)), const LocalDate(2024, 2, 29), const LocalDate(9999, 12, 31), ]; @@ -92,10 +94,11 @@ void main() { test('encodes LocalDate as signed varint64 in xlang payloads', () { final fory = Fory(); - final value = LocalDate.fromEpochDay(-1); + final value = LocalDate.fromEpochDay(Int64(-1)); final bytes = fory.serialize(value); - expect(bytes, equals(Uint8List.fromList([0x02, 0xff, TypeIds.date, 0x01]))); + expect( + bytes, equals(Uint8List.fromList([0x02, 0xff, TypeIds.date, 0x01]))); expect(fory.deserialize(bytes), equals(value)); }); @@ -104,8 +107,8 @@ void main() { final value = LocalDate.fromDateTime(DateTime.utc(2024, 1, 2, 3, 4, 5)); expect(value, equals(const LocalDate(2024, 1, 2))); - expect( - value.toEpochDay(), equals(const LocalDate(2024, 1, 2).toEpochDay())); + final Int64 epochDay = value.toEpochDay(); + expect(epochDay, equals(const LocalDate(2024, 1, 2).toEpochDay())); expect(value.toDateTime(), equals(DateTime.utc(2024, 1, 2))); }); @@ -113,15 +116,15 @@ void main() { final fory = Fory(); final cases = >[ MapEntry( - const Timestamp(0, 0), + _timestamp(0, 0), DateTime.fromMicrosecondsSinceEpoch(0, isUtc: true), ), MapEntry( - const Timestamp(-1, 1000), + _timestamp(-1, 1000), DateTime.fromMicrosecondsSinceEpoch(-999999, isUtc: true), ), MapEntry( - const Timestamp(1, 999999000), + _timestamp(1, 999999000), DateTime.fromMicrosecondsSinceEpoch(1999999, isUtc: true), ), MapEntry( @@ -157,7 +160,7 @@ void main() { () { final fory = Fory(); final roundTrip = fory.deserialize>( - fory.serialize([const Timestamp(-1, 999999000)]), + fory.serialize([_timestamp(-1, 999999000)]), ); expect( @@ -174,8 +177,7 @@ void main() { final fory = Fory(); expect( - fory.deserialize( - fory.serialize(const Timestamp(-1, 999999000))), + fory.deserialize(fory.serialize(_timestamp(-1, 999999000))), equals(DateTime.fromMicrosecondsSinceEpoch(-1, isUtc: true)), ); }); @@ -195,11 +197,11 @@ void main() { final cases = >[ MapEntry( DateTime.fromMicrosecondsSinceEpoch(-1, isUtc: true), - const Timestamp(-1, 999999000), + _timestamp(-1, 999999000), ), MapEntry( DateTime.fromMicrosecondsSinceEpoch(-1000001, isUtc: true), - const Timestamp(-2, 999999000), + _timestamp(-2, 999999000), ), ]; @@ -249,8 +251,8 @@ void main() { test('does not preserve references for repeated temporal values', () { final fory = Fory(); final duration = Duration(microseconds: 1); - final timestamp = Timestamp(-1, 1000); - final date = LocalDate.fromEpochDay(-1); + final timestamp = _timestamp(-1, 1000); + final date = LocalDate.fromEpochDay(Int64(-1)); final roundTrip = fory.deserialize>( fory.serialize( @@ -354,7 +356,7 @@ void main() { final fory = Fory(); expect( - () => fory.serialize(const Timestamp(0, 1000000000)), + () => fory.serialize(_timestamp(0, 1000000000)), throwsA( isA().having( (error) => error.toString(), @@ -367,7 +369,7 @@ void main() { test('rejects timestamp payloads with nanoseconds outside spec range', () { final fory = Fory(); - final bytes = Uint8List.fromList(fory.serialize(const Timestamp(0, 0))); + final bytes = Uint8List.fromList(fory.serialize(_timestamp(0, 0))); final view = ByteData.sublistView(bytes); view.setUint32(bytes.length - 4, 1000000000, Endian.little); diff --git a/dart/packages/fory/test/unsigned_serializer_test.dart b/dart/packages/fory/test/unsigned_serializer_test.dart index 7b39f7b7f5..6459f0f602 100644 --- a/dart/packages/fory/test/unsigned_serializer_test.dart +++ b/dart/packages/fory/test/unsigned_serializer_test.dart @@ -24,8 +24,10 @@ import 'package:test/test.dart'; part 'unsigned_serializer_test.fory.dart'; const int _uint32Midpoint = 0x80000000; -const int _uint64Midpoint = 0x8000000000000000; -const int _uint64Max = 0xffffffffffffffff; +const int _jsSafeIntMax = 9007199254740991; +const int _jsUnsafeInt = 9007199254740992; +final Uint64 _uint64Midpoint = Uint64.parseHex('8000000000000000'); +final Uint64 _uint64Max = Uint64.parseHex('ffffffffffffffff'); @ForyStruct() class UnsignedFields { @@ -44,13 +46,22 @@ class UnsignedFields { int u32Fixed = 0; @Uint64Type(encoding: LongEncoding.varint) - int u64Var = 0; + Uint64 u64Var = Uint64(0); @Uint64Type(encoding: LongEncoding.fixed) - int u64Fixed = 0; + Uint64 u64Fixed = Uint64(0); @Uint64Type(encoding: LongEncoding.tagged) - int u64Tagged = 0; + Uint64 u64Tagged = Uint64(0); + + @Uint64Type(encoding: LongEncoding.varint) + int u64VarInt = 0; + + @Uint64Type(encoding: LongEncoding.fixed) + int u64FixedInt = 0; + + @Uint64Type(encoding: LongEncoding.tagged) + int u64TaggedInt = 0; @Uint8Type() int? u8Nullable; @@ -65,13 +76,16 @@ class UnsignedFields { int? u32FixedNullable; @Uint64Type(encoding: LongEncoding.varint) - int? u64VarNullable; + Uint64? u64VarNullable; @Uint64Type(encoding: LongEncoding.fixed) - int? u64FixedNullable; + Uint64? u64FixedNullable; @Uint64Type(encoding: LongEncoding.tagged) - int? u64TaggedNullable; + Uint64? u64TaggedNullable; + + @Uint64Type(encoding: LongEncoding.varint) + int? u64VarIntNullable; } @ForyStruct() @@ -84,6 +98,84 @@ class UnsignedMetadataReader { int extra = 42; } +@ForyStruct() +class UnsignedIntFieldsReader { + UnsignedIntFieldsReader(); + + @Uint64Type(encoding: LongEncoding.varint) + int u64Var = 0; + + @Uint64Type(encoding: LongEncoding.fixed) + int u64Fixed = 0; + + @Uint64Type(encoding: LongEncoding.tagged) + int u64Tagged = 0; +} + +@ForyStruct() +class UnsignedWrapperFields { + UnsignedWrapperFields(); + + @Uint64Type(encoding: LongEncoding.varint) + Uint64 u64Var = Uint64(0); + + @Uint64Type(encoding: LongEncoding.fixed) + Uint64 u64Fixed = Uint64(0); + + @Uint64Type(encoding: LongEncoding.tagged) + Uint64 u64Tagged = Uint64(0); +} + +@ForyStruct() +class UnsignedWrapperAsIntFields { + UnsignedWrapperAsIntFields(); + + @Uint64Type(encoding: LongEncoding.varint) + int u64Var = 0; + + @Uint64Type(encoding: LongEncoding.fixed) + int u64Fixed = 0; + + @Uint64Type(encoding: LongEncoding.tagged) + int u64Tagged = 0; +} + +UnsignedFields _uint64ReadMismatchPayload(String encoding) { + final value = _smallUnsignedFields() + ..u64Var = Uint64(1) + ..u64Fixed = Uint64(1) + ..u64Tagged = Uint64(1); + switch (encoding) { + case 'varint': + value.u64Var = _uint64Max; + case 'fixed': + value.u64Fixed = _uint64Max; + case 'tagged': + value.u64Tagged = _uint64Max; + default: + throw ArgumentError.value(encoding, 'encoding'); + } + return value; +} + +UnsignedWrapperFields _schemaUint64ReadMismatchPayload(String encoding) { + final value = UnsignedWrapperFields() + ..u64Var = Uint64(1) + ..u64Fixed = Uint64(1) + ..u64Tagged = Uint64(1); + switch (encoding) { + case 'varint': + value.u64Var = _uint64Max; + case 'fixed': + value.u64Fixed = _uint64Max; + case 'tagged': + value.u64Tagged = _uint64Max; + default: + throw ArgumentError.value(encoding, 'encoding'); + } + return value; +} + void _registerUnsignedFields(Fory fory) { UnsignedSerializerTestFory.register( fory, @@ -102,6 +194,77 @@ void _registerUnsignedMetadataReader(Fory fory) { ); } +void _registerUnsignedIntFieldsReader(Fory fory) { + UnsignedSerializerTestFory.register( + fory, + UnsignedIntFieldsReader, + namespace: 'test', + typeName: 'UnsignedFields', + ); +} + +void _registerUnsignedWrapperFields(Fory fory) { + UnsignedSerializerTestFory.register( + fory, + UnsignedWrapperFields, + namespace: 'test', + typeName: 'UnsignedSchemaUint64Fields', + ); +} + +void _registerUnsignedWrapperAsIntFields(Fory fory) { + UnsignedSerializerTestFory.register( + fory, + UnsignedWrapperAsIntFields, + namespace: 'test', + typeName: 'UnsignedSchemaUint64Fields', + ); +} + +UnsignedFields _smallUnsignedFields() { + return UnsignedFields() + ..u8 = 0 + ..u16 = 1 + ..u32Var = 0 + ..u32Fixed = 1 + ..u64Var = Uint64(0) + ..u64Fixed = Uint64(1) + ..u64Tagged = Uint64(0) + ..u64VarInt = 0 + ..u64FixedInt = 1 + ..u64TaggedInt = 0 + ..u8Nullable = 1 + ..u16Nullable = 0 + ..u32VarNullable = 1 + ..u32FixedNullable = 0 + ..u64VarNullable = Uint64(1) + ..u64FixedNullable = Uint64(0) + ..u64TaggedNullable = Uint64(1) + ..u64VarIntNullable = 1; +} + +UnsignedFields _taggedBoundaryUnsignedFields() { + return UnsignedFields() + ..u8 = 0x7f + ..u16 = 0x7fff + ..u32Var = 0x7fffffff + ..u32Fixed = 0x80000000 + ..u64Var = Uint64(0x7fffffff) + ..u64Fixed = Uint64(0x80000000) + ..u64Tagged = Uint64(0x7fffffff) + ..u64VarInt = 0x7fffffff + ..u64FixedInt = 0x80000000 + ..u64TaggedInt = 0x7fffffff + ..u8Nullable = 0x80 + ..u16Nullable = 0x8000 + ..u32VarNullable = 0x80000000 + ..u32FixedNullable = 0x7fffffff + ..u64VarNullable = Uint64(0x80000000) + ..u64FixedNullable = Uint64(0x7fffffff) + ..u64TaggedNullable = Uint64(0x80000000) + ..u64VarIntNullable = 0x80000000; +} + UnsignedFields _midpointUnsignedFields() { return UnsignedFields() ..u8 = 0x80 @@ -111,13 +274,17 @@ UnsignedFields _midpointUnsignedFields() { ..u64Var = _uint64Midpoint ..u64Fixed = _uint64Midpoint ..u64Tagged = _uint64Midpoint + ..u64VarInt = _jsSafeIntMax + ..u64FixedInt = _jsSafeIntMax + ..u64TaggedInt = _jsSafeIntMax ..u8Nullable = 0x80 ..u16Nullable = 0x8000 ..u32VarNullable = _uint32Midpoint ..u32FixedNullable = _uint32Midpoint ..u64VarNullable = _uint64Midpoint ..u64FixedNullable = _uint64Midpoint - ..u64TaggedNullable = _uint64Midpoint; + ..u64TaggedNullable = _uint64Midpoint + ..u64VarIntNullable = _jsSafeIntMax; } UnsignedFields _maxUnsignedFields() { @@ -129,13 +296,17 @@ UnsignedFields _maxUnsignedFields() { ..u64Var = _uint64Max ..u64Fixed = _uint64Max ..u64Tagged = _uint64Max + ..u64VarInt = _jsSafeIntMax + ..u64FixedInt = _jsSafeIntMax + ..u64TaggedInt = _jsSafeIntMax ..u8Nullable = 0xff ..u16Nullable = 0xffff ..u32VarNullable = 0xffffffff ..u32FixedNullable = 0xffffffff ..u64VarNullable = _uint64Max ..u64FixedNullable = _uint64Max - ..u64TaggedNullable = _uint64Max; + ..u64TaggedNullable = _uint64Max + ..u64VarIntNullable = _jsSafeIntMax; } UnsignedFields _nullUnsignedFields() { @@ -144,19 +315,24 @@ UnsignedFields _nullUnsignedFields() { ..u16 = 2 ..u32Var = 3 ..u32Fixed = 4 - ..u64Var = 5 - ..u64Fixed = 6 - ..u64Tagged = 7 + ..u64Var = Uint64(5) + ..u64Fixed = Uint64(6) + ..u64Tagged = Uint64(7) + ..u64VarInt = 5 + ..u64FixedInt = 6 + ..u64TaggedInt = 7 ..u8Nullable = null ..u16Nullable = null ..u32VarNullable = null ..u32FixedNullable = null ..u64VarNullable = null ..u64FixedNullable = null - ..u64TaggedNullable = null; + ..u64TaggedNullable = null + ..u64VarIntNullable = null; } -void _expectUnsignedFieldsEqual(UnsignedFields actual, UnsignedFields expected) { +void _expectUnsignedFieldsEqual( + UnsignedFields actual, UnsignedFields expected) { expect(actual.u8, equals(expected.u8)); expect(actual.u16, equals(expected.u16)); expect(actual.u32Var, equals(expected.u32Var)); @@ -164,6 +340,9 @@ void _expectUnsignedFieldsEqual(UnsignedFields actual, UnsignedFields expected) expect(actual.u64Var, equals(expected.u64Var)); expect(actual.u64Fixed, equals(expected.u64Fixed)); expect(actual.u64Tagged, equals(expected.u64Tagged)); + expect(actual.u64VarInt, equals(expected.u64VarInt)); + expect(actual.u64FixedInt, equals(expected.u64FixedInt)); + expect(actual.u64TaggedInt, equals(expected.u64TaggedInt)); expect(actual.u8Nullable, equals(expected.u8Nullable)); expect(actual.u16Nullable, equals(expected.u16Nullable)); expect(actual.u32VarNullable, equals(expected.u32VarNullable)); @@ -171,6 +350,7 @@ void _expectUnsignedFieldsEqual(UnsignedFields actual, UnsignedFields expected) expect(actual.u64VarNullable, equals(expected.u64VarNullable)); expect(actual.u64FixedNullable, equals(expected.u64FixedNullable)); expect(actual.u64TaggedNullable, equals(expected.u64TaggedNullable)); + expect(actual.u64VarIntNullable, equals(expected.u64VarIntNullable)); } int _remoteFieldTypeId(Object value, String identifier) { @@ -193,16 +373,36 @@ bool _remoteFieldNullable(Object value, String identifier) { void main() { group('unsigned generated fields', () { - test('round trips midpoint, max, and null boundary cases', () { + test('round trips small, threshold, midpoint, max, and null cases', () { final fory = Fory(); _registerUnsignedFields(fory); for (final value in [ + _smallUnsignedFields(), + _taggedBoundaryUnsignedFields(), + _midpointUnsignedFields(), + _maxUnsignedFields(), + _nullUnsignedFields(), + ]) { + final roundTrip = + fory.deserialize(fory.serialize(value)); + _expectUnsignedFieldsEqual(roundTrip, value); + } + }); + + test('compatible mode round trips unsigned encoding edge cases', () { + final fory = Fory(compatible: true); + _registerUnsignedFields(fory); + + for (final value in [ + _smallUnsignedFields(), + _taggedBoundaryUnsignedFields(), _midpointUnsignedFields(), _maxUnsignedFields(), _nullUnsignedFields(), ]) { - final roundTrip = fory.deserialize(fory.serialize(value)); + final roundTrip = + fory.deserialize(fory.serialize(value)); _expectUnsignedFieldsEqual(roundTrip, value); } }); @@ -221,14 +421,30 @@ void main() { expect(_remoteFieldTypeId(roundTrip, 'u8'), equals(TypeIds.uint8)); expect(_remoteFieldTypeId(roundTrip, 'u16'), equals(TypeIds.uint16)); - expect(_remoteFieldTypeId(roundTrip, 'u32_var'), equals(TypeIds.varUint32)); - expect(_remoteFieldTypeId(roundTrip, 'u32_fixed'), equals(TypeIds.uint32)); - expect(_remoteFieldTypeId(roundTrip, 'u64_var'), equals(TypeIds.varUint64)); - expect(_remoteFieldTypeId(roundTrip, 'u64_fixed'), equals(TypeIds.uint64)); + expect( + _remoteFieldTypeId(roundTrip, 'u32_var'), equals(TypeIds.varUint32)); + expect( + _remoteFieldTypeId(roundTrip, 'u32_fixed'), equals(TypeIds.uint32)); + expect( + _remoteFieldTypeId(roundTrip, 'u64_var'), equals(TypeIds.varUint64)); + expect( + _remoteFieldTypeId(roundTrip, 'u64_fixed'), equals(TypeIds.uint64)); expect( _remoteFieldTypeId(roundTrip, 'u64_tagged'), equals(TypeIds.taggedUint64), ); + expect( + _remoteFieldTypeId(roundTrip, 'u64_var_int'), + equals(TypeIds.varUint64), + ); + expect( + _remoteFieldTypeId(roundTrip, 'u64_fixed_int'), + equals(TypeIds.uint64), + ); + expect( + _remoteFieldTypeId(roundTrip, 'u64_tagged_int'), + equals(TypeIds.taggedUint64), + ); expect(_remoteFieldNullable(roundTrip, 'u8'), isFalse); expect(_remoteFieldNullable(roundTrip, 'u16'), isFalse); @@ -239,6 +455,122 @@ void main() { expect(_remoteFieldNullable(roundTrip, 'u64_var_nullable'), isTrue); expect(_remoteFieldNullable(roundTrip, 'u64_fixed_nullable'), isTrue); expect(_remoteFieldNullable(roundTrip, 'u64_tagged_nullable'), isTrue); + expect(_remoteFieldNullable(roundTrip, 'u64_var_int_nullable'), isTrue); + }); + + test('web rejects JS-unsafe uint64 Dart int fields', () { + final cases = <({String name, UnsignedFields value})>[ + ( + name: 'varint', + value: _smallUnsignedFields()..u64VarInt = _jsUnsafeInt, + ), + ( + name: 'fixed', + value: _smallUnsignedFields()..u64FixedInt = _jsUnsafeInt, + ), + ( + name: 'tagged', + value: _smallUnsignedFields()..u64TaggedInt = _jsUnsafeInt, + ), + ( + name: 'nullable varint', + value: _smallUnsignedFields()..u64VarIntNullable = _jsUnsafeInt, + ), + ]; + + for (final testCase in cases) { + for (final compatible in [false, true]) { + final fory = Fory(compatible: compatible); + _registerUnsignedFields(fory); + if (identical(1, 1.0)) { + expect( + () => fory.serialize(testCase.value), + throwsA(isA()), + reason: '${testCase.name}, compatible=$compatible', + ); + } else { + final roundTrip = fory.deserialize( + fory.serialize(testCase.value), + ); + _expectUnsignedFieldsEqual(roundTrip, testCase.value); + } + } + } + }); + + test('web rejects negative uint64 Dart int fields', () { + if (!identical(1, 1.0)) { + return; + } + + final cases = <({String name, UnsignedFields value})>[ + ( + name: 'varint', + value: _smallUnsignedFields()..u64VarInt = -1, + ), + ( + name: 'fixed', + value: _smallUnsignedFields()..u64FixedInt = -1, + ), + ( + name: 'tagged', + value: _smallUnsignedFields()..u64TaggedInt = -1, + ), + ( + name: 'nullable varint', + value: _smallUnsignedFields()..u64VarIntNullable = -1, + ), + ]; + + for (final testCase in cases) { + for (final compatible in [false, true]) { + final fory = Fory(compatible: compatible); + _registerUnsignedFields(fory); + expect( + () => fory.serialize(testCase.value), + throwsA(isA()), + reason: '${testCase.name}, compatible=$compatible', + ); + } + } + }); + + test( + 'schema-consistent Dart int uint64 readers reject full-range wrapper payloads', + () { + final writer = Fory(); + final reader = Fory(); + _registerUnsignedWrapperFields(writer); + _registerUnsignedWrapperAsIntFields(reader); + + for (final encoding in ['varint', 'fixed', 'tagged']) { + expect( + () => reader.deserialize( + writer.serialize(_schemaUint64ReadMismatchPayload(encoding)), + ), + throwsA(isA()), + reason: encoding, + ); + } + }); + + test( + 'compatible Dart int uint64 readers reject full-range wrapper payloads', + () { + final writer = Fory(compatible: true); + final reader = Fory(compatible: true); + _registerUnsignedFields(writer); + _registerUnsignedIntFieldsReader(reader); + + for (final encoding in ['varint', 'fixed', 'tagged']) { + expect( + () => reader.deserialize( + writer.serialize(_uint64ReadMismatchPayload(encoding)), + ), + throwsA(isA()), + reason: encoding, + ); + } }); }); } diff --git a/docs/benchmarks/dart/README.md b/docs/benchmarks/dart/README.md index 090c930bf6..2348854a2d 100644 --- a/docs/benchmarks/dart/README.md +++ b/docs/benchmarks/dart/README.md @@ -4,17 +4,17 @@ This benchmark compares serialization and deserialization throughput for Apache ## Hardware and Runtime Info -| Key | Value | -| --------------------- | ---------------------------------------------------------------- | -| Timestamp | 2026-04-13T21:55:28.456625Z | -| OS | Version 26.2 (Build 25C56) | -| Host | Macbook-Air.local | -| CPU Cores (Logical) | 8 | -| Memory (GB) | 8.00 | -| Dart | 3.10.4 (stable) (Tue Dec 9 00:01:55 2025 -0800) on "macos_arm64" | -| Samples per case | 5 | -| Warmup per case (s) | 1.0 | -| Duration per case (s) | 1.5 | +| Key | Value | +| --------------------- | ----------------------------------------------------------------- | +| Timestamp | 2026-04-23T10:50:07.751368Z | +| OS | Version 15.7.2 (Build 24G325) | +| Host | MacBook-Pro.local | +| CPU Cores (Logical) | 12 | +| Memory (GB) | 48.00 | +| Dart | 3.10.7 (stable) (Tue Dec 23 00:01:57 2025 -0800) on "macos_arm64" | +| Samples per case | 5 | +| Warmup per case (s) | 1.0 | +| Duration per case (s) | 1.5 | ## Throughput Results @@ -22,18 +22,18 @@ This benchmark compares serialization and deserialization throughput for Apache | Datatype | Operation | Fory TPS | Protobuf TPS | Fastest | | ---------------- | ----------- | --------: | -----------: | ------------ | -| Struct | Serialize | 3,989,432 | 1,884,653 | fory (2.12x) | -| Struct | Deserialize | 5,828,197 | 4,199,680 | fory (1.39x) | -| Sample | Serialize | 1,649,722 | 500,167 | fory (3.30x) | -| Sample | Deserialize | 2,060,113 | 785,109 | fory (2.62x) | -| MediaContent | Serialize | 800,876 | 391,235 | fory (2.05x) | -| MediaContent | Deserialize | 1,315,115 | 683,533 | fory (1.92x) | -| StructList | Serialize | 1,456,396 | 367,506 | fory (3.96x) | -| StructList | Deserialize | 1,921,006 | 645,958 | fory (2.97x) | -| SampleList | Serialize | 411,144 | 48,508 | fory (8.48x) | -| SampleList | Deserialize | 464,273 | 103,558 | fory (4.48x) | -| MediaContentList | Serialize | 186,870 | 77,029 | fory (2.43x) | -| MediaContentList | Deserialize | 330,293 | 128,215 | fory (2.58x) | +| Struct | Serialize | 5,041,693 | 2,073,839 | fory (2.43x) | +| Struct | Deserialize | 6,395,290 | 4,991,881 | fory (1.28x) | +| Sample | Serialize | 1,783,688 | 552,140 | fory (3.23x) | +| Sample | Deserialize | 2,124,197 | 934,794 | fory (2.27x) | +| MediaContent | Serialize | 952,498 | 438,419 | fory (2.17x) | +| MediaContent | Deserialize | 1,649,039 | 737,340 | fory (2.24x) | +| StructList | Serialize | 1,945,119 | 399,007 | fory (4.87x) | +| StructList | Deserialize | 2,119,403 | 764,832 | fory (2.77x) | +| SampleList | Serialize | 475,413 | 52,512 | fory (9.05x) | +| SampleList | Deserialize | 508,939 | 116,236 | fory (4.38x) | +| MediaContentList | Serialize | 224,925 | 84,860 | fory (2.65x) | +| MediaContentList | Deserialize | 387,070 | 154,392 | fory (2.51x) | ## Serialized Size (bytes) diff --git a/docs/benchmarks/dart/mediacontent.png b/docs/benchmarks/dart/mediacontent.png index 17bd33a9b1..ba6e2d3aed 100644 Binary files a/docs/benchmarks/dart/mediacontent.png and b/docs/benchmarks/dart/mediacontent.png differ diff --git a/docs/benchmarks/dart/mediacontentlist.png b/docs/benchmarks/dart/mediacontentlist.png index a0d404e06f..02f785e428 100644 Binary files a/docs/benchmarks/dart/mediacontentlist.png and b/docs/benchmarks/dart/mediacontentlist.png differ diff --git a/docs/benchmarks/dart/sample.png b/docs/benchmarks/dart/sample.png index 11f2d7743e..f274160b44 100644 Binary files a/docs/benchmarks/dart/sample.png and b/docs/benchmarks/dart/sample.png differ diff --git a/docs/benchmarks/dart/samplelist.png b/docs/benchmarks/dart/samplelist.png index a4b8cb1c09..72e6cf7b06 100644 Binary files a/docs/benchmarks/dart/samplelist.png and b/docs/benchmarks/dart/samplelist.png differ diff --git a/docs/benchmarks/dart/struct.png b/docs/benchmarks/dart/struct.png index 7d616cbeda..f04b50825f 100644 Binary files a/docs/benchmarks/dart/struct.png and b/docs/benchmarks/dart/struct.png differ diff --git a/docs/benchmarks/dart/structlist.png b/docs/benchmarks/dart/structlist.png index 01484cfe96..0bf1fd0df5 100644 Binary files a/docs/benchmarks/dart/structlist.png and b/docs/benchmarks/dart/structlist.png differ diff --git a/docs/benchmarks/dart/throughput.png b/docs/benchmarks/dart/throughput.png index f0c4d91d8a..d41f05638e 100644 Binary files a/docs/benchmarks/dart/throughput.png and b/docs/benchmarks/dart/throughput.png differ diff --git a/docs/compiler/generated-code.md b/docs/compiler/generated-code.md index 87ea18cf45..6dd2ff6df5 100644 --- a/docs/compiler/generated-code.md +++ b/docs/compiler/generated-code.md @@ -962,10 +962,13 @@ Map byName = {}; ### Registration -Each schema includes a registration helper that handles all types in the file and transitively registers imported types: +Each generated Dart library includes a registration helper named after the input +file, such as `AddressbookFory` for `addressbook.dart`. The helper handles all +generated types in that file and transitively registers imported generated +types: ```dart -abstract final class ForyRegistration { +abstract final class AddressbookFory { static void register( Fory fory, Type type, { @@ -990,8 +993,8 @@ import 'generated/addressbook/addressbook.dart'; void main() { final fory = Fory(); - ForyRegistration.register(fory, Person, id: 100); - ForyRegistration.register(fory, Dog, id: 104); + AddressbookFory.register(fory, Person, id: 100); + AddressbookFory.register(fory, Dog, id: 104); // ... final person = Person() diff --git a/docs/guide/dart/cross-language.md b/docs/guide/dart/cross-language.md index a659c9257a..a4554b8c46 100644 --- a/docs/guide/dart/cross-language.md +++ b/docs/guide/dart/cross-language.md @@ -168,7 +168,7 @@ Fory matches fields by name or by stable field ID. For robust cross-language int Because Dart `int` is not itself a promise about the exact xlang wire width, prefer wrappers or numeric field annotations when exact cross-language interpretation matters: - `Int32` for xlang `int32` -- `UInt32` for xlang `uint32` +- `Uint32` for xlang `uint32` - `Float16`, `Bfloat16`, and `Float32` for reduced-width floating point - `Float16List` and `Bfloat16List` for 16-bit floating-point array payloads - `Timestamp`, `LocalDate`, and `Duration` for explicit temporal semantics diff --git a/docs/guide/dart/custom-serializers.md b/docs/guide/dart/custom-serializers.md index 280eb1ed8f..8c8c1ee449 100644 --- a/docs/guide/dart/custom-serializers.md +++ b/docs/guide/dart/custom-serializers.md @@ -48,13 +48,13 @@ final class PersonSerializer extends Serializer { void write(WriteContext context, Person value) { final buffer = context.buffer; buffer.writeUtf8(value.name); - buffer.writeInt64(value.age); + buffer.writeInt64FromInt(value.age); } @override Person read(ReadContext context) { final buffer = context.buffer; - return Person(buffer.readUtf8(), buffer.readInt64()); + return Person(buffer.readUtf8(), buffer.readInt64AsInt()); } } ``` diff --git a/docs/guide/dart/field-configuration.md b/docs/guide/dart/field-configuration.md index 4f16f2b7ad..5e0b439f02 100644 --- a/docs/guide/dart/field-configuration.md +++ b/docs/guide/dart/field-configuration.md @@ -110,7 +110,10 @@ class Sample { Available annotations: `@Int32Type`, `@Int64Type`, `@Uint8Type`, `@Uint16Type`, `@Uint32Type`, `@Uint64Type`. -Alternatively, use the explicit wrapper types (`Int32`, `UInt32`, etc.) described in [Supported Types](supported-types.md). +Alternatively, use explicit wrapper types such as `Int32`, `Int64`, `Uint32`, +and `Uint64` as described in [Supported Types](supported-types.md). Wrappers use +compact varint encodings by default; use annotations or generated field +metadata when a fixed-width or tagged encoding is required. ## Aligning Fields Across Languages diff --git a/docs/guide/dart/index.md b/docs/guide/dart/index.md index 0e814c4476..cab137c48c 100644 --- a/docs/guide/dart/index.md +++ b/docs/guide/dart/index.md @@ -24,6 +24,7 @@ Apache Fory™ Dart lets you serialize Dart objects to bytes and deserialize the ## Why Fory Dart? - **Cross-language**: serialize in Dart, deserialize in Java, Go, C#, and more without writing any glue code +- **Platform support**: use the same generated-serializer API on Dart VM/AOT, Flutter, and web - **Fast**: generated serializer code replaces reflection at runtime - **Schema evolution**: add or remove fields without breaking existing messages - **Circular references**: optional reference tracking handles shared or recursive object graphs @@ -114,25 +115,26 @@ dart run build_runner build --delete-conflicting-outputs - `fory.deserialize(bytes)` — returns a `T` - `@ForyStruct()` — marks a class for code generation - `@ForyField(...)` — per-field options (skip, ID, nullability, references) -- Integer wrappers: `Int8`, `Int16`, `Int32`, `Uint8`, `Uint16`, `Uint32`, `Uint64` +- Integer wrappers: `Int8`, `Int16`, `Int32`, `Int64`, `Uint8`, `Uint16`, `Uint32`, `Uint64` - Float wrappers: `Float16`, `Bfloat16`, `Float32` - 16-bit float arrays: `Float16List`, `Bfloat16List` - Time types: `LocalDate`, `Timestamp`, `Duration` ## Documentation -| Topic | Description | -| --------------------------------------------- | --------------------------------------------------------------- | -| [Configuration](configuration.md) | Runtime options, compatible mode, and safety limits | -| [Basic Serialization](basic-serialization.md) | `serialize`, `deserialize`, generated registration, root graphs | -| [Code Generation](code-generation.md) | `@ForyStruct`, build runner, and generated namespaces | -| [Type Registration](type-registration.md) | ID-based vs name-based registration and registration rules | -| [Custom Serializers](custom-serializers.md) | Manual `Serializer` implementations and unions | -| [Field Configuration](field-configuration.md) | `@ForyField`, field IDs, nullability, references, polymorphism | -| [Supported Types](supported-types.md) | Built-in xlang values, wrappers, collections, and structs | -| [Schema Evolution](schema-evolution.md) | Compatible structs and evolving schemas | -| [Cross-Language](cross-language.md) | Interoperability rules and field alignment | -| [Troubleshooting](troubleshooting.md) | Common errors, diagnostics, and validation steps | +| Topic | Description | +| ----------------------------------------------- | --------------------------------------------------------------- | +| [Configuration](configuration.md) | Runtime options, compatible mode, and safety limits | +| [Basic Serialization](basic-serialization.md) | `serialize`, `deserialize`, generated registration, root graphs | +| [Code Generation](code-generation.md) | `@ForyStruct`, build runner, and generated namespaces | +| [Type Registration](type-registration.md) | ID-based vs name-based registration and registration rules | +| [Custom Serializers](custom-serializers.md) | Manual `Serializer` implementations and unions | +| [Field Configuration](field-configuration.md) | `@ForyField`, field IDs, nullability, references, polymorphism | +| [Supported Types](supported-types.md) | Built-in xlang values, wrappers, collections, and structs | +| [Schema Evolution](schema-evolution.md) | Compatible structs and evolving schemas | +| [Cross-Language](cross-language.md) | Interoperability rules and field alignment | +| [Web Platform Support](web-platform-support.md) | Dart VM/AOT, Flutter, and web support, limits, and validation | +| [Troubleshooting](troubleshooting.md) | Common errors, diagnostics, and validation steps | ## Related Resources diff --git a/docs/guide/dart/supported-types.md b/docs/guide/dart/supported-types.md index 3d309ee5b5..e7ee82e737 100644 --- a/docs/guide/dart/supported-types.md +++ b/docs/guide/dart/supported-types.md @@ -37,20 +37,36 @@ The following Dart types serialize directly without any special handling: ## Integer Wrappers -Dart `int` is 64-bit at runtime. If the peer language expects a 32-bit integer (Java `int`, Go `int32`, C# `int`) and you send a Dart `int`, the deserialization may fail or silently truncate. +Dart VM/native `int` can represent signed 64-bit values, while Dart web `int` +is limited to JavaScript-safe integer precision. If the peer language expects a +32-bit integer (Java `int`, Go `int32`, C# `int`) and you send a Dart `int`, +the deserialization may fail or silently truncate. For browser and Flutter web +precision rules, see [Web Platform Support](web-platform-support.md). -Use an integer wrapper class to pin the exact wire width: +Use an integer wrapper or field annotation to select the wire type explicitly: ```dart final Int8 tiny = Int8(-1); // 8-bit signed final Int16 shortValue = Int16(7); // 16-bit signed -final Int32 age = Int32(36); // 32-bit signed — matches Java int, C# int, Go int32 -final UInt8 flags = UInt8(255); // 8-bit unsigned -final UInt16 port = UInt16(65535); // 16-bit unsigned -final UInt32 count = UInt32(4000000000); // 32-bit unsigned +final Int32 age = Int32(36); // 32-bit signed, varint by default +final Int64 seq = Int64(0); // signed 64-bit, varint by default +final Uint8 flags = Uint8(255); // 8-bit unsigned +final Uint16 port = Uint16(65535); // 16-bit unsigned +final Uint32 count = Uint32(4000000000); // 32-bit unsigned, varint by default +final Uint64 offset = Uint64(0); // unsigned 64-bit, varint by default ``` -Each wrapper clamps the stored value to the target bit width. +Each wrapper clamps or normalizes the stored value to the target bit width. +Root `Int32`, `Int64`, `Uint32`, and `Uint64` values use compact varint wire +types by default. Use `@Int64Type`, `@Uint32Type`, `@Uint64Type`, or generated +field metadata when a fixed-width or tagged encoding is required. + +On Dart VM, `Int64` and `Uint64` are extension types over `int`. Once a value is +passed through an `Object`-typed dynamic/root boundary, the VM cannot recover +whether it was originally a plain `int`, `Int64`, or `Uint64`. Use generated +field metadata or explicit `Buffer` APIs when native VM payloads must preserve +unsigned 64-bit identity across dynamic boundaries. Dart web uses wrapper +classes, so web root `Uint64` values keep `varuint64` metadata. ## Floating-Point Wrappers @@ -82,7 +98,7 @@ final timeout = const Duration(seconds: 30); The temporal wrappers expose conversion helpers: - `Timestamp.fromDateTime(...)` and `timestamp.toDateTime()` -- `LocalDate.fromEpochDay(...)`, `date.toEpochDay()` +- `LocalDate.fromEpochDay(Int64(...))`, `date.toEpochDay()` returns `Int64` - `LocalDate.fromDateTime(...)` and `date.toDateTime()` `Duration` support in Dart is exact to microseconds. Incoming xlang duration diff --git a/docs/guide/dart/troubleshooting.md b/docs/guide/dart/troubleshooting.md index 9266d05fb8..95e2c17763 100644 --- a/docs/guide/dart/troubleshooting.md +++ b/docs/guide/dart/troubleshooting.md @@ -1,6 +1,6 @@ --- title: Troubleshooting -sidebar_position: 10 +sidebar_position: 11 id: dart_troubleshooting license: | Licensed to the Apache Software Foundation (ASF) under one or more @@ -78,6 +78,48 @@ Checklist: 4. `Timestamp` / `LocalDate` instead of raw `DateTime` for date/time fields. 5. `compatible: true` on **both** sides if using schema evolution. +## Int64 or Uint64 values fail on web + +On Dart VM builds, Dart `int` can represent signed 64-bit values. On Dart web +builds, Dart `int` values are backed by JavaScript numbers and are only precise +inside the JS-safe integer range: + +```text +-9007199254740991 <= value <= 9007199254740991 +``` + +If a generated serializer writes an `int64` field declared as Dart `int`, +web builds reject values outside that range instead of silently writing +corrupted bytes. To exchange full signed 64-bit values on web, declare the +field as Fory's `Int64` wrapper: + +```dart +@ForyStruct() +class LedgerEntry { + LedgerEntry(); + + Int64 sequence = Int64(0); // full signed 64-bit range on VM and web +} +``` + +For unsigned 64-bit values, prefer `Uint64` rather than Dart `int`. Dart `int` +cannot represent the full `uint64` range on either VM or web: + +```dart +@ForyStruct() +class FileBlock { + FileBlock(); + + Uint64 offset = Uint64(0); // full unsigned 64-bit range +} +``` + +`@Int64Type` changes the wire encoding for a Dart `int` field, but it does not +remove the web integer precision limit. Use `Int64` for full-range signed +values and `Uint64` for full-range unsigned values. See +[Web Platform Support](web-platform-support.md) for the full browser support +matrix and migration guidance. + ## Running Tests Locally Main Dart package: @@ -101,3 +143,4 @@ dart test - [Cross-Language](cross-language.md) - [Code Generation](code-generation.md) - [Custom Serializers](custom-serializers.md) +- [Web Platform Support](web-platform-support.md) diff --git a/docs/guide/dart/web-platform-support.md b/docs/guide/dart/web-platform-support.md new file mode 100644 index 0000000000..3ca6154269 --- /dev/null +++ b/docs/guide/dart/web-platform-support.md @@ -0,0 +1,218 @@ +--- +title: Web Platform Support +sidebar_position: 10 +id: dart_web_platform_support +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Fory Dart supports Dart VM/AOT, Flutter, browser, and Flutter web builds +through generated serializers and platform-specific runtime implementations. +The public API and registration flow are the same across these platforms, but +web builds have stricter integer precision rules because Dart `int` is +represented by JavaScript numbers. + +## Supported Targets + +The Dart runtime supports: + +- Dart VM/JIT applications. +- Dart AOT/native applications. +- Flutter mobile and desktop applications. +- Dart applications compiled to JavaScript for browsers. +- Flutter web applications. +- Generated `@ForyStruct` serializers and manually registered serializers on + all supported targets. + +## Code Generation Is Required + +Fory Dart uses explicit registration instead of runtime reflection. For +annotated structs, run code generation and register the generated serializer +before serializing or deserializing values: + +```dart +import 'package:fory/fory.dart'; + +part 'account.fory.dart'; + +@ForyStruct() +class Account { + Account(); + + String name = ''; + Int64 sequence = Int64(0); +} + +void main() { + final fory = Fory(); + AccountFory.register( + fory, + Account, + namespace: 'example', + typeName: 'Account', + ); + + final bytes = fory.serialize(Account()..name = 'web'); + final account = fory.deserialize(bytes); + print(account.name); +} +``` + +Generate the companion file before building or testing: + +```bash +cd dart/packages/fory +dart run build_runner build --delete-conflicting-outputs +``` + +The registration call is the same on VM/AOT, Flutter, and web. Manual +serializers use `registerSerializer(...)`; generated structs use the generated +`register` wrapper. + +## 64-Bit Integer Rules + +Dart VM `int` values are signed 64-bit values. Dart web `int` values are backed +by JavaScript numbers and are precise only in the JS-safe integer range: + +```text +-9007199254740991 <= value <= 9007199254740991 +``` + +Use this rule when choosing field types: + +| Logical value | Recommended Dart field type on web | Notes | +| ---------------------------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------- | +| Signed 64-bit value within JS-safe range | `int` | Works with default `int64` mapping and `@Int64Type` encodings. | +| Full signed 64-bit range | `Int64` | Preserves values outside the JS-safe range. | +| Unsigned 64-bit value | `Uint64` | Required for values that do not fit in signed or JS-safe Dart `int`. | +| 8/16/32-bit integer | `Int8`, `Int16`, `Int32`, `Uint8`, `Uint16`, `Uint32` or annotations | Use wrappers or numeric annotations to match peer runtimes exactly. | + +`@Int64Type` controls the wire encoding of a Dart `int` field: + +```dart +@ForyStruct() +class SafeCounter { + SafeCounter(); + + @Int64Type(encoding: LongEncoding.tagged) + int count = 0; // keep web values inside the JS-safe range +} +``` + +It does not make Dart `int` capable of storing every 64-bit value on web. For +full-range signed values, use `Int64`: + +```dart +@ForyStruct() +class FullRangeCounter { + FullRangeCounter(); + + Int64 count = Int64(0); +} +``` + +For unsigned values, use `Uint64`: + +```dart +@ForyStruct() +class StorageExtent { + StorageExtent(); + + Uint64 byteOffset = Uint64(0); +} +``` + +## Custom Serializers + +Custom serializers can use the same `Buffer`, `WriteContext`, and `ReadContext` +APIs on VM/AOT, Flutter, and web. For 64-bit values: + +- Use `buffer.writeInt64(Int64(...))` and `buffer.readInt64()` for full-range + signed 64-bit values. +- Use `buffer.writeUint64(Uint64(...))` and `buffer.readUint64()` for full-range + unsigned 64-bit values. +- Use `writeInt64FromInt`, `writeVarInt64FromInt`, and matching `AsInt` reads + only when the value is intended to be a Dart `int` and therefore must stay + JS-safe on web. + +Example: + +```dart +final class OffsetSerializer extends Serializer { + const OffsetSerializer(); + + @override + void write(WriteContext context, StorageExtent value) { + context.buffer.writeUint64(value.byteOffset); + } + + @override + StorageExtent read(ReadContext context) { + return StorageExtent()..byteOffset = context.buffer.readUint64(); + } +} +``` + +## Collections And Typed Arrays + +`List`, `Set`, `Map`, `Uint8List`, numeric typed arrays, `Int64List`, and +`Uint64List` are supported on web. The `Int64List` and `Uint64List` +implementations preserve 64-bit values without depending on JavaScript integer +precision. Use the Fory wrapper list types when the wire type is `int64_array` +or `uint64_array`. + +## Testing Browser Builds + +Run the package tests in both VM and Chrome when changing code that must work on +web: + +```bash +cd dart/packages/fory +dart run build_runner build --delete-conflicting-outputs +dart test +dart test -p chrome +``` + +If Chrome tests fail with a stale generated file or missing part file, rerun +`build_runner` and then retry the test command from `dart/packages/fory`. + +## Common Web Failures + +### `Dart int value ... is outside the JS-safe signed int64 range` + +The serializer is trying to write a Dart `int` as a signed 64-bit value on web, +but the value is outside the range that JavaScript numbers can represent +exactly. Change the field type to `Int64`, or keep the value inside the JS-safe +range. + +### `Int64 value ... is not a JS-safe int` + +The deserializer read a full-range `Int64`, but the target field or custom +serializer asked for a Dart `int`. Change the field type to `Int64`, or decode +with `readInt64()` instead of an `AsInt` helper. + +### `Uint64 value ... is not a JS-safe int` + +The code is converting a `Uint64` to Dart `int` on web. Keep the value as +`Uint64` unless the application has already validated that it is in the +JS-safe non-negative range. + +## Related Topics + +- [Supported Types](supported-types.md) +- [Field Configuration](field-configuration.md) +- [Code Generation](code-generation.md) +- [Troubleshooting](troubleshooting.md) diff --git a/docs/specification/xlang_implementation_guide.md b/docs/specification/xlang_implementation_guide.md index e83384122a..7537633183 100644 --- a/docs/specification/xlang_implementation_guide.md +++ b/docs/specification/xlang_implementation_guide.md @@ -366,17 +366,19 @@ The normal Dart integration path is: 1. annotate structs with `@ForyStruct` 2. annotate field overrides with `@ForyField` 3. run `build_runner` -4. from the source library, bind the generated metadata privately and register - generated types through `Fory.register(...)` +4. call the generated per-library helper, such as + `Fory.register(...)`, to bind private generated metadata and + register generated types Generated code should emit: - private serializer classes - private metadata constants -- private generated installation helpers per annotated library -- generated binding installation that keeps serializer factories private +- a public per-library registration helper that users call from application code +- private generated installation helpers that keep serializer factories private -Generated code should not create a public global registry or a second public API +The public helper should be a thin generated wrapper around the runtime +registration API, not a public global registry or a second unrelated runtime API family. ## Directory Layout