Commit a756f48
authored
feat(firehose) firehose client (#6850)
* feat(firehose): amplify firehose client sdk generation and directory structure (#6826)
feat(firehose): scaffold amplify_firehose_dart package
Create the Dart-only Firehose client package with:
- pubspec.yaml (SDK ^3.9.0, matching KDS dependency set)
- sdk.yaml for PutRecordBatch operation
- Generated Smithy SDK client (firehose_client, models, serializers)
- Firehose API limits (500 records, 1000 KB/record, 4 MB/batch)
- version.dart, analysis_options, dart_test.yaml, .gitignore
- Skeleton barrel export (amplify_firehose_dart.dart)
- Register Firehose component in root pubspec.yaml
No business logic yet
* refactor(kinesis): extract shared record cache into amplify_record_cache_dart
Create amplify_record_cache_dart package with shared caching infrastructure:
- RecordCacheException hierarchy (const constructors)
- Record/RecordInput models (partitionKey optional, dataSize caller-computed)
- RecordStorage base + SqliteRecordStorage, InMemoryRecordStorage, IndexedDbRecordStorage
- RecordCacheDatabase (Drift, parameterized dbPrefix)
- Sender interface + SendResult (replaces KDS-specific PutRecordsResult)
- RecordClient, AutoFlushScheduler, FlushStrategy, FlushData, RecordData, ClearCacheData
- Platform resolution (VM/web/stub conditional exports)
Update amplify_kinesis_dart to depend on shared package:
- KinesisSender implements Sender interface (sendBatch replaces putRecords)
- Partition key validation moved from RecordStorage to AmplifyKinesisClient
- createKinesisRecordInputNow computes dataSize with partition key
- All test imports updated, zero behavioral change
* chore: revert dependabot.yaml changes from aft generate
The aft generate workflows command regenerated the entire dependabot.yaml
with entries for many unrelated packages. Reverting to keep this PR scoped
to the record cache extraction only.
* fix: remove IndexedDB storage from barrel export to fix VM test compilation
The barrel file was unconditionally exporting record_storage_indexeddb.dart
which imports dart:js_interop. This caused VM tests to fail with
"Dart library 'dart:js_interop' is not available on this platform".
IndexedDB storage is only reachable through the platform conditional
export (record_storage_platform_web.dart), not the barrel.
* fix(test): close default client before creating large-cache client in validation test
Drift warns when multiple RecordCacheDatabase instances exist simultaneously.
Close the default client first, then reassign so tearDown handles cleanup.
* fix(test): increase maxBytesPerBatch in large-cache validation test
The 10 MiB test record exceeded the 5 MiB batch limit, causing
getRecordsByStream to filter it out. Bumped to 20 MiB to match
the cache size.
* fix(test): use correct KDS maxBytesPerBatch (10 MiB) in all test files
All test files were using 5 MiB for maxBytesPerBatch, but the KDS
default was 10 MiB (matching maxRecordSizeBytes). This mismatch
caused getRecordsByStream to filter out records at the max size limit.
* feat(firehose): add exception hierarchy and depend on shared record cache
- AmplifyFirehoseException sealed hierarchy with .from() mapper
- FirehoseStorageException, FirehoseValidationException,
FirehoseLimitExceededException, FirehoseUnknownException,
FirehoseClientClosedException
- Depend on amplify_record_cache_dart for storage/caching infrastructure
- Re-export shared FlushStrategy, FlushData, RecordData, ClearCacheData
- Export Firehose SDK escape hatch types
- Exception mapping tests
- Slim pubspec: removed direct drift/web/db deps (now transitive via shared pkg)
* feat(firehose): add FirehoseSender, AmplifyFirehoseClient, and client options
- FirehoseSender implements shared Sender interface (calls PutRecordBatch)
- AmplifyFirehoseClient with create(), record(), flush(), clearCache(),
enable(), disable(), close() — mirrors KDS client structure
- AmplifyFirehoseClientOptions (cacheMaxBytes, maxRetries, flushStrategy)
- record() computes dataSize as data.length (no partition key)
- Uses shared RecordClient, AutoFlushScheduler, platform storage
- SDK escape hatch via firehoseClient getter
- withRecordClient constructor for testing
* test(firehose): add client and sender tests, remove unused defaultRecoverySuggestion
- AmplifyFirehoseClient tests: initialization, record(), flush(),
clearCache(), enable/disable, close, closed-state errors
- FirehoseSender tests: request building, response categorization
(success/retryable/failed), empty records handling
- Remove unused defaultRecoverySuggestion from exception file
- Clean up pubspec_overrides (remove stale amplify_kinesis_dart override)
* docs: restore stripped doc comments and error code comment in KDS client
Restore detailed dartdoc comments on AmplifyKinesisClient methods
(create, kinesisClient, record, flush, clearCache, disable, close,
_wrapError) and the error code comment in KinesisSender that were
accidentally removed during the extraction rewrite.
* fix: assert non-null partitionKey in KinesisSender instead of defaulting to empty string
KDS always provides a partition key. Using ?? '' silently hides bugs.
Use ! to assert non-null since KDS records always have partitionKey set.
* chore: export defaultRecoverySuggestion from shared package, mark as internal, document drift dev dep
- Export defaultRecoverySuggestion from shared barrel (remove hide clause)
- KDS exception file now uses the shared constant instead of its own
- Mark amplify_record_cache_dart as internal (publish_to: none)
- Add comment explaining why drift is a dev dependency in KDS
* fix: rename Drift table class back to KinesisRecords to avoid breaking change
The table name in SQLite is derived from the Drift class name.
Renaming from KinesisRecords to CachedRecords would change the table
from 'kinesis_records' to 'cached_records', breaking existing users.
Firehose is in the Kinesis family so the name is semantically fine.
Regenerated .g.dart and updated all references.
* feat(firehose): add exception hierarchy and depend on shared record cache
- AmplifyFirehoseException sealed hierarchy with .from() mapper
- FirehoseStorageException, FirehoseValidationException,
FirehoseLimitExceededException, FirehoseUnknownException,
FirehoseClientClosedException
- Depend on amplify_record_cache_dart for storage/caching infrastructure
- Re-export shared FlushStrategy, FlushData, RecordData, ClearCacheData
- Export Firehose SDK escape hatch types
- Exception mapping tests
- Slim pubspec: removed direct drift/web/db deps (now transitive via shared pkg)
* feat(firehose): add FirehoseSender, AmplifyFirehoseClient, and client options
- FirehoseSender implements shared Sender interface (calls PutRecordBatch)
- AmplifyFirehoseClient with create(), record(), flush(), clearCache(),
enable(), disable(), close() — mirrors KDS client structure
- AmplifyFirehoseClientOptions (cacheMaxBytes, maxRetries, flushStrategy)
- record() computes dataSize as data.length (no partition key)
- Uses shared RecordClient, AutoFlushScheduler, platform storage
- SDK escape hatch via firehoseClient getter
- withRecordClient constructor for testing
* test(firehose): add client and sender tests, remove unused defaultRecoverySuggestion
- AmplifyFirehoseClient tests: initialization, record(), flush(),
clearCache(), enable/disable, close, closed-state errors
- FirehoseSender tests: request building, response categorization
(success/retryable/failed), empty records handling
- Remove unused defaultRecoverySuggestion from exception file
- Clean up pubspec_overrides (remove stale amplify_kinesis_dart override)
* refactor: collapse dbPrefix and storeName into single storageName parameter
Both params always had the same value. Simplified to one storageName
used for both the SQLite database file name and IndexedDB store name.
* refactor: extract shared splitResults helper for response categorization
Both KDS and Firehose senders had identical logic to split batch
responses into success/retryable/failed buckets. Extracted into
a shared splitResults() function in the record cache package,
matching the Android implementation pattern.
* chore: format sender files
* Revert "chore: format sender files"
This reverts commit 566a03b.
* Reapply "chore: format sender files"
This reverts commit 9dadf7b.
* Revert "refactor: collapse dbPrefix and storeName into single storageName parameter"
This reverts commit 24a5655.
* fix: correct KDS storage names to match released client
KDS uses different naming for SQLite (kinesis_records_$id) vs IndexedDB
(amplify_kinesis_$id). Updated dbPrefix to 'amplify_kinesis' and
storeName to 'kinesis_records' to match the released client. SQLite
database now uses storeName for file naming, dbPrefix for IndexedDB.
Firehose follows same pattern: amplify_firehose / firehose_records.
* chore: remove accidentally committed untracked/ephemeral files
Remove generated and ephemeral files from aws_kinesis_datastreams/example/
and amplify_kinesis/example/ that were accidentally included in a
previous commit. These files (CDK outputs, Flutter ephemeral files,
iOS/macOS generated plugin registrants) should not be tracked.
* fix: add trailing underscore to dbPrefix for IndexedDB compatibility
The released KDS client on main uses 'amplify_kinesis_$identifier' for
the IndexedDB database name. The shared package concatenates
'$dbPrefix$identifier', so dbPrefix must include the trailing
underscore to produce the same name and avoid breaking existing
customers' cached data.
KDS: 'amplify_kinesis' → 'amplify_kinesis_'
Firehose: 'amplify_firehose' → 'amplify_firehose_'1 parent 7664b1c commit a756f48
File tree
14 files changed
+922
-49
lines changed- packages/kinesis
- amplify_firehose_dart
- lib
- src
- exception
- impl
- test
- amplify_kinesis_dart/lib/src
- impl
- amplify_record_cache_dart/lib/src
- db
- sender
- storage/platform
14 files changed
+922
-49
lines changedLines changed: 22 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
7 | 23 | | |
8 | | - | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
Lines changed: 220 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
Lines changed: 33 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
Lines changed: 111 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
0 commit comments