Open
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6845 +/- ##
=======================================
Coverage 43.22% 43.22%
=======================================
Files 99 99
Lines 7769 7769
Branches 3400 3400
=======================================
Hits 3358 3358
Misses 4411 4411 🚀 New features to boost your workflow:
|
ekjotmultani
changed the title
ekjotmultani
changed the title
ekjotmultani
force-pushed
the
feat/amplify-firehose-client-feature
branch
2 times, most recently
from
a756f48 to
dbfee7f
Compare
…structure (#6826) feat(firehose): scaffold amplify_firehose_dart package Create the Dart-only Firehose client package with: - pubspec.yaml (SDK ^3.9.0, matching KDS dependency set) - sdk.yaml for PutRecordBatch operation - Generated Smithy SDK client (firehose_client, models, serializers) - Firehose API limits (500 records, 1000 KB/record, 4 MB/batch) - version.dart, analysis_options, dart_test.yaml, .gitignore - Skeleton barrel export (amplify_firehose_dart.dart) - Register Firehose component in root pubspec.yaml No business logic yet
* refactor(kinesis): extract shared record cache into amplify_record_cache_dart Create amplify_record_cache_dart package with shared caching infrastructure: - RecordCacheException hierarchy (const constructors) - Record/RecordInput models (partitionKey optional, dataSize caller-computed) - RecordStorage base + SqliteRecordStorage, InMemoryRecordStorage, IndexedDbRecordStorage - RecordCacheDatabase (Drift, parameterized dbPrefix) - Sender interface + SendResult (replaces KDS-specific PutRecordsResult) - RecordClient, AutoFlushScheduler, FlushStrategy, FlushData, RecordData, ClearCacheData - Platform resolution (VM/web/stub conditional exports) Update amplify_kinesis_dart to depend on shared package: - KinesisSender implements Sender interface (sendBatch replaces putRecords) - Partition key validation moved from RecordStorage to AmplifyKinesisClient - createKinesisRecordInputNow computes dataSize with partition key - All test imports updated, zero behavioral change
* feat(firehose): amplify firehose client sdk generation and directory structure (#6826) feat(firehose): scaffold amplify_firehose_dart package Create the Dart-only Firehose client package with: - pubspec.yaml (SDK ^3.9.0, matching KDS dependency set) - sdk.yaml for PutRecordBatch operation - Generated Smithy SDK client (firehose_client, models, serializers) - Firehose API limits (500 records, 1000 KB/record, 4 MB/batch) - version.dart, analysis_options, dart_test.yaml, .gitignore - Skeleton barrel export (amplify_firehose_dart.dart) - Register Firehose component in root pubspec.yaml No business logic yet * refactor(kinesis): extract shared record cache into amplify_record_cache_dart Create amplify_record_cache_dart package with shared caching infrastructure: - RecordCacheException hierarchy (const constructors) - Record/RecordInput models (partitionKey optional, dataSize caller-computed) - RecordStorage base + SqliteRecordStorage, InMemoryRecordStorage, IndexedDbRecordStorage - RecordCacheDatabase (Drift, parameterized dbPrefix) - Sender interface + SendResult (replaces KDS-specific PutRecordsResult) - RecordClient, AutoFlushScheduler, FlushStrategy, FlushData, RecordData, ClearCacheData - Platform resolution (VM/web/stub conditional exports) Update amplify_kinesis_dart to depend on shared package: - KinesisSender implements Sender interface (sendBatch replaces putRecords) - Partition key validation moved from RecordStorage to AmplifyKinesisClient - createKinesisRecordInputNow computes dataSize with partition key - All test imports updated, zero behavioral change * chore: revert dependabot.yaml changes from aft generate The aft generate workflows command regenerated the entire dependabot.yaml with entries for many unrelated packages. Reverting to keep this PR scoped to the record cache extraction only. * fix: remove IndexedDB storage from barrel export to fix VM test compilation The barrel file was unconditionally exporting record_storage_indexeddb.dart which imports dart:js_interop. This caused VM tests to fail with "Dart library 'dart:js_interop' is not available on this platform". IndexedDB storage is only reachable through the platform conditional export (record_storage_platform_web.dart), not the barrel. * fix(test): close default client before creating large-cache client in validation test Drift warns when multiple RecordCacheDatabase instances exist simultaneously. Close the default client first, then reassign so tearDown handles cleanup. * fix(test): increase maxBytesPerBatch in large-cache validation test The 10 MiB test record exceeded the 5 MiB batch limit, causing getRecordsByStream to filter it out. Bumped to 20 MiB to match the cache size. * fix(test): use correct KDS maxBytesPerBatch (10 MiB) in all test files All test files were using 5 MiB for maxBytesPerBatch, but the KDS default was 10 MiB (matching maxRecordSizeBytes). This mismatch caused getRecordsByStream to filter out records at the max size limit. * feat(firehose): add exception hierarchy and depend on shared record cache - AmplifyFirehoseException sealed hierarchy with .from() mapper - FirehoseStorageException, FirehoseValidationException, FirehoseLimitExceededException, FirehoseUnknownException, FirehoseClientClosedException - Depend on amplify_record_cache_dart for storage/caching infrastructure - Re-export shared FlushStrategy, FlushData, RecordData, ClearCacheData - Export Firehose SDK escape hatch types - Exception mapping tests - Slim pubspec: removed direct drift/web/db deps (now transitive via shared pkg) * feat(firehose): add FirehoseSender, AmplifyFirehoseClient, and client options - FirehoseSender implements shared Sender interface (calls PutRecordBatch) - AmplifyFirehoseClient with create(), record(), flush(), clearCache(), enable(), disable(), close() — mirrors KDS client structure - AmplifyFirehoseClientOptions (cacheMaxBytes, maxRetries, flushStrategy) - record() computes dataSize as data.length (no partition key) - Uses shared RecordClient, AutoFlushScheduler, platform storage - SDK escape hatch via firehoseClient getter - withRecordClient constructor for testing * test(firehose): add client and sender tests, remove unused defaultRecoverySuggestion - AmplifyFirehoseClient tests: initialization, record(), flush(), clearCache(), enable/disable, close, closed-state errors - FirehoseSender tests: request building, response categorization (success/retryable/failed), empty records handling - Remove unused defaultRecoverySuggestion from exception file - Clean up pubspec_overrides (remove stale amplify_kinesis_dart override) * docs: restore stripped doc comments and error code comment in KDS client Restore detailed dartdoc comments on AmplifyKinesisClient methods (create, kinesisClient, record, flush, clearCache, disable, close, _wrapError) and the error code comment in KinesisSender that were accidentally removed during the extraction rewrite. * fix: assert non-null partitionKey in KinesisSender instead of defaulting to empty string KDS always provides a partition key. Using ?? '' silently hides bugs. Use ! to assert non-null since KDS records always have partitionKey set. * chore: export defaultRecoverySuggestion from shared package, mark as internal, document drift dev dep - Export defaultRecoverySuggestion from shared barrel (remove hide clause) - KDS exception file now uses the shared constant instead of its own - Mark amplify_record_cache_dart as internal (publish_to: none) - Add comment explaining why drift is a dev dependency in KDS * fix: rename Drift table class back to KinesisRecords to avoid breaking change The table name in SQLite is derived from the Drift class name. Renaming from KinesisRecords to CachedRecords would change the table from 'kinesis_records' to 'cached_records', breaking existing users. Firehose is in the Kinesis family so the name is semantically fine. Regenerated .g.dart and updated all references. * feat(firehose): add exception hierarchy and depend on shared record cache - AmplifyFirehoseException sealed hierarchy with .from() mapper - FirehoseStorageException, FirehoseValidationException, FirehoseLimitExceededException, FirehoseUnknownException, FirehoseClientClosedException - Depend on amplify_record_cache_dart for storage/caching infrastructure - Re-export shared FlushStrategy, FlushData, RecordData, ClearCacheData - Export Firehose SDK escape hatch types - Exception mapping tests - Slim pubspec: removed direct drift/web/db deps (now transitive via shared pkg) * feat(firehose): add FirehoseSender, AmplifyFirehoseClient, and client options - FirehoseSender implements shared Sender interface (calls PutRecordBatch) - AmplifyFirehoseClient with create(), record(), flush(), clearCache(), enable(), disable(), close() — mirrors KDS client structure - AmplifyFirehoseClientOptions (cacheMaxBytes, maxRetries, flushStrategy) - record() computes dataSize as data.length (no partition key) - Uses shared RecordClient, AutoFlushScheduler, platform storage - SDK escape hatch via firehoseClient getter - withRecordClient constructor for testing * test(firehose): add client and sender tests, remove unused defaultRecoverySuggestion - AmplifyFirehoseClient tests: initialization, record(), flush(), clearCache(), enable/disable, close, closed-state errors - FirehoseSender tests: request building, response categorization (success/retryable/failed), empty records handling - Remove unused defaultRecoverySuggestion from exception file - Clean up pubspec_overrides (remove stale amplify_kinesis_dart override) * refactor: collapse dbPrefix and storeName into single storageName parameter Both params always had the same value. Simplified to one storageName used for both the SQLite database file name and IndexedDB store name. * refactor: extract shared splitResults helper for response categorization Both KDS and Firehose senders had identical logic to split batch responses into success/retryable/failed buckets. Extracted into a shared splitResults() function in the record cache package, matching the Android implementation pattern. * chore: format sender files * Revert "chore: format sender files" This reverts commit 566a03b. * Reapply "chore: format sender files" This reverts commit 9dadf7b. * Revert "refactor: collapse dbPrefix and storeName into single storageName parameter" This reverts commit 24a5655. * fix: correct KDS storage names to match released client KDS uses different naming for SQLite (kinesis_records_$id) vs IndexedDB (amplify_kinesis_$id). Updated dbPrefix to 'amplify_kinesis' and storeName to 'kinesis_records' to match the released client. SQLite database now uses storeName for file naming, dbPrefix for IndexedDB. Firehose follows same pattern: amplify_firehose / firehose_records. * chore: remove accidentally committed untracked/ephemeral files Remove generated and ephemeral files from aws_kinesis_datastreams/example/ and amplify_kinesis/example/ that were accidentally included in a previous commit. These files (CDK outputs, Flutter ephemeral files, iOS/macOS generated plugin registrants) should not be tracked. * fix: add trailing underscore to dbPrefix for IndexedDB compatibility The released KDS client on main uses 'amplify_kinesis_$identifier' for the IndexedDB database name. The shared package concatenates '$dbPrefix$identifier', so dbPrefix must include the trailing underscore to produce the same name and avoid breaking existing customers' cached data. KDS: 'amplify_kinesis' → 'amplify_kinesis_' Firehose: 'amplify_firehose' → 'amplify_firehose_'
* feat(firehose): add amplify_firehose Flutter wrapper package - AmplifyFirehoseClient Flutter wrapper that resolves storage path via path_provider automatically - Platform-conditional path provider (IO/web/stub) - Re-exports all types from amplify_firehose_dart except the Dart client - Mirrors amplify_kinesis Flutter wrapper structure * chore: remove accidentally committed flutter crash log * chore(firehose): add CHANGELOG, LICENSE, and README to Flutter wrapper * feat(firehose): add integration test infrastructure and example app - CDK backend (infra-gen2/backends/kinesis/firehose/) with Firehose delivery stream to S3, auth with email sign-in, IAM permissions for PutRecord/PutRecordBatch - Register firehose backend in deploy_gen2.dart - Example Flutter app with Amplify Auth + Firehose client demo - Integration tests: record/flush happy path, cache behavior, batch records, client lifecycle - Setup utils with AmplifyAuthCredentialsProvider - pull_test_backend.sh for CI config download Note: Backend needs to be deployed and S3 bucket configured before CI can run integration tests. * chore: regenerate workflows for firehose packages via aft generate workflows Adds amplify_firehose.yaml, amplify_firehose_example.yaml workflows and updates amplify_firehose_dart.yaml with native test job and shared package path triggers. Note: dependabot.yaml changes from aft excluded (repo-wide regen). * chore: split Firehose into own deploy category and add backend scaffolding - Move firehose backend from Kinesis category to its own Firehose category in deploy_gen2.dart so it deploys independently without touching the released KDS backend. - Add missing npm scaffolding files (package.json, tsconfig.json, .gitignore) for the firehose backend. - Update pull_test_backend.sh to use AFS_FIREHOSE_GEN2_BUCKET_NAME. - Simplify setup_utils to use amplifyConfig directly (no amplifyEnvironments needed with dedicated category). * fix: add network client entitlement for macOS integration tests * test(firehose): port full integration test suite from KDS Add all test cases from the KDS integration tests, adapted for Firehose (no partition key, 4 MiB batch limit, 1000 KiB record limit): - Core happy path (record+flush, empty flush, disabled drops, enable/ disable lifecycle, concurrent flush) - Cache behavior (limit exceeded, clear cache) - Error paths (nonexistent stream, invalid credentials, oversized record) - Retry exhaustion (invalid stream dropped after maxRetries) - Stress tests (high volume, repeated cycles, concurrent producers) - Auto-flush (scheduler auto-starts, drains on interval) - PutRecordBatch size limits (payload exceeding batch limit) - Persistence (records survive across client instances) - Escape hatch (SDK client accessible) - Client lifecycle (close prevents operations) * fix: suppress useDataProtection warning in example app for CI * chore: format setup_utils.dart * debug: add --debug flag to pull_test_backend.sh to diagnose S3 403 * revert: remove --debug flag from pull_test_backend.sh * fix: add test_driver for web integration tests * chore: add platform runners for all target platforms Add android, ios, linux, macos, web, and windows platform directories generated by flutter create. These are required for CI integration tests to run on each platform. * chore: bump amplify_firehose version to 2.11.0 to align with release train
ekjotmultani
force-pushed
the
feat/amplify-firehose-client-feature
branch
from
a1791c7 to
ebd1bf8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
Amplify Firehose Library, dart and flutter impl
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.