Skip to content

Commit d477433

Browse files
SelaseKaySelaseKay
authored
feat(firestore): introduce pipeline functionality (Dart API) (#18096)
* feat(firestore): introduce pipeline functionality (Dart API) * fix * fix: restore unrelated regressions in settings, changelog, pubspec and exports Made-with: Cursor * fix * chore: fix formatting * chore: add pipeline stages tests nd fix bugs * chore: remove unnecessary parentheses in pipeline stage tests * test: add aggregate stage serialization tests for pipeline stages * test: add unit tests for pipeline expressions, ordering, sampling, and snapshot handling * chore: improve formatting and readability in pipeline test files * chore: update docs for PipelineSnapshot and PipelineResult * chore: fix ci * test: add serialization tests for arrayContainsAll and related expressions in pipeline expressions * feat: add arrayContainsAll expressions for enhanced array handling in pipeline expressions * refactor: remove PipelineFilter * fix: update arrayConcat method to accept Object? and convert to Expression * chore: remove unsupported expressions * chore: add timestamp unit validation and enhance timestamp expression methods * refactor: rename replace methods to stringReplaceAll * refactor: update test for stringReplaceAll serialization * chore: fix ci * chore: fix ci * docs: add documentation with examples for pipeline functions * chore: add pipeline_example Flutter project and integration tests * refactor: improve code formatting and add new execute options tests * fix ci * chore: add pipeline e2e test workflow and script for cloud_firestore * refactor: remove commented-out button for stringReplaceAll in pipeline example * chore: update e2e tests pipeline to include additional branches for triggering * chore: add test comment in pipeline example for clarity * chore: rename e2e pipeline and add iOS testing steps with Firebase configuration * test ci * chore: add workflow_dispatch trigger to e2e tests pipeline * chore: remove Firebase options condition from e2e tests pipeline jobs * chore: update e2e tests pipeline to run on macOS and implement flutter drive for integration tests * chore: update cloud_firestore dependency to use local path in pipeline example * chore: add dependency overrides for local development in cloud_firestore pipeline example * chore: update analysis options and refactor integration tests * refactor: improve formatting * fix ci * chore: remove Linux pipeline example files and associated configurations * chore: refine e2e tests pipeline triggers and enhance iOS project setup for Swift Package Manager * chore: add branch trigger for firestore-pipelines-ios in e2e tests pipeline * chore: update minimum target version for iOS to 15 * chore: add iOS and macOS paths to e2e tests pipeline triggers * refactor: improve code formatting in pipeline integration tests * chore: add branch triggers for Android and Web in e2e tests pipeline * fix formatting * trigger CI * chore: add more expression tests * chore: add more tests * fix ci * chore: include web package in e2e tests pipeline * chore: remove support for mapFromPairs(not supported on sdks) * chore: remove support for arraySlice(not supported on SDKs) * chore: remove arraySlice and _ArraySliceExpression due to SDK incompatibility * feat(firestore, android): Android implementation for Pipeline APIs (#18098) * feat(firestore): Android implementation for Pipeline APIs * refactor: improve handling of timestamps and array expressions in pipeline * chore: add support for more Expression functions in Android * chore: remove unsupported 'replace' expression from Android Firestore pipeline API * chore: enhance PipelineParser to support execute options for Firestore pipelines * chore: remove commented-out button for stringReplaceAll in pipeline example * trigger CI * trigger CI * trigger CI * refactor: enhance expression parsing logic in ExpressionParsers * feat(firestore, web): Web implementation for Pipeline APIs (#18100) * feat(firestore): Web implementation for Pipeline APIs * chore: fix ci * chore: inject firestore pipelines script for web * chore: add comments for Firestore Pipelines script injection in web implementation * refactor: remove unecessary comment * chore: add support for missing Expressions and fix bugs * chore: fix ci * feat: enhance pipeline expression handling with new expressions and error handling * refactor: rename 'replace' case to 'string_replace_all' in pipeline expression parser * chore: update Pipeline execution to accept options for index mode * fix: resolve failing mapGet test on web * fix: correctly parse 'not' expression arguments * trigger CI * fix tests * fix tests * fix: update type casting for boolean expression in PipelineExpressionParserWeb * fix ci * feat(firestore, iOS): iOS implementation for Pipeline APIs (#18099) * feat(firestore): iOS implementation for Pipeline APIs * fix: conditionally import FIRPipelineBridge.h for iOS * chore: add macOS support for FLTPipelineParser by linking to iOS implementation * fix: conditionally include FIRPipelineBridge.h for macOS support in FLTPipelineParser * chore: add support for missing Expression function * refactor: clean up formatting in FLTPipelineParser.m for improved readability * chore: enhance FLTPipelineParser with support for array and map expressions, ensuring proper argument handling * chore: implement conditional expression and current timestamp handling in FLTPipelineParser * chore: add 'find_nearest' stage support in FLTPipelineParser with validation for vector_field, vector_value, and distance_measure * chore: introduce FLTFirebaseFirestoreErrorCodePipelineParse for improved error handling in pipeline parsing * trigger CI * fix: update header inclusion logic for FirebaseFirestore in FLTPipelineParser.m * trigger CI * chore: enhance FLTPipelineParser with new expression handling * chore: update e2e tests pipeline to ignore specific paths for pull requests and pushes * chore: add new aggregate functions and expression methods * fix: update end-to-end test to conditionally skip based on platform * chore: fix ci * chore: fix ci * feat: add new pipeline expressions for regex, string manipulation, and aggregation * fix ci * fix: add conditional import for FIRVectorValue to FLTPipelineParser.m * fix: update argument passing in PipelineExpressionParserWeb for type expression * refactor: update type handling in pipeline expressions to use enum Type for better clarity and consistency * fix ci
1 parent e9c99ef commit d477433

187 files changed

Lines changed: 20846 additions & 50 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/e2e_tests_pipeline.yaml‎

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
name: e2e-pipeline
2+
3+
concurrency:
4+
group: ${{ github.workflow }}-${{ github.ref }}
5+
cancel-in-progress: true
6+
7+
on:
8+
pull_request:
9+
paths-ignore:
10+
- 'docs/**'
11+
- 'website/**'
12+
- '**/example/**'
13+
- '**.md'
14+
push:
15+
branches:
16+
- main
17+
- firestore-pipelines-dart-api-v2
18+
- firestore-pipelines-ios
19+
- firestore-pipelines-android
20+
- firestore-pipelines-web
21+
paths-ignore:
22+
- 'docs/**'
23+
- 'website/**'
24+
- '**/example/**'
25+
- '**.md'
26+
27+
jobs:
28+
pipeline-e2e-android:
29+
runs-on: ubuntu-latest
30+
timeout-minutes: 45
31+
env:
32+
AVD_ARCH: x86_64
33+
AVD_API_LEVEL: 34
34+
AVD_TARGET: google_apis
35+
steps:
36+
- uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938
37+
- uses: actions/setup-node@1d0ff469b7ec7b3cb9d8673fde0c81c44821de2a
38+
name: Install Node.js 20
39+
with:
40+
node-version: '20'
41+
- uses: actions/setup-java@8df1039502a15bceb9433410b1a100fbe190c53b
42+
with:
43+
distribution: 'temurin'
44+
java-version: '21'
45+
- uses: subosito/flutter-action@f2c4f6686ca8e8d6e6d0f28410eeef506ed66aff
46+
with:
47+
channel: 'stable'
48+
cache: true
49+
cache-key: "flutter-:os:-:channel:-:version:-:arch:-:hash:"
50+
pub-cache-key: "flutter-pub-:os:-:channel:-:version:-:arch:-:hash:"
51+
- uses: bluefireteam/melos-action@c7dcb921b23cc520cace360b95d02b37bf09cdaa
52+
with:
53+
run-bootstrap: false
54+
melos-version: '5.3.0'
55+
- name: Inject Firebase config for pipeline E2E
56+
env:
57+
FIREBASE_OPTIONS_DART: ${{ secrets.PIPELINE_E2E_FIREBASE_OPTIONS_DART }}
58+
GOOGLE_SERVICES_JSON: ${{ secrets.PIPELINE_E2E_GOOGLE_SERVICES_JSON }}
59+
GOOGLE_SERVICE_INFO_PLIST: ${{ secrets.PIPELINE_E2E_GOOGLE_SERVICE_INFO_PLIST }}
60+
run: |
61+
echo "$FIREBASE_OPTIONS_DART" > packages/cloud_firestore/cloud_firestore/pipeline_example/lib/firebase_options.dart
62+
echo "$GOOGLE_SERVICES_JSON" > packages/cloud_firestore/cloud_firestore/pipeline_example/android/app/google-services.json
63+
echo "$GOOGLE_SERVICE_INFO_PLIST" > packages/cloud_firestore/cloud_firestore/pipeline_example/ios/Runner/GoogleService-Info.plist
64+
- name: Bootstrap package
65+
run: melos bootstrap --scope "cloud_firestore*"
66+
- name: Enable KVM
67+
run: |
68+
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
69+
sudo udevadm control --reload-rules
70+
sudo udevadm trigger --name-match=kvm
71+
- name: Gradle cache
72+
uses: gradle/actions/setup-gradle@v4
73+
- name: Free Disk Space (Ubuntu)
74+
uses: AdityaGarg8/remove-unwanted-software@90e01b21170618765a73370fcc3abbd1684a7793
75+
with:
76+
remove-dotnet: true
77+
remove-haskell: true
78+
remove-codeql: true
79+
remove-docker-images: true
80+
remove-large-packages: true
81+
- name: AVD cache
82+
uses: actions/cache/restore@1bd1e32a3bdc45362d1e726936510720a7c30a57
83+
continue-on-error: true
84+
id: avd-cache
85+
with:
86+
path: |
87+
~/.android/avd/*
88+
~/.android/adb*
89+
key: avd-${{ runner.os }}-${{ env.AVD_API_LEVEL }}-${{ env.AVD_TARGET }}-${{ env.AVD_ARCH }}
90+
- name: Start AVD then run pipeline E2E tests
91+
uses: reactivecircus/android-emulator-runner@b530d96654c385303d652368551fb075bc2f0b6b
92+
with:
93+
api-level: ${{ env.AVD_API_LEVEL }}
94+
target: ${{ env.AVD_TARGET }}
95+
arch: ${{ env.AVD_ARCH }}
96+
emulator-build: 14214601
97+
working-directory: packages/cloud_firestore/cloud_firestore/pipeline_example
98+
script: |
99+
flutter test integration_test/pipeline/pipeline_live_test.dart --timeout 10x --dart-define=CI=true -d emulator-5554
100+
- name: Ensure Appium is shut down
101+
run: |
102+
pgrep -f appium && pkill -f appium || echo "No Appium process found"
103+
- name: Save Android Emulator Cache
104+
if: github.ref == 'refs/heads/main'
105+
uses: actions/cache/save@1bd1e32a3bdc45362d1e726936510720a7c30a57
106+
continue-on-error: true
107+
with:
108+
key: ${{ steps.avd-cache.outputs.cache-primary-key }}
109+
path: |
110+
~/.android/avd/*
111+
~/.android/adb*
112+
113+
pipeline-e2e-web:
114+
runs-on: macos-latest
115+
timeout-minutes: 25
116+
steps:
117+
- uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938
118+
- uses: actions/setup-node@1d0ff469b7ec7b3cb9d8673fde0c81c44821de2a
119+
name: Install Node.js 20
120+
with:
121+
node-version: '20'
122+
- uses: subosito/flutter-action@f2c4f6686ca8e8d6e6d0f28410eeef506ed66aff
123+
with:
124+
channel: 'stable'
125+
cache: true
126+
cache-key: "flutter-:os:-:channel:-:version:-:arch:-:hash:"
127+
pub-cache-key: "flutter-pub-:os:-:channel:-:version:-:arch:-:hash:"
128+
- uses: bluefireteam/melos-action@c7dcb921b23cc520cace360b95d02b37bf09cdaa
129+
with:
130+
run-bootstrap: false
131+
melos-version: '5.3.0'
132+
- name: Inject Firebase config for pipeline E2E
133+
env:
134+
FIREBASE_OPTIONS_DART: ${{ secrets.PIPELINE_E2E_FIREBASE_OPTIONS_DART }}
135+
GOOGLE_SERVICES_JSON: ${{ secrets.PIPELINE_E2E_GOOGLE_SERVICES_JSON }}
136+
GOOGLE_SERVICE_INFO_PLIST: ${{ secrets.PIPELINE_E2E_GOOGLE_SERVICE_INFO_PLIST }}
137+
run: |
138+
echo "$FIREBASE_OPTIONS_DART" > packages/cloud_firestore/cloud_firestore/pipeline_example/lib/firebase_options.dart
139+
echo "$GOOGLE_SERVICES_JSON" > packages/cloud_firestore/cloud_firestore/pipeline_example/android/app/google-services.json
140+
echo "$GOOGLE_SERVICE_INFO_PLIST" > packages/cloud_firestore/cloud_firestore/pipeline_example/ios/Runner/GoogleService-Info.plist
141+
- name: Bootstrap package
142+
run: melos bootstrap --scope "cloud_firestore*"
143+
- name: Run pipeline E2E tests (Chrome)
144+
working-directory: packages/cloud_firestore/cloud_firestore/pipeline_example
145+
# Web devices are not supported for the `flutter test` command yet. As a
146+
# workaround we use the `flutter drive` command. Tracking issue:
147+
# https://github.com/flutter/flutter/issues/66264
148+
run: |
149+
chromedriver --port=4444 --trace-buffer-size=100000 &
150+
flutter drive --target=./integration_test/pipeline/pipeline_live_test.dart --driver=./test_driver/integration_test.dart -d chrome --dart-define=CI=true | tee output.log
151+
output=$(<output.log)
152+
if [[ "$output" =~ \[E\] ]]; then
153+
echo "All tests did not pass. Please check the logs for more information."
154+
exit 1
155+
fi
156+
shell: bash
157+
158+
pipeline-e2e-ios:
159+
runs-on: macos-15
160+
timeout-minutes: 50
161+
steps:
162+
- uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938
163+
- uses: actions/setup-node@1d0ff469b7ec7b3cb9d8673fde0c81c44821de2a
164+
name: Install Node.js 20
165+
with:
166+
node-version: '20'
167+
- name: Xcode
168+
run: sudo xcode-select -s /Applications/Xcode_16.4.app/Contents/Developer
169+
- uses: actions/setup-java@8df1039502a15bceb9433410b1a100fbe190c53b
170+
with:
171+
distribution: 'temurin'
172+
java-version: '21'
173+
- uses: subosito/flutter-action@f2c4f6686ca8e8d6e6d0f28410eeef506ed66aff
174+
with:
175+
channel: 'stable'
176+
cache: true
177+
cache-key: "flutter-:os:-:channel:-:version:-:arch:-:hash:"
178+
pub-cache-key: "flutter-pub-:os:-:channel:-:version:-:arch:-:hash:"
179+
- name: Enable Swift Package Manager for iOS
180+
run: flutter config --enable-swift-package-manager
181+
- uses: bluefireteam/melos-action@c7dcb921b23cc520cace360b95d02b37bf09cdaa
182+
with:
183+
run-bootstrap: false
184+
melos-version: '5.3.0'
185+
- name: Inject Firebase config for pipeline E2E
186+
env:
187+
FIREBASE_OPTIONS_DART: ${{ secrets.PIPELINE_E2E_FIREBASE_OPTIONS_DART }}
188+
GOOGLE_SERVICES_JSON: ${{ secrets.PIPELINE_E2E_GOOGLE_SERVICES_JSON }}
189+
GOOGLE_SERVICE_INFO_PLIST: ${{ secrets.PIPELINE_E2E_GOOGLE_SERVICE_INFO_PLIST }}
190+
run: |
191+
echo "$FIREBASE_OPTIONS_DART" > packages/cloud_firestore/cloud_firestore/pipeline_example/lib/firebase_options.dart
192+
echo "$GOOGLE_SERVICES_JSON" > packages/cloud_firestore/cloud_firestore/pipeline_example/android/app/google-services.json
193+
echo "$GOOGLE_SERVICE_INFO_PLIST" > packages/cloud_firestore/cloud_firestore/pipeline_example/ios/Runner/GoogleService-Info.plist
194+
- name: Bootstrap package
195+
run: melos bootstrap --scope "cloud_firestore*"
196+
- name: Prepare iOS project for Swift Package Manager
197+
working-directory: packages/cloud_firestore/cloud_firestore/pipeline_example/ios
198+
run: |
199+
if [ -f Podfile ]; then pod deintegrate; fi
200+
rm -f Podfile Podfile.lock
201+
rm -rf Pods
202+
- uses: futureware-tech/simulator-action@e89aa8f93d3aec35083ff49d2854d07f7186f7f5
203+
id: simulator
204+
with:
205+
model: "iPhone 16"
206+
- name: Build iOS (simulator)
207+
working-directory: packages/cloud_firestore/cloud_firestore/pipeline_example
208+
run: |
209+
flutter build ios --no-codesign --simulator --debug --target=./integration_test/pipeline/pipeline_live_test.dart --dart-define=CI=true
210+
- name: Run pipeline E2E tests (iOS)
211+
working-directory: packages/cloud_firestore/cloud_firestore/pipeline_example
212+
env:
213+
SIMULATOR: ${{ steps.simulator.outputs.udid }}
214+
run: |
215+
flutter test integration_test/pipeline/pipeline_live_test.dart -d "$SIMULATOR" --timeout 10x --dart-define=CI=true

‎melos.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,13 @@ scripts:
172172
description: |
173173
Run all e2e tests for cloud_firestore.
174174
175+
test:e2e:pipeline:
176+
run: |
177+
cd packages/cloud_firestore/cloud_firestore/pipeline_example
178+
flutter test integration_test/pipeline/pipeline_live_test.dart
179+
description: |
180+
Run pipeline e2e tests.
181+
175182
test:e2e:firebase_performance:
176183
run: |
177184
cd packages/firebase_performance/firebase_performance/example

‎packages/cloud_firestore/cloud_firestore/android/src/main/java/io/flutter/plugins/firebase/firestore/FlutterFirebaseFirestorePlugin.java‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@
2828
import com.google.firebase.firestore.MemoryCacheSettings;
2929
import com.google.firebase.firestore.PersistentCacheIndexManager;
3030
import com.google.firebase.firestore.PersistentCacheSettings;
31+
import com.google.firebase.firestore.Pipeline;
32+
import com.google.firebase.firestore.PipelineResult;
3133
import com.google.firebase.firestore.Query;
3234
import com.google.firebase.firestore.QuerySnapshot;
3335
import com.google.firebase.firestore.SetOptions;
@@ -52,6 +54,7 @@
5254
import io.flutter.plugins.firebase.firestore.streamhandler.TransactionStreamHandler;
5355
import io.flutter.plugins.firebase.firestore.utils.ExceptionConverter;
5456
import io.flutter.plugins.firebase.firestore.utils.PigeonParser;
57+
import io.flutter.plugins.firebase.firestore.utils.PipelineParser;
5558
import java.util.ArrayList;
5659
import java.util.HashMap;
5760
import java.util.List;
@@ -1007,4 +1010,65 @@ public void documentReferenceSnapshot(
10071010
parameters.getServerTimestampBehavior()),
10081011
PigeonParser.parseListenSource(source))));
10091012
}
1013+
1014+
@Override
1015+
public void executePipeline(
1016+
@NonNull GeneratedAndroidFirebaseFirestore.FirestorePigeonFirebaseApp app,
1017+
@NonNull List<Map<String, Object>> stages,
1018+
@Nullable Map<String, Object> options,
1019+
@NonNull
1020+
GeneratedAndroidFirebaseFirestore.Result<
1021+
GeneratedAndroidFirebaseFirestore.PigeonPipelineSnapshot>
1022+
result) {
1023+
cachedThreadPool.execute(
1024+
() -> {
1025+
try {
1026+
FirebaseFirestore firestore = getFirestoreFromPigeon(app);
1027+
1028+
// Execute pipeline using Android Firestore SDK
1029+
Pipeline.Snapshot snapshot = PipelineParser.executePipeline(firestore, stages, options);
1030+
1031+
// Convert Pipeline.Snapshot to PigeonPipelineSnapshot
1032+
List<GeneratedAndroidFirebaseFirestore.PigeonPipelineResult> pipelineResults =
1033+
new ArrayList<>();
1034+
1035+
// Iterate through snapshot results
1036+
for (PipelineResult pipelineResult : snapshot.getResults()) {
1037+
GeneratedAndroidFirebaseFirestore.PigeonPipelineResult.Builder resultBuilder =
1038+
new GeneratedAndroidFirebaseFirestore.PigeonPipelineResult.Builder();
1039+
if (pipelineResult.getRef() != null) {
1040+
resultBuilder.setDocumentPath(pipelineResult.getRef().getPath());
1041+
}
1042+
1043+
if (pipelineResult.getCreateTime() != null) {
1044+
resultBuilder.setCreateTime(pipelineResult.getCreateTime().toDate().getTime());
1045+
}
1046+
if (pipelineResult.getUpdateTime() != null) {
1047+
resultBuilder.setUpdateTime(pipelineResult.getUpdateTime().toDate().getTime());
1048+
}
1049+
1050+
Map<String, Object> data = pipelineResult.getData();
1051+
if (data != null) {
1052+
resultBuilder.setData(data);
1053+
}
1054+
1055+
pipelineResults.add(resultBuilder.build());
1056+
}
1057+
1058+
// Build the snapshot
1059+
GeneratedAndroidFirebaseFirestore.PigeonPipelineSnapshot.Builder snapshotBuilder =
1060+
new GeneratedAndroidFirebaseFirestore.PigeonPipelineSnapshot.Builder();
1061+
snapshotBuilder.setResults(pipelineResults);
1062+
1063+
// Set execution time when available. Do not fabricate a value when null.
1064+
if (snapshot.getExecutionTime() != null) {
1065+
snapshotBuilder.setExecutionTime(snapshot.getExecutionTime().toDate().getTime());
1066+
}
1067+
1068+
result.success(snapshotBuilder.build());
1069+
} catch (Exception e) {
1070+
ExceptionConverter.sendErrorToFlutter(result, e);
1071+
}
1072+
});
1073+
}
10101074
}

0 commit comments

Comments
 (0)