Implement ensemble test framework (#2258)

* Enhance ensemble test framework with new features and improvements

- Updated .gitignore to include IntelliJ/Android Studio module files and runtime-generated GetStorage files.
- Added new scripts in melos.yaml for running Ensemble YAML tests and generating test schemas.
- Introduced a reset method in Ensemble class to clear singleton state between tests.
- Implemented a new ensemble_test_runner package with a CLI for running declarative YAML tests.
- Added extensive documentation for the new test runner, including a step vocabulary and JSON schema for test files.
- Developed assertion engine and extended step handlers to support a wider range of test actions and assertions.

This commit lays the groundwork for a more robust testing framework for Ensemble applications.

* Update README and schema descriptions to reflect app-local test paths; refactor test discovery and execution to support new directory structure. Add new test cases for helloApp and remove legacy tests. Enhance YAML test app patcher for dynamic test directory resolution.

* Update schema references and regenerate ensemble_tests_schema.json for app-local tests. Modify documentation and YAML files to reflect new schema path and ensure consistency across the project.

* Remove generated Flutter environment files to clean up the project structure.

* Enhance YAML test app patcher to validate test directory configuration and presence of test files. Add error handling for missing configuration and implement tests to verify directory resolution and test file detection.

* Implement error handling for missing declarative tests in CLI and enhance output extraction for known failures. Update tests to verify new functionality.

* Remove additional generated Flutter environment files to further clean up the project structure.

* Enhance ensemble test runner CLI with validation and reporting features. Add `--doctor` command for setup validation, implement JSON reporting options, and improve error messages for widget visibility and API calls. Update documentation with usage examples and clarify test directory requirements.

* Enhance ensemble test runner with new features for test authoring and validation. Add commands for app inspection, test scaffolding, and improved reporting options including JUnit format. Update README and documentation to reflect new functionalities and usage examples. Modify schema to include additional metadata fields for tests.

* Update ensemble dependency to version v1.2.46 in pubspec.yaml

* Implement timeout feature for ensemble test runner CLI. Add `--timeout` argument to specify test suite duration, with default set to 10 minutes. Update README with usage examples and modify CLI output handling to accommodate timeout settings.

* Add agent debug logging functionality to ensemble test runner

Introduce a new `agentDebugLog` function for logging debug information during test execution. This includes logging key events such as test entry, execution plan readiness, configuration building, and widget pumping. Update relevant files to integrate logging at various stages of the test harness and runner processes, enhancing traceability and debugging capabilities.

* Enhance local definition provider with detailed agent logging

Add comprehensive logging throughout the LocalDefinitionProvider class to track the loading of app configurations, themes, widgets, scripts, and actions. This includes logging before and after significant operations to improve traceability and debugging capabilities. Update the ensemble test harness to log the state before and after updating the app bundle, ensuring better visibility during test execution.

* Add agent debug logging to CLI test execution

Integrate additional logging within the `runEnsembleYamlTestsCli` function to capture key events during the CLI startup and before the Flutter test process. This enhancement improves traceability and debugging capabilities by logging the current working directory and arguments passed to the CLI.

* Add agent debug logging throughout assertion engine, entry point, and test reporter

Integrate detailed logging using the `agentDebugLog` function in the AssertionEngine, EnsembleTestEntry, and TestReporter classes. This enhancement captures key events and state information during test execution, improving traceability and debugging capabilities across the test runner's workflow.

* Refactor local provider and test runner by removing agent debug logging

Eliminate the `agentDebugLog` function calls from the LocalDefinitionProvider, AssertionEngine, and EnsembleTestRunner classes to streamline the codebase. This cleanup enhances readability and maintainability by removing unnecessary logging statements while retaining core functionality. Additionally, update the test harness to mock various method channels for improved testing capabilities.

* Update ensemble test harness to allow loading of app bundle through existing config path. This change preserves test-installed API providers and avoids unnecessary initialization of real providers, enhancing the test execution environment.

* Refactor LocalDefinitionProvider to use rootBundle for asset loading

Replace custom asset loading method with rootBundle.loadString for improved consistency and performance in loading local assets. This change enhances the readability of the code and aligns with best practices for asset management in Flutter applications.

* Refactor LocalDefinitionProvider to utilize a centralized asset loading method

Replace direct calls to rootBundle.loadString with a new _loadLocalAssetString method for improved consistency and maintainability in loading local assets. This change enhances code readability and aligns with best practices for asset management in Flutter applications.

* Add temporary storage path for tests in EnsembleTestHarness

Introduce a static variable for a temporary storage path in the EnsembleTestHarness class, enhancing the test environment by directing certain method channel calls to this path instead of the current directory. This change improves test isolation and management of temporary files during test execution.

Author: sharjeelyunus

.gitignore

@@ -11,6 +11,9 @@ starter/ios/Runner.xcodeproj/project.pbxproj
 starter/node_modules
 starter/package-lock.json
 
+# IntelliJ / Android Studio module files (generated by Melos / IDE)
+*.iml
+
 # FVM Version Cache
 .fvm/
 
@@ -31,6 +34,14 @@ starter/package-lock.json
 starter/android/app/src/main/java/io/
 starter/ios/Runner/GeneratedPluginRegistrant.h
 starter/ios/Runner/GeneratedPluginRegistrant.m
+starter/macos/Flutter/ephemeral/Flutter-Generated.xcconfig
+starter/macos/Flutter/ephemeral/flutter_export_environment.sh
+
+# GetStorage local persistence (runtime-generated when running starter)
+starter/GetStorage.gs
+starter/GetStorage.bak
+starter/system.gs
+starter/system.bak
 
 # remove molos .iml files
-**/melos_*.iml
+**/melos_*.iml

melos.yaml

@@ -15,6 +15,21 @@ ignore:
   - modules/ensemble_ts_interpreter
   - "**/build/**"
 
+scripts:
+  ensemble:test:
+    description: Run declarative Ensemble YAML tests in starter.
+    run: cd starter && dart run ensemble_test_runner:ensemble_test
+
+  ensemble:generate-test-schema:
+    description: Regenerate ensemble_tests_schema.json from the step vocabulary.
+    run: |
+      cd packages/ensemble_test_runner && dart run tool/generate_schema.dart
+
+  ensemble:generate-step-registry:
+    description: Regenerate test_step_registry.dart from tool/generate_step_registry.dart.
+    run: |
+      cd packages/ensemble_test_runner && dart run tool/generate_step_registry.dart
+
 command:
   version:
     updateGitTagRefs: true

modules/ensemble/lib/ensemble.dart

@@ -98,6 +98,11 @@ class Ensemble extends WithEnsemble with EnsembleRouteObserver {
   late FirebaseApp ensembleFirebaseApp;
   static final Map<String, dynamic> externalDataContext = {};
 
+  /// Clears [initManagers] singleton state between Flutter widget tests.
+  static void resetInitManagersForTest() {
+    _instance._completer = null;
+  }
+
   /// initialize all the singleton/managers. Note that this function can be
   /// called multiple times since it's being called inside a widget.
   /// The actual code block to initialize the managers is guaranteed to run

modules/ensemble/lib/ensemble_app.dart

@@ -159,6 +159,7 @@ class EnsembleAppState extends State<EnsembleApp> with WidgetsBindingObserver {
   bool _hasInternet = true;
   late final StreamSubscription<List<ConnectivityResult>>
       _connectivitySubscription;
+  SemanticsHandle? _testSemanticsHandle;
 
   @override
   void initState() {
@@ -195,7 +196,7 @@ class EnsembleAppState extends State<EnsembleApp> with WidgetsBindingObserver {
       }
     });
     if (EnvConfig().isTestMode) {
-      SemanticsBinding.instance.ensureSemantics();
+      _testSemanticsHandle = SemanticsBinding.instance.ensureSemantics();
     }
   }
 
@@ -245,6 +246,7 @@ class EnsembleAppState extends State<EnsembleApp> with WidgetsBindingObserver {
   @override
   void dispose() {
     _connectivitySubscription.cancel();
+    _testSemanticsHandle?.dispose();
     WidgetsBinding.instance.removeObserver(this);
     super.dispose();
   }

modules/ensemble/lib/framework/definition_providers/local_provider.dart

@@ -1,4 +1,3 @@
-import 'dart:io';
 import 'dart:ui';
 
 import 'package:ensemble/action/action_scope_util.dart';
@@ -13,7 +12,6 @@ import 'package:flutter_i18n/flutter_i18n.dart';
 import 'package:flutter_i18n/loaders/decoders/yaml_decode_strategy.dart';
 import 'package:yaml/yaml.dart';
 import 'dart:convert';
-import 'package:flutter/foundation.dart' as foundation;
 
 /**
  * Store all the definitions and assets locally together with the App
@@ -43,8 +41,7 @@ class LocalDefinitionProvider extends FileDefinitionProvider {
     }
     // Note: Web with local definition caches even if we disable browser cache
     // so you may need to re-run the app on definition changes
-    var pageStr =
-        await rootBundle.loadString('${path}screens/$screen.yaml');
+    var pageStr = await _loadLocalAssetString('${path}screens/$screen.yaml');
     if (pageStr.isEmpty) {
       return ScreenDefinition(YamlMap());
     }
@@ -55,7 +52,7 @@ class LocalDefinitionProvider extends FileDefinitionProvider {
   Future<AppBundle> getAppBundle({bool? bypassCache = false}) async {
     try {
       final configString =
-          await rootBundle.loadString('${path}config/appConfig.json');
+          await _loadLocalAssetString('${path}config/appConfig.json');
       final Map<String, dynamic> appConfigMap = json.decode(configString);
       if (appConfigMap.isNotEmpty) {
         appConfig = UserAppConfig(
@@ -75,7 +72,7 @@ class LocalDefinitionProvider extends FileDefinitionProvider {
 
   Future<YamlMap?> _readFile(String file) async {
     try {
-      var value = await rootBundle.loadString(path + file);
+      var value = await _loadLocalAssetString(path + file);
       return loadYaml(value);
     } catch (error) {
       // ignore error
@@ -92,7 +89,7 @@ class LocalDefinitionProvider extends FileDefinitionProvider {
     try {
       // Get the manifest content
       final manifestContent =
-          await rootBundle.loadString(path + '.manifest.json');
+          await _loadLocalAssetString(path + '.manifest.json');
       final Map<String, dynamic> manifestMap = json.decode(manifestContent);
 
       // Process App Widgets
@@ -130,8 +127,8 @@ class LocalDefinitionProvider extends FileDefinitionProvider {
           for (var script in scriptsList) {
             try {
               // Load the script content in string
-              final scriptContent = await rootBundle
-                  .loadString("${path}scripts/${script["name"]}.js");
+              final scriptContent = await _loadLocalAssetString(
+                  "${path}scripts/${script["name"]}.js");
               code[script["name"]] = scriptContent;
             } catch (e) {
               // ignore error
@@ -209,3 +206,8 @@ class LocalDefinitionProvider extends FileDefinitionProvider {
     return this;
   }
 }
+
+Future<String> _loadLocalAssetString(String key) async {
+  final data = await rootBundle.load(key);
+  return utf8.decode(data.buffer.asUint8List());
+}

modules/ensemble/lib/framework/definition_providers/screen_selector_security.dart

@@ -1,13 +1,10 @@
-import 'package:meta/meta.dart';
-
 /// Returns true when [screen] is a single safe path segment for resolving
 /// screen YAML from a remote base URL or from bundled assets under
 /// `.../screens/<screen>.yaml`.
 ///
 /// Rejects empty values, `..`, path separators, percent-encoded separators, and
 /// ASCII control characters so untrusted inputs cannot traverse outside the
 /// intended screens directory.
-@visibleForTesting
 bool isSafeRemoteScreenSelector(String screen) {
   if (screen.isEmpty || screen.length > 256) {
     return false;