feat(variables): persist scripted variable changes by default + re-enable disabled scripting APIs (usebruno#8315)

* feat(variables): add variable persistence with scripting

feat(collections): implement script-driven update for collection variables, ensuring direct root modification and draft synchronization

feat(collections): enhance script variable management with baseline tracking and draft preservation

* feat(variables): add runtime variable updates and optimize disk writes by implementing dirty flags

fix(collections): handle errors during environment persistence in script execution

feat(collections): implement baseline clearing for script execution and optimize variable update handling

feat(tests): add default persistence tests for environment variables and update runtime variable handling

refactor(collections): streamline variable update handling and improve draft management by removing redundant comments and optimizing code clarity

test(collection-vars): add verification for draft edits and script variable visibility in collection settings UI

refactor(collection-vars): update header value selection logic for improved clarity and accuracy in draft isolation tests

* feat(global-environments): enhance global environment updates to resolve stale active UIDs and improve persistence logic

- Updated the `updateGlobalEnvironments` reducer to handle stale active UIDs by matching against environment names.
- Improved the logic for setting global environments and active UIDs to ensure consistency after disk reloads.
- Removed outdated tests related to persisted values in favor of more relevant assertions for environment variable handling.

* feat(variables): enhance typed value handling and persistence in global and collection environments

- Added tests to infer data types (number, boolean, object) when setting environment and collection variables.
- Updated the logic to preserve existing data types when variables are not modified by scripts.
- Implemented dirty flags to track changes in typed variables, ensuring accurate persistence across sessions.
- Refactored related tests to verify the correct behavior of typed variables in various scenarios.

* refactor(variables): streamline data type inference and enhance deletion methods

- Removed redundant data type inference logic from global and collection variable updates to simplify the codebase.
- Updated deletion methods in the Bru class to use Object.keys for improved resilience against user-defined properties.
- Added tests to ensure deletion methods function correctly even when properties are shadowed.
- Enhanced clarity in draft merge tests by standardizing keyboard shortcuts for selecting all text.

* fix(tests): correct variable naming and improve environment panel interactions

- Updated test cases to reflect the correct variable name 'wasSaved' instead of 'was-saved'.
- Modified environment panel interaction to remove forced click, enhancing test reliability.
- Added a utility function to close the environment panel in safe mode tests for better readability and maintainability.

* feat(runtime): enhance variable management and cleanup logic

- Introduced a new method to clear script-driven variable baselines for collections, ensuring no stale data leaks into new requests.
- Updated the handling of runtime variables in the Bru class to track changes with a new dirty flag, improving state management.
- Refactored the application of script environment variables to prevent direct mutations, ensuring immutability and cleaner state updates.
- Enhanced the response handling in the script runtime to conditionally include runtime variables based on their dirty state.

* feat(variables): improve request handling and state management for collections and environments

- Enhanced event listeners to clear global environment baselines on both 'testrun-started' and 'request-queued' events, preventing stale data issues.
- Updated global environment and collection variable update events to ignore stale updates from superseded requests, ensuring accurate state management.
- Refactored the Bru class to optimize variable management, including checks for existing keys before updates and deletions, improving performance and reliability.
- Introduced request UID tracking to maintain consistency across variable updates during concurrent requests.

* refactor(collections): update action to clear script variable baselines

- Replaced the dispatch of `_clearScriptGlobalEnvBaseline` with `clearScriptVariableBaselines` to improve clarity and maintainability in the Redux action handling for collections.

* feat(environments): introduce getScriptModifiedKeys utility for improved variable management

- Added a new utility function, `getScriptModifiedKeys`, to identify keys modified by scripts relative to a baseline, enhancing the handling of data types during variable updates.
- Updated the application of script environment variables to prevent overwriting user-defined draft changes during no-op writes.
- Refactored related logic in collections and global environments to utilize the new utility, ensuring accurate state management and improved clarity in the Redux slices.

* refactor(global-environments): simplify active UID resolution logic in updateGlobalEnvironments reducer

- Streamlined the logic for resolving the active global environment UID by consolidating conditions into a more concise format.
- Removed outdated comments to enhance code clarity and maintainability.
- Updated tests to ensure accurate resolution of active UIDs based on incoming environment data.

* refactor(tests): remove outdated comments and streamline environment variable row expectations

- Eliminated comments related to state sync and inference issues to enhance code clarity.
- Adjusted expectations for environment variable row rendering in tests, focusing on relevant assertions.

* feat(tests): add comprehensive tests for secret variable persistence in environments

- Introduced new test cases to validate the preservation of secret variables when updated via scripts in both collection and global environments.
- Implemented tests to ensure that secret values are encrypted before storage and can be correctly decrypted for subsequent requests.
- Added fixtures and environment configurations for testing secret variable behavior in both bru and yml formats.
- Enhanced utility functions for managing environment configurations and interactions within the test suite.

* feat(tests): enhance environment variable tests and add global variable persistence

- Updated MultiLineEditor and SingleLineEditor components to include data-testid for secret reveal toggle buttons, improving testability.
- Introduced new tests for global environment variable persistence, ensuring non-secret variables survive app restarts and are correctly interpolated.
- Added fixtures for workspace and collections to support the new global variable tests, enhancing the overall test coverage for environment management.
- Refactored utility functions to streamline interactions with environment variables in tests.

* refactor(collections): optimize environment and collection saving logic

- Simplified the persistence logic for active environments by directly constructing the environment copy, reducing unnecessary cloning.
- Updated the collection saving process to utilize the fresh collection state, ensuring accurate data is saved without drafts.
- Enhanced error handling during the save operations to improve reliability and maintainability.

* feat(tests): implement collection variable persistence tests

- Added multiple test cases to validate the persistence of collection variables across app restarts, including typed values and multiple variable settings.
- Created new fixtures for collection variables to support the tests, ensuring accurate simulation of variable management scenarios.
- Enhanced the existing collection management logic to ensure that variables are correctly set and deleted as per the test requirements.

* feat(tests): add tests for typed global environment variable persistence

- Introduced a new test suite to validate the persistence of typed global environment variables across app restarts, ensuring correct data types are maintained.
- Created a fixture for the test collection to simulate setting global variables with various data types, including number, boolean, object, and string.
- Enhanced the test logic to verify that the environment file reflects the correct state before and after application restarts.

* fix(tests): update request tab close interaction in variable persistence tests

* fix(tests): improve hover interaction for collection actions in runner tests

- Updated the hover logic for revealing collection actions to handle sidebar re-renders more reliably.
- Replaced one-shot hover with a polling mechanism to ensure visibility of actions, enhancing test stability.

* refactor(environments): streamline environment variable handling and remove ephemeral metadata logic

- Simplified the comparison logic for environment variables by removing unnecessary ephemeral metadata handling.
- Updated the saving process to directly use the environment variables without stripping metadata, enhancing clarity and maintainability.
- Removed outdated comments and unused utility functions related to ephemeral variables, improving code cleanliness.

* fix(ipc): update persistActiveEnvironment to handle requestUid for stale updates

- Modified the persistActiveEnvironment function to accept a requestUid parameter, allowing for better management of stale updates.
- Enhanced the logic to prevent disk writes for superseded requests, improving data integrity during environment persistence.

* refactor(bru): remove unused envName variable in deleteAllEnvVars method

- Eliminated the envName variable from the deleteAllEnvVars method, simplifying the logic for deleting environment variables.
- Cleaned up the method by removing unnecessary checks related to the envName, enhancing code clarity and maintainability.

* fix(bru): prevent deletion of internal __name__ variable in deleteEnvVar method

- Added a check in the deleteEnvVar method to silently ignore attempts to delete the internal __name__ variable, preserving its integrity.
- Updated tests to verify that the __name__ variable remains unchanged when deleteEnvVar is called with this key.
- Enhanced runtime tests to ensure compatibility with QuickJS by confirming that environment variables set with persist options are handled correctly.

* feat(tests): add legacy support test for environment variable persistence

- Introduced a new test suite to validate that the legacy argument for setting environment variables with persistence is still functional in version 4.
- Created a fixture to simulate the legacy syntax, ensuring that the variable is correctly persisted on disk without errors.
- Enhanced integration testing to confirm that the legacy behavior aligns with the current implementation, maintaining backward compatibility.

* test(tests): enhance legacy environment variable persistence tests for safe and developer modes

- Updated the test suite for `bru.setEnvVar` to verify that the legacy persist flag is correctly handled in both safe and developer modes.
- Introduced a helper function to streamline the verification process and ensure consistent behavior across different execution contexts.
- Adjusted the test logic to reset the environment state between mode switches, maintaining test integrity.
- Improved hover interaction in multiple persistent variable tests to ensure reliable visibility of actions during execution.

* fix(EnvironmentVariablesTable): correct change detection logic for environment variables

- Updated the logic for determining changes in environment variables to compare active current and saved values instead of previously used variablesToSave and savedValues.
- This change ensures accurate detection of modifications before saving, improving user feedback when no changes are present.

* test(tests): enhance secret variable persistence tests for environment configurations

- Updated the test suites for `bru.setEnvVar` and `bru.setGlobalEnvVar` to include interactions with the secrets tab, ensuring visibility of secret variables during various states of the environment.
- Added checks to confirm that the eye toggle functionality correctly reveals the values of secret variables after setting and overwriting them.
- Improved test coverage for secret variable persistence, validating that the expected values are displayed in both collection and global environment contexts.

Author: sanish-bruno

packages/bruno-app/src/components/EnvironmentVariablesTable/index.js

@@ -445,18 +445,7 @@ const EnvironmentVariablesTable = ({
     const otherCurrent = namedValues.filter((variable) => !belongsToActiveTab(variable));
     const otherSaved = savedValues.filter((variable) => !belongsToActiveTab(variable));
 
-    // Compare against what's on disk: for an ephemeral overlay, that's
-    // `persistedValue`, not the scripted value Redux is holding.
-    const baselineForCompare = (v) => {
-      const stripped = stripEnvVarUid(v);
-      if (v?.ephemeral && v?.persistedValue !== undefined) {
-        stripped.value = v.persistedValue;
-      }
-      return stripped;
-    };
-    // Compare without UIDs; only the active tab's subset decides if there's anything to save.
-    const hasChanges
-      = JSON.stringify(activeCurrent.map(stripEnvVarUid)) !== JSON.stringify(activeSaved.map(baselineForCompare));
+    const hasChanges = JSON.stringify(activeCurrent.map(stripEnvVarUid)) !== JSON.stringify(activeSaved.map(stripEnvVarUid));
     if (!hasChanges) {
       toast.error('No changes to save');
       return;
@@ -759,11 +748,6 @@ const EnvironmentVariablesTable = ({
                       isSecret={variable.secret}
                       onChange={(newValue) => {
                         formik.setFieldValue(`${actualIndex}.value`, newValue, true);
-                        // Clear ephemeral metadata when user manually edits the value
-                        if (variable.ephemeral) {
-                          formik.setFieldValue(`${actualIndex}.ephemeral`, undefined, false);
-                          formik.setFieldValue(`${actualIndex}.persistedValue`, undefined, false);
-                        }
                         // Append a new empty row when editing value on the last row
                         if (isLastRow) {
                           setTimeout(() => {

packages/bruno-app/src/components/Environments/ConfirmCloseEnvironment/index.js

@@ -33,15 +33,15 @@ const ConfirmCloseEnvironment = ({ onCancel, onCloseWithoutSave, onSaveAndClose,
 
         <div className="flex justify-between mt-6">
           <div>
-            <Button color="danger" onClick={onCloseWithoutSave}>
+            <Button color="danger" onClick={onCloseWithoutSave} data-testid="env-unsaved-close-without-save">
               Don't Save
             </Button>
           </div>
           <div className="flex gap-2">
-            <Button size="sm" color="secondary" variant="ghost" onClick={onCancel}>
+            <Button size="sm" color="secondary" variant="ghost" onClick={onCancel} data-testid="env-unsaved-cancel">
               Cancel
             </Button>
-            <Button onClick={onSaveAndClose}>
+            <Button onClick={onSaveAndClose} data-testid="env-unsaved-save-and-close">
               Save
             </Button>
           </div>

packages/bruno-app/src/components/MultiLineEditor/index.js

@@ -219,7 +219,7 @@ class MultiLineEditor extends Component {
    */
   secretEye = (isSecret) => {
     return isSecret === true ? (
-      <button className="mx-2" onClick={() => this.toggleVisibleSecret()}>
+      <button className="mx-2" data-testid="secret-reveal-toggle" onClick={() => this.toggleVisibleSecret()}>
         {this.state.maskInput === true ? (
           <IconEyeOff size={18} strokeWidth={2} />
         ) : (

packages/bruno-app/src/components/SingleLineEditor/index.js

@@ -302,7 +302,7 @@ class SingleLineEditor extends Component {
    */
   secretEye = (isSecret) => {
     return isSecret === true ? (
-      <button type="button" className="mx-2" onClick={() => this.toggleVisibleSecret()}>
+      <button type="button" className="mx-2" data-testid="secret-reveal-toggle" onClick={() => this.toggleVisibleSecret()}>
         {this.state.maskInput === true ? (
           <IconEyeOff size={18} strokeWidth={2} />
         ) : (

packages/bruno-app/src/providers/App/useIpcEvents.js

@@ -22,10 +22,11 @@ import {
   runFolderEvent,
   runRequestEvent,
   scriptEnvironmentUpdateEvent,
+  runtimeVariablesUpdateEvent,
   streamDataReceived,
   setDotEnvVariables
 } from 'providers/ReduxStore/slices/collections';
-import { collectionAddEnvFileEvent, openCollectionEvent, hydrateCollectionWithUiStateSnapshot, mergeAndPersistEnvironment } from 'providers/ReduxStore/slices/collections/actions';
+import { collectionAddEnvFileEvent, openCollectionEvent, hydrateCollectionWithUiStateSnapshot, persistActiveEnvironment, collectionVariablesUpdateEvent } from 'providers/ReduxStore/slices/collections/actions';
 import {
   workspaceOpenedEvent,
   workspaceConfigUpdatedEvent,
@@ -35,7 +36,7 @@ import { workspaceDotEnvUpdateEvent, setWorkspaceDotEnvVariables } from 'provide
 import toast from 'react-hot-toast';
 import { useDispatch, useStore } from 'react-redux';
 import { isElectron } from 'utils/common/platform';
-import { globalEnvironmentsUpdateEvent, updateGlobalEnvironments } from 'providers/ReduxStore/slices/global-environments';
+import { globalEnvironmentsUpdateEvent, updateGlobalEnvironments, _clearScriptGlobalEnvBaseline } from 'providers/ReduxStore/slices/global-environments';
 import { collectionAddOauth2CredentialsByUrl, collectionClearOauth2CredentialsByCredentialsId, updateCollectionLoadingState, collectionLoadedFromTree } from 'providers/ReduxStore/slices/collections/index';
 import { addLog } from 'providers/ReduxStore/slices/logs';
 import { loadNotifications } from 'providers/ReduxStore/slices/notifications';
@@ -201,23 +202,36 @@ const useIpcEvents = () => {
       }
     });
 
-    const removeScriptEnvUpdateListener = ipcRenderer.on('main:script-environment-update', (val) => {
-      dispatch(scriptEnvironmentUpdateEvent(val));
+    const removeRuntimeVariablesUpdateListener = ipcRenderer.on('main:runtime-variables-update', (val) => {
+      dispatch(runtimeVariablesUpdateEvent(val));
     });
 
-    const removePersistentEnvVariablesUpdateListener = ipcRenderer.on('main:persistent-env-variables-update', (val) => {
-      dispatch(mergeAndPersistEnvironment(val));
+    const removeScriptEnvUpdateListener = ipcRenderer.on('main:script-environment-update', (val) => {
+      dispatch(scriptEnvironmentUpdateEvent(val));
+      if (val.collectionUid) {
+        dispatch(persistActiveEnvironment(val.collectionUid, val.requestUid));
+      }
     });
 
     const removeGlobalEnvironmentVariablesUpdateListener = ipcRenderer.on('main:global-environment-variables-update', (val) => {
       dispatch(globalEnvironmentsUpdateEvent(val));
     });
 
+    const removeCollectionVariablesUpdateListener = ipcRenderer.on('main:collection-variables-update', (val) => {
+      dispatch(collectionVariablesUpdateEvent(val));
+    });
+
     const removeCollectionRenamedListener = ipcRenderer.on('main:collection-renamed', (val) => {
       dispatch(collectionRenamedEvent(val));
     });
 
     const removeRunFolderEventListener = ipcRenderer.on('main:run-folder-event', (val) => {
+      // Folder runs reuse the workspace baseline across N requests; clear it
+      // per request so request N's global-env update doesn't diff against
+      // request N-1's pre-flush snapshot.
+      if (val.type === 'testrun-started' || val.type === 'request-queued') {
+        dispatch(_clearScriptGlobalEnvBaseline());
+      }
       dispatch(runFolderEvent(val));
     });
 
@@ -393,7 +407,8 @@ const useIpcEvents = () => {
       removeHttpStreamNewDataListener();
       removeHttpStreamEndListener();
       removeCollectionLoadingStateListener();
-      removePersistentEnvVariablesUpdateListener();
+      removeCollectionVariablesUpdateListener();
+      removeRuntimeVariablesUpdateListener();
       removeSystemResourcesListener();
       gitVersionListener();
       removeLoadNotificationsListener();