Merge branch 'data-integrations:develop' into GoogleBigQSource

Author: Praveena2607

docs/BigQueryExecute-action.md

@@ -62,6 +62,11 @@ cache that will be flushed whenever tables in the query are modified.
 It is only applicable when users choose to store the query results in a BigQuery table.
 More information can be found [here](https://cloud.google.com/data-fusion/docs/how-to/customer-managed-encryption-keys)
 
+**Destination Table Write Preference**: Provides the following options as write preferences for the destination table:
+* **Write if Empty**: Only write if the table is empty
+* **Append to Table**: Add results to existing data. Schema should match.
+* **Overwrite Table**: Replace all existing data. Schema will also be overriden.
+
 **Row As Arguments**: Row as arguments. For example, if the query is 'select min(id) as min_id, max(id) as max_id from my_dataset.my_table',
 an arguments for 'min_id' and 'max_id' will be set based on the query results. Plugins further down the pipeline can then
 reference these values with macros ${min_id} and ${max_id}.

src/e2e-test/features/bigquery/sink/GCSToBigQuery_WithMacro.feature

@@ -56,7 +56,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
     Then Close the pipeline logs
     Then Close the preview
     Then Deploy the pipeline
-    Then Run the Pipeline in Runtime
+    Then Click on the Runtime Arguments Dropdown button
     Then Enter runtime argument value "projectId" for key "gcsProjectId"
     Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
     Then Enter runtime argument value "serviceAccount" for key "serviceAccount"

src/e2e-test/features/bigquery/source/BigQueryToGCS_WithMacro.feature

@@ -51,7 +51,7 @@ Feature: BigQuery source - Verification of BigQuery to GCS successful data trans
     Then Click on preview data for GCS sink
     Then Close the preview data
     Then Deploy the pipeline
-    Then Run the Pipeline in Runtime
+    Then Click on the Runtime Arguments Dropdown button
     Then Enter runtime argument value "projectId" for key "bqProjectId"
     Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
     Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"

src/e2e-test/features/bigqueryexecute/BQExecute_WithMacro.feature

@@ -19,7 +19,7 @@ Feature: BigQueryExecute - Verify data transfer using BigQuery Execute plugin wi
     Then Validate "BigQuery Execute" plugin properties
     Then Close the Plugin Properties page
     Then Save and Deploy Pipeline
-    Then Run the Pipeline in Runtime
+    Then Click on the Runtime Arguments Dropdown button
     Then Enter runtime argument value "projectId" for key "bqeProjectId"
     Then Enter runtime argument value "bqExecuteQuery" for key "bqeSQl"
     Then Enter runtime argument value "projectId" for key "bqeDatasetProjectId"

src/e2e-test/features/bigtable/BigTableToBigTable.feature

@@ -42,7 +42,7 @@ Feature: BigTable source - Verification of BigTable to BigTable Successful Data
     Then Close the Plugin Properties page
     Then Save the pipeline
     Then Deploy the pipeline
-    Then Run the Pipeline in Runtime
+    Then Click on the Runtime Arguments Dropdown button
     Then Enter runtime argument value "cbtSourceOutputSchema" for key "macroKeyString"
     Then Run the Pipeline in Runtime with runtime arguments
     Then Wait till pipeline is in running state
@@ -79,7 +79,7 @@ Feature: BigTable source - Verification of BigTable to BigTable Successful Data
     Then Close the Plugin Properties page
     Then Save the pipeline
     Then Deploy the pipeline
-    Then Run the Pipeline in Runtime
+    Then Click on the Runtime Arguments Dropdown button
     Then Enter runtime argument value "cbtSourceOutputSchema" for key "macroKeyString"
     Then Run the Pipeline in Runtime with runtime arguments
     Then Wait till pipeline is in running state
@@ -115,7 +115,7 @@ Feature: BigTable source - Verification of BigTable to BigTable Successful Data
     Then Close the Plugin Properties page
     Then Save the pipeline
     Then Deploy the pipeline
-    Then Run the Pipeline in Runtime
+    Then Click on the Runtime Arguments Dropdown button
     Then Enter runtime argument value "cbtSourceOutputSchema" for key "macroKeyString"
     Then Run the Pipeline in Runtime with runtime arguments
     Then Wait till pipeline is in running state

src/e2e-test/features/gcs/source/GCSSourceError.feature

@@ -11,6 +11,7 @@ Feature: GCS source - Verify GCS Source plugin error scenarios
       | property        |
       | path            |
       | format          |
+      | referenceName   |
 
   Scenario: To verify Error message for invalid bucket name
     Given Open Datafusion Project to configure pipeline

src/e2e-test/features/gcs/source/GCSSourceToBigQuery.feature

@@ -29,7 +29,7 @@ Feature:GCSCopy - Verification of successful objects copy from one bucket to ano
     Then Validate "GCS Copy" plugin properties
     Then Close the Plugin Properties page
     Then Save and Deploy Pipeline
-    Then Run the Pipeline in Runtime
+    Then Click on the Runtime Arguments Dropdown button
     Then Enter runtime argument value "projectId" for key "projectId"
     Then Enter runtime argument value "gcsCsvFile" for GCSCopy property sourcePath key "SourcePath"
     Then Enter runtime argument value for GCSCopy property destination path key "DestPath"

src/e2e-test/features/gcscopy/GCSCopy_WithMacro.feature

@@ -24,29 +24,26 @@ Feature: GCSCreate - Verification of GCS Create plugin
     Then Verify that the object "gcsCreateObject2" created successfully
 
   @GCS_CSV_TEST @GCSCreate_Required
-  Scenario: Verify the pipeline with GCSCreate should fail on second run when fail if objects exists is true
+  Scenario: Verify that pipeline with GCSCreate failed when fail if objects exists property set to true
     Given Open Datafusion Project to configure pipeline
     When Expand Plugin group in the LHS plugins list: "Conditions and Actions"
     When Select plugin: "GCS Create" from the plugins list as: "Conditions and Actions"
     When Navigate to the properties page of plugin: "GCS Create"
     Then Enter the GCS Create property projectId "projectId"
-    Then Enter the GCS Create property objects to create as path "gcsCreateObject1"
+    Then Enter the GCS Create property objects to create as path "gcsCsvFile"
     Then Select GCS Create property fail if objects exists as "true"
     Then Override Service account details if set in environment variables
     Then Validate "GCS Create" plugin properties
     Then Close the GCS Create properties
     Then Save and Deploy Pipeline
     Then Run the Pipeline in Runtime
     Then Wait till pipeline is in running state
-    Then Verify the pipeline status is "Succeeded"
-    Then Run the Pipeline in Runtime
-    Then Wait for pipeline to be in status: "Failed" with a timeout of 240 seconds
     Then Open and capture logs
     Then Verify the pipeline status is "Failed"
     Then Close the pipeline logs
 
   @GCS_CSV_TEST @GCSCreate_Required
-  Scenario: Verify the pipeline with GCSCreate should not fail on second run when fail if objects exists is false
+  Scenario: Verify that pipeline with GCSCreate succeeded when fail if objects exists property set to false
     Given Open Datafusion Project to configure pipeline
     When Expand Plugin group in the LHS plugins list: "Conditions and Actions"
     When Select plugin: "GCS Create" from the plugins list as: "Conditions and Actions"
@@ -60,9 +57,6 @@ Feature: GCSCreate - Verification of GCS Create plugin
     Then Save and Deploy Pipeline
     Then Run the Pipeline in Runtime
     Then Wait till pipeline is in running state
-    Then Verify the pipeline status is "Succeeded"
-    Then Run the Pipeline in Runtime
-    Then Wait for pipeline to be in status: "Succeeded" with a timeout of 240 seconds
     Then Open and capture logs
     Then Verify the pipeline status is "Succeeded"
     Then Close the pipeline logs

src/e2e-test/features/gcscreate/GCSCreate.feature

@@ -17,7 +17,7 @@ Feature: GCSCreate - Verification of GCS Create plugin with macro arguments
     Then Validate "GCS Create" plugin properties
     Then Close the GCS Create properties
     Then Save and Deploy Pipeline
-    Then Run the Pipeline in Runtime
+    Then Click on the Runtime Arguments Dropdown button
     Then Enter runtime argument value "projectId" for key "gcsCreateProjectId"
     Then Enter runtime argument value "gcsCreateObject1" for GCSCreate property key "gcsCreateObjectsToCreate"
     Then Enter runtime argument value "true" for GCSCreate property key "gcsCreateFailIfObjectExists"