Add all missing options to the Import/Export Data functionality, and update the syntax of the COPY command to align with the latest standards. #8583

Updated documentation.

Author: akshay-joshi

docs/en_US/images/import_export_columns.png

@@ -33,18 +33,29 @@ Use the fields in the *General* tab to specify import and export preferences:
 * Use the drop-down listbox in the *Encoding* field to specify the type of
   character encoding.
 
+* use the drop-down listbox in the *On Error* field to specify how to behave
+  when encountering an error converting a columns input value into its data type.
+  An error_action value of stop means fail the command, while ignore means discard
+  the input row and continue with the next one. The default is stop. This option is
+  available from PG/EPAS version 17 and above.
+
+* use the drop-down listbox in the *Log Verbosity* field to specify the amount
+  of messages emitted by a COPY command: default or verbose. This is currently
+  used in Import only when ON_ERROR option is set to ignore. This option is
+  available from PG/EPAS version 17 and above.
+
 .. image:: images/import_export_options.png
     :alt: Import Export data dialog options tab
     :align: center
 
 * Use the fields in the *Options* tab to specify additional information:
 
-   * Move the *OID* switch to the *Yes* position to include the *OID* column.
-     The *OID* is a system-assigned value that may not be modified. The default
-     is *No*.
    * Move the *Header* switch to the *Yes* position to include the table header
      with the data rows. If you include the table header, the first row of the
      file will contain the column names.
+   * Move the *Freeze* switch to the *Yes* position to requests copying the
+     data with rows already frozen, just as they would be after running the
+     VACUUM FREEZE command.
    * If you are exporting data, specify the delimiter that will separate the
      columns within the target file in the *Delimiter* field. The separating
      character can be a colon, semicolon, a vertical bar, or a tab.
@@ -54,8 +65,11 @@ Use the fields in the *General* tab to specify import and export preferences:
      be a single quote or a double quote.
    * Specify a character that should appear before a data character that matches
      the *QUOTE* value in the *Escape* field.
-   * Use the *NULL Strings* field to specify a string that will represent a null
+   * Use the *NULL String* field to specify a string that will represent a null
      value within the source or target file.
+   * Use the *Default String* field to specify a string that will represent a default value.
+     Each time the string is found in the input file, the default value of the corresponding
+     column will be used. This option is available from PG/EPAS version 16 and above.
 
 Click the *Columns* tab to continue.
 
@@ -71,29 +85,39 @@ or exported:
   the left of the column name. Click an empty spot inside the field to access
   the drop-down list.
 
+* If enabled, click inside the *Force Quote columns* field to forces quoting
+  to be used for all non-NULL values in each specified column. NULL output is
+  never quoted. To delete a column, click the *x* to the left of the column name.
+
 * If enabled, click inside the *NOT NULL columns* field to select one or more
   columns that will not be checked for a NULL value. To delete a column, click
   the *x* to the left of the column name.
 
+* If enabled, click inside the *NULL columns* field to match the specified columns
+  values against the null string, even if it has been quoted, and if a match is
+  found set the value to NULL. To delete a column, click the *x* to the left of the
+  column name.
+
 After completing the *Import/Export data* dialog, click the *OK* button to
-perform the import or export.  pgAdmin will inform you when the background
+perform the import or export. pgAdmin will notify you when the background
 process completes:
 
 .. image:: images/import_export_complete.png
     :alt: Import Export data completion notification
     :align: center
 
-Use the **Stop Process** button to stop the Import/Export process.
 
-Use the *Click here for details* link on the notification to open the *Process
+Use the *View Processes* button on the notification to open the *Process
 Watcher* and review detailed information about the execution of the command
 that performed the import or export:
 
+Use the **End Process** button to end the Import/Export process.
+
 .. image:: images/import_export_pw.png
     :alt: Import Export data process watcher
     :align: center
 
-.. note:: If you are running *pgAdmin* in *Server Mode* you can click on the |sm_icon| icon in the process watcher window to open the file location in the Storage Manager. You can use the :ref:`Storage Manager <storage_manager>` to download the backup file on the client machine .
+.. note:: If you are running *pgAdmin* in *Server Mode* you can click on the |sm_icon| icon in the process watcher window to open the file location in the Storage Manager. You can use the :ref:`Storage Manager <storage_manager>` to download the exported file on the client machine .
 
 
 .. |sm_icon| image:: images/sm_icon.png

docs/en_US/images/import_export_complete.png

@@ -42,7 +42,7 @@ Process Watcher
 
 The Process Watcher logs all the activity associated with the process/task and provides
 additional information for troubleshooting
-Use the **Stop Process** button to stop the Backup process.
+Use the **End Process** button to end the process.
 
 .. note:: If you are running *pgAdmin* in *Server Mode* you can click on the |sm_icon| icon in the process watcher window to open the file location in the Storage Manager. You can use the :ref:`Storage Manager <storage_manager>` to download the backup file on the client machine .
 

docs/en_US/images/import_export_general.png

@@ -12,6 +12,7 @@ notes for it.
    :maxdepth: 1
 
 
+   release_notes_9_4
    release_notes_9_3
    release_notes_9_2
    release_notes_9_1

docs/en_US/images/import_export_options.png

@@ -0,0 +1,31 @@
+***********
+Version 9.4
+***********
+
+Release date: 2025-05-29
+
+This release contains a number of bug fixes and new features since the release of pgAdmin 4 v9.3.
+
+Supported Database Servers
+**************************
+**PostgreSQL**: 13, 14, 15, 16 and 17
+
+**EDB Advanced Server**: 13, 14, 15, 16 and 17
+
+Bundled PostgreSQL Utilities
+****************************
+**psql**, **pg_dump**, **pg_dumpall**, **pg_restore**: 17.2
+
+
+New features
+************
+
+  | `Issue #8583 <https://github.com/pgadmin-org/pgadmin4/issues/8583>`_ -  Add all missing options to the Import/Export Data functionality, and update the syntax of the COPY command to align with the latest standards.
+
+Housekeeping
+************
+
+
+Bug fixes
+*********
+

docs/en_US/images/import_export_pw.png

@@ -23,11 +23,14 @@
 
 from config import PG_DEFAULT_DRIVER
 from pgadmin.model import Server
-from pgadmin.utils.constants import MIMETYPE_APP_JS, SERVER_NOT_FOUND
+from pgadmin.utils.constants import SERVER_NOT_FOUND
 from pgadmin.settings import get_setting, store_setting
 from pgadmin.tools.user_management.PgAdminPermissions import AllPermissionTypes
 
 MODULE_NAME = 'import_export'
+NOT_NULL_COLUMNS = 'not_null_columns'
+NULL_COLUMNS = 'null_columns'
+FORCE_QUOTE_COLUMNS = 'force_quote_columns'
 
 
 class ImportExportModule(PgAdminModule):
@@ -142,32 +145,11 @@ def index():
     return bad_request(errormsg=_("This URL cannot be called directly."))
 
 
-def _get_ignored_column_list(data, driver, conn):
-    """
-    Get list of ignored columns for import/export.
-    :param data: Data.
-    :param driver: PG Driver.
-    :param conn: Connection.
-    :return: return ignored column list.
-    """
-    icols = None
-
-    if data['icolumns']:
-        ignore_cols = data['icolumns']
-
-        # format the ignore column list required as per copy command
-        # requirement
-        if ignore_cols and len(ignore_cols) > 0:
-            icols = ", ".join([
-                driver.qtIdent(conn, col)
-                for col in ignore_cols])
-    return icols
-
-
-def _get_required_column_list(data, driver, conn):
+def _get_formatted_column_list(data, key, driver, conn):
     """
     Get list of required columns for import/export.
     :param data: Data.
+    :param key: Key.
     :param driver: PG Driver.
     :param conn: Connection.
     :return: return required column list.
@@ -176,8 +158,23 @@ def _get_required_column_list(data, driver, conn):
 
     # format the column import/export list required as per copy command
     # requirement
-    if data['columns']:
-        columns = data['columns']
+
+    # If key is FORCE_QUOTE_COLUMNS and total columns is equal to selected
+    # columns for force quote then return '*'
+    if (key in data and 'total_columns' in data and
+        key == FORCE_QUOTE_COLUMNS and
+            len(data[key]) == data['total_columns']):
+        cols = '*'
+    # if server version is >= 17 and key is either NULL_COLUMNS or
+    # NOT_NULL_COLUMNS and total columns is equal to selected columns then
+    # return '*'
+    elif (key in data and 'total_columns' in data and
+          conn.manager.version >= 170000 and
+          key in [NULL_COLUMNS, NOT_NULL_COLUMNS] and
+          len(data[key]) == data['total_columns']):
+        cols = '*'
+    elif key in data:
+        columns = data[key]
         if columns and len(columns) > 0:
             for col in columns:
                 if cols:
@@ -192,8 +189,9 @@ def _get_required_column_list(data, driver, conn):
 
 def _save_import_export_settings(settings):
     settings = {key: settings[key] for key in settings if key not in
-                ['icolumns', 'columns', 'database', 'schema', 'table',
-                 'save_btn_icon']}
+                ['columns', 'database', 'schema', 'table', 'save_btn_icon',
+                 'not_null_columns', 'null_columns', 'force_quote_columns',
+                 'total_columns']}
 
     if settings['is_import']:
         settings['import_file_name'] = settings['filename']
@@ -283,21 +281,24 @@ def create_import_export_job(sid):
     else:
         return bad_request(errormsg=_('Please specify a valid file'))
 
-    # Get required and ignored column list
-    icols = _get_ignored_column_list(data, driver, conn)
-    cols = _get_required_column_list(data, driver, conn)
+    # Get required and other columns list
+    cols = _get_formatted_column_list(data, 'columns', driver, conn)
+    not_null_cols = _get_formatted_column_list(data, NOT_NULL_COLUMNS,
+                                               driver, conn)
+    null_cols = _get_formatted_column_list(data, NULL_COLUMNS, driver,
+                                           conn)
+    quote_cols = _get_formatted_column_list(data, FORCE_QUOTE_COLUMNS,
+                                            driver, conn)
 
     # Save the settings
     _save_import_export_settings(new_settings)
 
     # Create the COPY FROM/TO  from template
-    query = render_template(
-        'import_export/sql/cmd.sql',
-        conn=conn,
-        data=data,
-        columns=cols,
-        ignore_column_list=icols
-    )
+    temp_path = 'import_export/sql/#{0}#/cmd.sql'.format(manager.version)
+    query = render_template(temp_path, conn=conn, data=data, columns=cols,
+                            not_null_columns=not_null_cols,
+                            null_columns=null_cols,
+                            force_quote_columns=quote_cols)
 
     args = ['--command', query]