diff --git a/migration-tools/tts-backup-python/README.md b/migration-tools/tts-backup-python/README.md index 1969d60..7f900ea 100644 --- a/migration-tools/tts-backup-python/README.md +++ b/migration-tools/tts-backup-python/README.md @@ -1,76 +1,44 @@ # Transportable Tablespaces Using Backup -Transportable Tablespaces can be used to move tablespaces from customer -on-premise or another database cloud service into ADB-S. Tablespaces can -be transported when creating a new database in ADB-S or as modify -operation on an existing database. +Transportable Tablespaces feature can be used to migrate Oracle database tablespaces from customer on-premise or another Oracle Database Cloud Service into Oracle Autonomous AI Database Cloud Service. User tablespaces along with the necessary schemas can be migrated using the Transportable Tablespaces mechanism. Customers can migrate both bigfile and smallfile tabalespaces. Tablespaces be can encrypted or unencrypted. Migration can be done in full or incremental modes. Customers can use Oracle Cloud Infrastructure (OCI) Object Storage or File Storage Service (FSS) as intermediary for the migration. Source database can be of version 11g or higher. Tablespaces can be migrated to Oracle Autonomous AI Database version 19c or 23ai. ## Step-by-step guide -Transporting Tablespaces involves the following steps. +Transporting Tablespaces involves below high level steps. -1. Create Object Storage Buckets - -2. Create Dynamic Group and Policy - -3. Backup Tablespaces on Source Database - -4. Create or Modify database in ADB-S by specifying intent to transport - tablespaces using a tag +1. Create OCI Object Storage Buckets or FSS Mount Targets for backup +2. Backup tablespaces on source database using Oracle-provided Backup Utility. +3. Create Dynamic Group and Policy to allow Resource Principal access if using OCI Object Storage +4. Provision an Autonomous AI Database by providing migration inputs. ### Create Object Storage Buckets -Transportable Tablespaces needs two object storage buckets - one for -backups and another for metadata. Create the buckets in your Oracle -Storage Cloud Service account. Note the URLs of the buckets as they are -needed as inputs for the operation. Use [Oracle Cloud Infrastructure Object Storage Native URI Format](https://docs.oracle.com/en/cloud/paas/autonomous-database/serverless/adbsb/file-uri-formats.html) for the URL. +Transportable Tablespaces needs two OCI object storage buckets - one for backups and another for metadata. Create the buckets in your Oracle Storage Cloud Service account. Note the URLs of the buckets as they are needed as inputs for the operation. Use [Oracle Cloud Infrastructure Object Storage Native URI Format](https://docs.oracle.com/en/cloud/paas/autonomous-database/serverless/adbsb/file-uri-formats.html) for the URL. Example: [https://objectstorage.region.oraclecloud.com/n/\/b/\](https://objectstorage.region.oraclecloud.com/n/namespace-string/b/bucket/o/filename) -To make Object Storage URI work, please [generate an API signing key](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm)]. -Download the private key .pem file and the API signing key config file with below contents to source database host. - +To make Object Storage URI work, please [generate an API signing key](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm). Download the private key .pem file and the API signing key config file with below contents to source database host. +``` [DEFAULT] user=ocid1.user.oc1..xxxxx - fingerprint=f6:d6:e5:xxxxx - tenancy=ocid1.tenancy.oc1..xxxxx - region=us-ashburn-1 - key_file= +``` Note - User should have read and write access to the object storage buckets. -## Create Dynamic Group and Policy - -Transportable Tablespaces functionality will download metadata from metadata bucket using [OCI Resource Principal](https://docs.oracle.com/en/cloud/paas/autonomous-database/serverless/adbsb/resource-principal.html)]. - -### Create Dynamic Group and Policy to allow access to the metadata bucket using the resource principal. - -1. Create a Dynamic Group **TTSDynamicGroup** with matching rule:\ - ALL {resource.type = \'autonomousdatabase\', [resource.compartment.id](http://resource.compartment.id) = \'your_Compartment_OCID\'} - -2. Create a Policy using the dynamic group with Policy Statement:\ - Allow dynamic-group **TTSDynamicGroup** to manage buckets in tenancy\ - Allow dynamic-group **TTSDynamicGroup** to manage objects in tenancy\ - \ - Prepend domain name to the dynamic group name if needed as below.\ - Allow dynamic-group \<*identity_domain_name\>*/**TTSDynamicGroup** to manage buckets in tenancy\ - Allow dynamic-group \<*identity_domain_name\>*/**TTSDynamicGroup** to manage objects in tenancy - ### Backup Tablespaces on Source Database #### Pre-requisites - Create a Project Directory that will used as staging location on the host running the source database. - Download [Oracle Database Backup Cloud Module](https://www.oracle.com/database/technologies/oracle-cloud-backup-downloads.html) to the Project Directory. Unzip the downloaded opc_installer.zip in the project directory. -- Download [tts-backup-python.zip](file:////confluence/download/attachments/11465427748/tts-backup-python.zip%3fversion=33&modificationDate=1764740426000&api=v2) to the project directory. Unzip the tts-backup-python.zip in the project directory. -- Provide inputs for backup in the **tts-backup-env.txt** file. -- [tts-backup-python.zip for VodaFone](file:////confluence/download/attachments/11465427748/tts-backup-python.zip%3fversion=33&modificationDate=1764740426000&api=v2) +- Download Transportable Tablespaces Backup Utility files to the project directory. +- Provide inputs for backup in the **tts-backup-env.txt** file. #### TTS Backup Tool inputs @@ -90,10 +58,10 @@ Open tts-backup-env.txt file downloaded to the project directory and provide the ***DB_SVC_NAME*** : (REQUIRED INPUT) Database service name, used for connecting to the database.  \ ***ORAHOME*** : (REQUIRED INPUT) Database home, \$ORACLE_HOME.\ ***DBUSER*** : (REQUIRED INPUT) Username for connecting to the database. User should have sysdba privileges. \ -***DBPASSWORD*** : (REQUIRED INPUT) Password for connection to the database. (Provide as CLI argument or Runtime input)\ +***DBPASSWORD*** : (RUNTIME INPUT) Password for connection to the database. DO NOT provide in the file. Provide as CLI runtime input when prompted.\ ***DB_VERSION*** : (REQUIRED INPUT) DB Version, supported values are 11g, 12c, 19c, 23ai. -##### Object Storage Service (OSS) inputs ( Required if using OSS for transport. Leave them empty if using FSS. ) +##### Object Storage Service (OSS) inputs ( Required if using OCI Object Storage for migration. Leave them empty if using FSS. ) ***TTS_BACKUP_URL*** : (REQUIRED INPUT) Object storage bucket uri for backup files.      \ ***TTS_BUNDLE_URL*** : (REQUIRED INPUT) Object storage bucket uri for transportable tablespace bundle.\ @@ -103,20 +71,22 @@ Open tts-backup-env.txt file downloaded to the project directory and provide the ***OCI_PROXY_HOST :** ***(OPTIONAL INPUT) HTTP proxy server.\ ***OCI_PROXY_HOST :** ***(OPTIONAL INPUT)  HTTP proxy server connection port. -##### File Storage Service (FSS) inputs ( Required if using FSS for transport. Leave them empty if using OSS. ) +##### File Storage Service (FSS) inputs ( Required if using FSS for migration. Leave them empty if using OSS. ) ***TTS_FSS_CONFIG*** : (REQUIRED INPUT) FSS configuration should given in the format FSS:\:\:\\ -***TTS_FSS_MOUNT_DIR*** : (REQUIRED INPUT) Absolute path where file system was mounted on source database host\ +***TTS_FSS_MOUNT_DIR*** : (REQUIRED INPUT) Absolute path where file system was mounted on source database host\ \ -Refer to [] for details on how File System should be configured for use by ADB-S. +Refer to []{.underline} for details on how FIle System should be configured for use by ADB-S. ##### TDE keys inputs -***TDE_WALLET_STORE_PASSWORD*** : (REQUIRED only if any of the tablespaces are TDE Encrypted). Required to export TDE KEYS. (Provide as CLI argument or Runtime input) +***TDE_WALLET_STORE_PASSWORD*** : (RUNTIME INPUT, REQUIRED only if any of the tablespaces are TDE Encrypted). DO NOT provide in the file. Provide as CLI runtime input when prompted. ##### Final backup inputs  -***FINAL_BACKUP*** : (REQUIRED INPUT) Final backup will be done only if FINAL_BACKUP=yes or YES. Accepted values YES, yes, NO, no. Used to indicate incremental operation. Specify YES for non-incremental operation. Specify NO for incremental backups. Last operation should be run with YES for schema to be exported. +***FINAL_BACKUP*** : (REQUIRED INPUT) Accepted values TRUE/true or FALSE/false. Value should be TRUE for a non-incremental migration. For incremental migration, +value should be FALSE for all incremental backups including the first backup. Set the value to TRUE for final backup of an incremental migration. Tablespaces +being transported must be set to READ-ONLY when FINAL_BACKUP is set to TRUE. Tablespace and Schema metadata will be exported only when FINAL_BAKCUP is set to TRUE. ##### Performance inputs      @@ -125,55 +95,62 @@ Refer to [*/**TTSDynamicGroup** to manage buckets in tenancy\ + Allow dynamic-group \<*identity_domain_name\>*/**TTSDynamicGroup** to manage objects in tenancy -#### Modify ADB-S database to transport tablespaces +### Create Autonomous AI Database with Migration inputs -Use the below steps to transport tablespaces to an existing database. +Create an Autonomous AI Database from OCI Console using the steps below.  - 1. Go to OCI Console → Oracle Database → Autonomous Database - 2. Select and click on the database for transporting tablespaces - 3. Verify ***Storage*** in ***Resource allocation*** section. Use ***Manage resource allocation*** tab to increase storage if needed. - 4. If this is the first time specifying **ADB\$TTS_BUNDLE_URL** tag on the database: - a. Go to ***More Actions* *→* *Tags** ****menu item on the Autonomous Database Details page.* - b. Select ***Tag namespace*** **as ***None (add a free-form tag)***, ***Tag key*** as ***ADB\$TTS_BUNDLE_URL***, \ - **Tag value** as metadata bundle url given by the TTS backup tool. - c. Click on ***Add Tag.*** - d. Submit ***Add Tags***. - 5. If **ADB\$TTS_BUNDLE_URL** tag was already specified during create or a previous update of the database: - a. Click on the **Tags** tab on the Autonomous Database Details page. - b. Click on **Free-Form tags** tab and edit the **ADB\$TTS_BUNDLE_URL** tag.  - c. Specify the new URL and submit **Save** action. +1. Go to OCI Console → Oracle Database → Autonomous AI Database +2. Click on ***Create Autonomous Database*** +3. Provide all the necessary inputs. +4. Select database version that is equal to or greater than the source database. +5. Specify ***Storage (TB)*** in ***Configure the database*** section to match size of tablespace(s) being transported. +6. Expand ***Migration section*** on the page. +7. Specify OCI Object Storage URL or FSS Mount Target path to the metadata bundle as given by Backup Utility. +8. Submit ***Create Autonomous Database.*** -The operation will trigger the transport tablespaces job on the database. +The operation will first create the database and then trigger migration operation. For non-incremental operation, migration will perform all the necessary steps and the created database will have the transported tablespaces. In case of incremental operation, this is the first step in the migration process. First backup pieces will be restored to the created database. User has to continue with incremental backup up to the final backup to complete the migration process. -#### To transport tablespaces using incremental database backups +### Incremental Migration -Create a database or update an existing database by specifying **ADB\$TTS_BUNDLE_URL** of level 0 backup.\ -For each incremental and final backup, edit the tag using the URL corresponding to that backup as mentioned in **Step 5** above. \ -Before taking the final backup, alter all tablespaces being transported as read-only. Specify FINAL_BACKUP as YES in tts-backup-env.txt.\ -Datafiles with incremental changes are restored to ADB-S from level 0 to final step. Metadata is imported on final step. +If user is performing an incremental migration operation, repeat **Backup Tablespaces** step at source database for each incremental using the same set of inputs. Backup Utility will output a new OCI Object Storage URL / FSS Mount Target path corresponding to that increment. Use the URL/Path to update the Autonomous AI Database created with the first backup above. Set FINAL_BACKUP=TRUE in the input file before performing final backup. -#### To transport tablespaces using non-incremental database backups +#### Modify Autonomous AI Database with Migration inputs -Non-incremental is a one time operation where datafiles are restored and metadata is imported to ADB-S.\ -Alter all tabalespaces being transported as read-only. Specify FINAL_BACKUP as YES in tts-backup-env.txt.\ -Create a database or update an existing database by specifying **ADB\$TTS_BUNDLE_URL** of one time backup. +1. Go to OCI Console → Oracle Database → Autonomous AI Database +2. Select and click on the database created with the first backup. +3. Verify ***Storage*** in ***Resource allocation*** section. Use ***Manage resource allocation*** tab to increase storage if needed. +4. Expand ***Migration section*** on the page. +5. Specify OCI Object Storage URL or FSS Mount Target path to the metadata bundle as given by Backup Utility for the increment or final backup. +6. Click on **Save**. +The operation will trigger the transport tablespaces job on the database. diff --git a/migration-tools/tts-backup-python/tts-backup-env-sample.txt b/migration-tools/tts-backup-python/tts-backup-env-sample.txt new file mode 100644 index 0000000..ef31978 --- /dev/null +++ b/migration-tools/tts-backup-python/tts-backup-env-sample.txt @@ -0,0 +1,55 @@ +################################################################################ +### Project and Tablespace inputs ### +################################################################################ +PROJECT_NAME=tts_project +DATABASE_NAME=orclpdb1 +TABLESPACES=emptbs,salestbs +SCHEMAS=user1,user2,user3 + + +################################################################################ +### Database connection inputs ### +################################################################################ +HOSTNAME=mydbhost1 +LSNR_PORT=1521 +DB_SVC_NAME=orclpdb1.company.com +ORAHOME=/opt/oracle/product/19c/dbhome_1 +DBUSER=mydbuser +DB_VERSION=19c + + +################################################################################ +### Object Storage Service (OSS) inputs ### +################################################################################ +# The following inputs are required only in case of OSS based transport ### +# Please leave them empty if using FSS. ### +################################################################################ +TTS_BACKUP_URL=https://objectstorage.us-ashburn-1.oraclecloud.com/n/mynamespace1/b/tts-backup +TTS_BUNDLE_URL=https://objectstorage.us-ashburn-1.oraclecloud.com/n/mynamespace1/b/tts-metadata +OCI_INSTALLER_PATH=/home/oracle/opc_installer/oci_installer/oci_install.jar +CONFIG_FILE=/home/oracle/OCI_CONFIG/config +COMPARTMENT_OCID=ocid1.compartment.oc1..xxxxx +OCI_PROXY_HOST=proxy.company.com +OCI_PROXY_PORT=80 + + +################################################################################ +### File Storage Service (FSS) inputs ### +################################################################################ +# The following inputs are required only in case of FSS based transport ### +# Please leave them empty if using Object storage. ### +################################################################################ +TTS_FSS_CONFIG=FSS:myFSS:myfss.vcn.com:/myFSS +TTS_FSS_MOUNT_DIR=/u01/app/oracle/myFSS + +################################################################################ +### Final backup inputs ### +################################################################################ +FINAL_BACKUP=TRUE/FALSE + + +################################################################################ +### Performance inputs ### +################################################################################ +PARALLELISM= +CPU_COUNT=