Skip to content

corrected README and added tts-backup-env-sample #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
malcherm merged 1 commit into from
Dec 16, 2025
Merged

corrected README and added tts-backup-env-sample #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 67 additions & 90 deletions migration-tools/tts-backup-python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,76 +1,44 @@
# Transportable Tablespaces Using Backup

Transportable Tablespaces can be used to move tablespaces from customer
on-premise or another database cloud service into ADB-S. Tablespaces can
be transported when creating a new database in ADB-S or as modify
operation on an existing database.
Transportable Tablespaces feature can be used to migrate Oracle database tablespaces from customer on-premise or another Oracle Database Cloud Service into Oracle Autonomous AI Database Cloud Service. User tablespaces along with the necessary schemas can be migrated using the Transportable Tablespaces mechanism. Customers can migrate both bigfile and smallfile tabalespaces. Tablespaces be can encrypted or unencrypted. Migration can be done in full or incremental modes. Customers can use Oracle Cloud Infrastructure (OCI) Object Storage or File Storage Service (FSS) as intermediary for the migration. Source database can be of version 11g or higher. Tablespaces can be migrated to Oracle Autonomous AI Database version 19c or 23ai.

## Step-by-step guide

Transporting Tablespaces involves the following steps.
Transporting Tablespaces involves below high level steps.

1. Create Object Storage Buckets

2. Create Dynamic Group and Policy

3. Backup Tablespaces on Source Database

4. Create or Modify database in ADB-S by specifying intent to transport
tablespaces using a tag
1. Create OCI Object Storage Buckets or FSS Mount Targets for backup
2. Backup tablespaces on source database using Oracle-provided Backup Utility.
3. Create Dynamic Group and Policy to allow Resource Principal access if using OCI Object Storage
4. Provision an Autonomous AI Database by providing migration inputs.

### Create Object Storage Buckets

Transportable Tablespaces needs two object storage buckets - one for
backups and another for metadata. Create the buckets in your Oracle
Storage Cloud Service account. Note the URLs of the buckets as they are
needed as inputs for the operation. Use [Oracle Cloud Infrastructure Object Storage Native URI Format](https://docs.oracle.com/en/cloud/paas/autonomous-database/serverless/adbsb/file-uri-formats.html) for the URL.
Transportable Tablespaces needs two OCI object storage buckets - one for backups and another for metadata. Create the buckets in your Oracle Storage Cloud Service account. Note the URLs of the buckets as they are needed as inputs for the operation. Use [Oracle Cloud Infrastructure Object Storage Native URI Format](https://docs.oracle.com/en/cloud/paas/autonomous-database/serverless/adbsb/file-uri-formats.html) for the URL.

Example:
[https://objectstorage.region.oraclecloud.com/n/\<namespace-string\>/b/\<bucket-name\>](https://objectstorage.region.oraclecloud.com/n/namespace-string/b/bucket/o/filename)

To make Object Storage URI work, please [generate an API signing key](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm)].
Download the private key .pem file and the API signing key config file with below contents to source database host.

To make Object Storage URI work, please [generate an API signing key](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm). Download the private key .pem file and the API signing key config file with below contents to source database host.
```
[DEFAULT]

user=ocid1.user.oc1..xxxxx

fingerprint=f6:d6:e5:xxxxx

tenancy=ocid1.tenancy.oc1..xxxxx

region=us-ashburn-1

key_file=<path to the downloaded private key file>

```
Note - User should have read and write access to the object storage buckets.

## Create Dynamic Group and Policy

Transportable Tablespaces functionality will download metadata from metadata bucket using [OCI Resource Principal](https://docs.oracle.com/en/cloud/paas/autonomous-database/serverless/adbsb/resource-principal.html)].

### Create Dynamic Group and Policy to allow access to the metadata bucket using the resource principal.

1. Create a Dynamic Group **TTSDynamicGroup** with matching rule:\
ALL {resource.type = \'autonomousdatabase\', [resource.compartment.id](http://resource.compartment.id) = \'your_Compartment_OCID\'}

2. Create a Policy using the dynamic group with Policy Statement:\
Allow dynamic-group **TTSDynamicGroup** to manage buckets in tenancy\
Allow dynamic-group **TTSDynamicGroup** to manage objects in tenancy\
\
Prepend domain name to the dynamic group name if needed as below.\
Allow dynamic-group \<*identity_domain_name\>*/**TTSDynamicGroup** to manage buckets in tenancy\
Allow dynamic-group \<*identity_domain_name\>*/**TTSDynamicGroup** to manage objects in tenancy

### Backup Tablespaces on Source Database

#### Pre-requisites

- Create a Project Directory that will used as staging location on the host running the source database.
- Download [Oracle Database Backup Cloud Module](https://www.oracle.com/database/technologies/oracle-cloud-backup-downloads.html) to the Project Directory. Unzip the downloaded opc_installer.zip in the project directory.
- Download [tts-backup-python.zip](file:////confluence/download/attachments/11465427748/tts-backup-python.zip%3fversion=33&modificationDate=1764740426000&api=v2) to the project directory. Unzip the tts-backup-python.zip in the project directory.
- Provide inputs for backup in the **tts-backup-env.txt** file.
- [tts-backup-python.zip for VodaFone](file:////confluence/download/attachments/11465427748/tts-backup-python.zip%3fversion=33&modificationDate=1764740426000&api=v2)
- Download Transportable Tablespaces Backup Utility files to the project directory.
- Provide inputs for backup in the **tts-backup-env.txt** file.

#### TTS Backup Tool inputs

Expand All @@ -90,10 +58,10 @@ Open tts-backup-env.txt file downloaded to the project directory and provide the
***DB_SVC_NAME*** : (REQUIRED INPUT) Database service name, used for connecting to the database.  \
***ORAHOME*** : (REQUIRED INPUT) Database home, \$ORACLE_HOME.\
***DBUSER*** : (REQUIRED INPUT) Username for connecting to the database. User should have sysdba privileges. \
***DBPASSWORD*** : (REQUIRED INPUT) Password for connection to the database. (Provide as CLI argument or Runtime input)\
***DBPASSWORD*** : (RUNTIME INPUT) Password for connection to the database. DO NOT provide in the file. Provide as CLI runtime input when prompted.\
***DB_VERSION*** : (REQUIRED INPUT) DB Version, supported values are 11g, 12c, 19c, 23ai.

##### Object Storage Service (OSS) inputs ( Required if using OSS for transport. Leave them empty if using FSS. )
##### Object Storage Service (OSS) inputs ( Required if using OCI Object Storage for migration. Leave them empty if using FSS. )

***TTS_BACKUP_URL*** : (REQUIRED INPUT) Object storage bucket uri for backup files.      \
***TTS_BUNDLE_URL*** : (REQUIRED INPUT) Object storage bucket uri for transportable tablespace bundle.\
Expand All @@ -103,20 +71,22 @@ Open tts-backup-env.txt file downloaded to the project directory and provide the
***OCI_PROXY_HOST :** ***(OPTIONAL INPUT) HTTP proxy server.\
***OCI_PROXY_HOST :** ***(OPTIONAL INPUT)  HTTP proxy server connection port.

##### File Storage Service (FSS) inputs ( Required if using FSS for transport. Leave them empty if using OSS. )
##### File Storage Service (FSS) inputs ( Required if using FSS for migration. Leave them empty if using OSS. )

***TTS_FSS_CONFIG*** : (REQUIRED INPUT) FSS configuration should given in the format FSS:\<FIle System Name\>:\<FQDN of Mount Target\>:\<Export Path\>\
***TTS_FSS_MOUNT_DIR*** : (REQUIRED INPUT) Absolute path where file system was mounted on source database host\
***TTS_FSS_MOUNT_DIR*** : (REQUIRED INPUT) Absolute path where file system was mounted on source database host\
\
Refer to [<https://blogs.oracle.com/datawarehousing/post/attach-file-system-autonomous-database>] for details on how File System should be configured for use by ADB-S.
Refer to [<https://blogs.oracle.com/datawarehousing/post/attach-file-system-autonomous-database>]{.underline} for details on how FIle System should be configured for use by ADB-S.

##### TDE keys inputs

***TDE_WALLET_STORE_PASSWORD*** : (REQUIRED only if any of the tablespaces are TDE Encrypted). Required to export TDE KEYS. (Provide as CLI argument or Runtime input)
***TDE_WALLET_STORE_PASSWORD*** : (RUNTIME INPUT, REQUIRED only if any of the tablespaces are TDE Encrypted). DO NOT provide in the file. Provide as CLI runtime input when prompted.

##### Final backup inputs 

***FINAL_BACKUP*** : (REQUIRED INPUT) Final backup will be done only if FINAL_BACKUP=yes or YES. Accepted values YES, yes, NO, no. Used to indicate incremental operation. Specify YES for non-incremental operation. Specify NO for incremental backups. Last operation should be run with YES for schema to be exported.
***FINAL_BACKUP*** : (REQUIRED INPUT) Accepted values TRUE/true or FALSE/false. Value should be TRUE for a non-incremental migration. For incremental migration,
value should be FALSE for all incremental backups including the first backup. Set the value to TRUE for final backup of an incremental migration. Tablespaces
being transported must be set to READ-ONLY when FINAL_BACKUP is set to TRUE. Tablespace and Schema metadata will be exported only when FINAL_BAKCUP is set to TRUE.

##### Performance inputs     

Expand All @@ -125,55 +95,62 @@ Refer to [<https://blogs.oracle.com/datawarehousing/post/attach-file-system-auto

Leave these a blank unless really needed.

### Create or Modify ADB-S database with TTS tag
**Backup Utility Sample Inputs**

Here is an example of [backup utility sample inputs] (https://https://github.com/oracle-devrel/oracle-autonomous-database-samples/tree/main/migration-tools/tts-backup-python/tts_backup_utility_sample_inputs.txt

#### Create ADB-S database to transport tablespaces
Run the TTS Backup Tool from the project directory as below. User will be prompted for database password and optional TDE wallet store password.

Use the below steps to transport tablespaces while creating an ADB-S database.
**Run Backup Utility**
```
$ python3 tts-backup.py 
Enter value for required variable DBPASSWORD: Test
Enter value for optional variable TDE_WALLET_STORE_PASSWORD 
Required only if any of the tablespaces are TDE encrypted (leave empty and press Enter if not applicable):

1. Go to OCI Console → Oracle Database → Autonomous Database
2. Click on ***Create Autonomous Database***
3. Provide all the necessary inputs.
4. Select database version that is equal to or greater than the source database.
5. Specify ***Storage (TB)*** in ***Configure the database*** section to match size of tablespace(s) being transported.
6. Click on ***Show advanced options*** at the bottom of the page. Click on ***Tags*** tab in the section.
7. Select ***Tag namespace*** **as ***None (add a free-form tag)***, ***Tag key*** as ***ADB\$TTS_BUNDLE_URL***,
**Tag value** as metadata bundle url given by the TTS backup tool.
8. Click on ***Add Tag.***
9. Submit ***Create Autonomous Database.***
```
The tool will take backups of the tablespace datafiles and create a metadata bundle. Both backups and the bundle will be uploaded to the provided OCI Object Storage buckets or FSS Mount Targets. Backup Utility will output an URL to OCI Object Storage metadata bundle or FSS Mount Target path for metadata bundle. User should note the given URL/Path as that will be needed as migration input when creating Autonomous AI Database for the migration.

The operation will first create the database and then trigger transport tablespaces job.
### Create Dynamic Group and Policy (for OCI Object Storage backups)

Transportable Tablespaces functionality will download metadata from OCI Object Storage metadata bucket using [OCI Resource Principal](https://docs.oracle.com/en/cloud/paas/autonomous-database/serverless/adbsb/resource-principal.html). Create Dynamic Group and Policy to allow access to the metadata bucket using the resource principal. This step is not required if using FSS for backups.

1. Create a Dynamic Group **TTSDynamicGroup** with matching rule:\
ALL {resource.type = \'autonomousdatabase\', [resource.compartment.id](http://resource.compartment.id) = \'your_Compartment_OCID\'}
2. Create a Policy using the dynamic group with Policy Statement:\
Allow dynamic-group **TTSDynamicGroup** to manage buckets in tenancy\
Allow dynamic-group **TTSDynamicGroup** to manage objects in tenancy\
\
Prepend domain name to the dynamic group name if needed as below.\
Allow dynamic-group \<*identity_domain_name\>*/**TTSDynamicGroup** to manage buckets in tenancy\
Allow dynamic-group \<*identity_domain_name\>*/**TTSDynamicGroup** to manage objects in tenancy

#### Modify ADB-S database to transport tablespaces
### Create Autonomous AI Database with Migration inputs

Use the below steps to transport tablespaces to an existing database.
Create an Autonomous AI Database from OCI Console using the steps below. 

1. Go to OCI Console → Oracle Database → Autonomous Database
2. Select and click on the database for transporting tablespaces
3. Verify ***Storage*** in ***Resource allocation*** section. Use ***Manage resource allocation*** tab to increase storage if needed.
4. If this is the first time specifying **ADB\$TTS_BUNDLE_URL** tag on the database:
a. Go to ***More Actions* *→* *Tags** ****menu item on the Autonomous Database Details page.*
b. Select ***Tag namespace*** **as ***None (add a free-form tag)***, ***Tag key*** as ***ADB\$TTS_BUNDLE_URL***, \
**Tag value** as metadata bundle url given by the TTS backup tool.
c. Click on ***Add Tag.***
d. Submit ***Add Tags***.
5. If **ADB\$TTS_BUNDLE_URL** tag was already specified during create or a previous update of the database:
a. Click on the **Tags** tab on the Autonomous Database Details page.
b. Click on **Free-Form tags** tab and edit the **ADB\$TTS_BUNDLE_URL** tag. 
c. Specify the new URL and submit **Save** action.
1. Go to OCI Console → Oracle Database → Autonomous AI Database
2. Click on ***Create Autonomous Database***
3. Provide all the necessary inputs.
4. Select database version that is equal to or greater than the source database.
5. Specify ***Storage (TB)*** in ***Configure the database*** section to match size of tablespace(s) being transported.
6. Expand ***Migration section*** on the page.
7. Specify OCI Object Storage URL or FSS Mount Target path to the metadata bundle as given by Backup Utility.
8. Submit ***Create Autonomous Database.***

The operation will trigger the transport tablespaces job on the database.
The operation will first create the database and then trigger migration operation. For non-incremental operation, migration will perform all the necessary steps and the created database will have the transported tablespaces. In case of incremental operation, this is the first step in the migration process. First backup pieces will be restored to the created database. User has to continue with incremental backup up to the final backup to complete the migration process.

#### To transport tablespaces using incremental database backups
### Incremental Migration

Create a database or update an existing database by specifying **ADB\$TTS_BUNDLE_URL** of level 0 backup.\
For each incremental and final backup, edit the tag using the URL corresponding to that backup as mentioned in **Step 5** above. \
Before taking the final backup, alter all tablespaces being transported as read-only. Specify FINAL_BACKUP as YES in tts-backup-env.txt.\
Datafiles with incremental changes are restored to ADB-S from level 0 to final step. Metadata is imported on final step.
If user is performing an incremental migration operation, repeat **Backup Tablespaces** step at source database for each incremental using the same set of inputs. Backup Utility will output a new OCI Object Storage URL / FSS Mount Target path corresponding to that increment. Use the URL/Path to update the Autonomous AI Database created with the first backup above. Set FINAL_BACKUP=TRUE in the input file before performing final backup.

#### To transport tablespaces using non-incremental database backups
#### Modify Autonomous AI Database with Migration inputs

Non-incremental is a one time operation where datafiles are restored and metadata is imported to ADB-S.\
Alter all tabalespaces being transported as read-only. Specify FINAL_BACKUP as YES in tts-backup-env.txt.\
Create a database or update an existing database by specifying **ADB\$TTS_BUNDLE_URL** of one time backup.
1. Go to OCI Console → Oracle Database → Autonomous AI Database
2. Select and click on the database created with the first backup.
3. Verify ***Storage*** in ***Resource allocation*** section. Use ***Manage resource allocation*** tab to increase storage if needed.
4. Expand ***Migration section*** on the page.
5. Specify OCI Object Storage URL or FSS Mount Target path to the metadata bundle as given by Backup Utility for the increment or final backup.
6. Click on **Save**.

The operation will trigger the transport tablespaces job on the database.
55 changes: 55 additions & 0 deletions migration-tools/tts-backup-python/tts-backup-env-sample.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
################################################################################
### Project and Tablespace inputs ###
################################################################################
PROJECT_NAME=tts_project
DATABASE_NAME=orclpdb1
TABLESPACES=emptbs,salestbs
SCHEMAS=user1,user2,user3


################################################################################
### Database connection inputs ###
################################################################################
HOSTNAME=mydbhost1
LSNR_PORT=1521
DB_SVC_NAME=orclpdb1.company.com
ORAHOME=/opt/oracle/product/19c/dbhome_1
DBUSER=mydbuser
DB_VERSION=19c


################################################################################
### Object Storage Service (OSS) inputs ###
################################################################################
# The following inputs are required only in case of OSS based transport ###
# Please leave them empty if using FSS. ###
################################################################################
TTS_BACKUP_URL=https://objectstorage.us-ashburn-1.oraclecloud.com/n/mynamespace1/b/tts-backup
TTS_BUNDLE_URL=https://objectstorage.us-ashburn-1.oraclecloud.com/n/mynamespace1/b/tts-metadata
OCI_INSTALLER_PATH=/home/oracle/opc_installer/oci_installer/oci_install.jar
CONFIG_FILE=/home/oracle/OCI_CONFIG/config
COMPARTMENT_OCID=ocid1.compartment.oc1..xxxxx
OCI_PROXY_HOST=proxy.company.com
OCI_PROXY_PORT=80


################################################################################
### File Storage Service (FSS) inputs ###
################################################################################
# The following inputs are required only in case of FSS based transport ###
# Please leave them empty if using Object storage. ###
################################################################################
TTS_FSS_CONFIG=FSS:myFSS:myfss.vcn.com:/myFSS
TTS_FSS_MOUNT_DIR=/u01/app/oracle/myFSS

################################################################################
### Final backup inputs ###
################################################################################
FINAL_BACKUP=TRUE/FALSE


################################################################################
### Performance inputs ###
################################################################################
PARALLELISM=
CPU_COUNT=