| description | Solutions to the generally asked questions related to the RudderStack data warehouse integrations |
|---|
Rudderstack gives you the flexibility to choose when to sync the latest data into the warehouse. The default option is 30 minutes, but you can choose to extend this value to go up to 24 hours. You can also configure the time at which the data is to be loaded.
Yes, there is! You could do that by configuring below values in config.toml file.
warehouseSyncFreqIgnore = true #if set true this will ignore syncFrequency and syncStartingAt values which are configured in UI. By default this is false
uploadFreqInS=1800 #This field lets you control syncPeriod if above field set to true
The warehouse upload status can be viewed in Live Events section of the destination on the RudderStack dashboard, as shown:
The default namespace will be the source name with some modifications to the name. However, RudderStack also provides an option in the dashboard to change the namespace of the dataset.
Please refer to the warehouse-specific destination settings for configuring the namespace in the RudderStack UI.
If you are using the hosted RudderStack service, the following IPs need to be whitelisted in your destination's security configuration:
3.216.35.9734.198.90.24154.147.40.62
| Parameter | Description | Default |
|---|---|---|
RSERVER_WAREHOUSE_REDSHIFT_MAX_PARALLEL_LOADS
|
Defines the number of concurrent tables that are synced to redshift in a given upload. Increase this as per your infra capability. |
3 |
RSERVER_WAREHOUSE_STAGING_FILES_BATCH_SIZE
|
Defines the number of staging files that are batched and synced in a single upload. Increase this to batch together more files and reduce the number of uploads needed for a given volume of data |
960` |
RSERVER_WAREHOUSE_NO_OF_WORKERS
|
Number of concurrent uploads to a warehouse. Eg. 8 uploads to different schemas in redshift can be done simultaneously | 8 |
[k8s] warehouse_slave.replicaCount
|
Number of warehouse slaves (responsible for load file generation) when warehouse is run as separate service in k8s. Increase this for speeding up the load file generation step of the upload |
2 |
RSERVER_WAREHOUSE_NO_OF_SLAVE_WORKER_ROUTINES
|
Number of go-routines creating load files in a warehouse slave process. Increase/Decrease this as per memory allocated to the warehouse slave pod |
4 |
In case you have any other queries related to using the RudderStack data warehouse integrations, please feel free to contact us. You can also join our Slack channel and start a conversation with us any time. We will be happy to help you.