config-parameters.md

description	Various types of configuration parameters explained with their types, descriptions, and default values

Configuration Parameters

This document describes the various configuration parameters for the config.toml file. You can fine-tune them to suit your application's needs.

Global Parameters

No.	Parameter name	Type	Description	Default
1	maxProcess	int	Number of parallel threads used in server. Should be set to number of cores.	12
2	gwDBRetentionInHr	int64	Time in hours The amount of time events are stored in gateway database after they have been processed	0
3	routerDBRetention	int64	Time in seconds The amount of time events are stored in router database after they have been processed	0
4	enableProcessor	bool	A boolean that enables or disables the processor module. Will be set to false when running in degraded mode	true
5	enableRouter	bool	A boolean that enables or disables the router module. Will be set to false when running in degraded mode	true
6	enableStats	bool	A boolean that enables or disables stats.	true

[Gateway]

No.	Parameter name	Type	Description	Default
1	webPort	int	The port on which the server runs.	8080
2	maxDBWriterProcess	int	As requests come in to the gateway and are batched, maxDBWriterProcess writers are run to send these batches to db and config-backend	64
3	CustomVal	string	For creation of a job in jobsDB, CustomVal i.e.GW will be assigned for the CustomVal column in jobsDB	"GW"
4	maxBatchSize	int	Batch size used in Gateway. Requests are batched up to size maxBatchSize before writing to DB.	32
5	batchTimeoutInMS	int64	Time in MilliSeconds In case the request batches do not meet the maxBatchSize, every 20 milliseconds these request batches are are sent to db and config-backend.	20
6	maxReqSizeInKB	int	One example where an error message ("Request size exceeds max limit") is thrown for a particular request is when its size in KB crosses maxReqSizeInKB.	100000
7	enableDedup	bool	Enable or disable deduplication of events. We use message_id to de-dup. Duplicate events are dropped at the gateway.	false
8	dedupWindowInS	int	Time in Seconds Events with the same message_id with in this time frame are considered duplicate and are dropped	86400
9	enableRateLimit	bool	Rate limit the number of requests accepted by the gateway. Used for running hosted service	false

[SourceDebugger]

No.	Parameter name	Type	Description	Default
1	disableEventUploads	bool	A bool to enable or disable event schema upload.	false
2	maxBatchSize	int	Maximum size of the live events batch sent to config-backend.	32
3	maxESQueueSize	int	Maximum size of the live events queue in the memory.	1024
4	maxRetry	int	Maximum number of attempts by the server to upload the request batches, in case of errors.	3
5	batchTimeoutInS	int64	Time in seconds In case the request batches do not meet the maxBatchSize, every 2 seconds these request batches are uploaded to config-backend.	2
6	retrySleepInMS	int64	Time in milliseconds In case of error while uploading the request batches, the server instead of retrying continuously for maxRetry times will sleep for 100 milliseconds before retrying.	100

[JobsDB]

No.	Parameter name	Type	Description	Default
1	jobDoneMigrateThres	float64	If (deletedJobsCount / totalJobsCount) > jobDoneMigrateThres, migrate the table of the jobs that have been processed.	0.8
2	jobStatusMigrateThres	float64	If (statusCount / totalCount) > jobStatusMigrateThres, migrate the table of jobs that have been processed.	5
3	maxDSSize	int	The maximum size of a table. If the maximum size is reached, migrate all the jobs in that table.	100000
4	maxMigrateOnce	int	The maximum number of tables that can be migrated together.	10
5	maxTableSizeInMB	int	Maximum size of tables in MB. Tables are migrated if they cross this limit.	300
6	migrateDSLoopSleepDurationInS	int64	Time in seconds Time to sleep before migrating datasets.	30
7	maxMigrateDSProbe	int64	The maximum number of tables probed to find if they can be migrated.	10
8	addNewDSLoopSleepDurationInS	int64	Time in seconds Time to sleep before adding a dataset.	5
9	backupCheckSleepDurationIns	int64	Time in seconds Time interval to sleep before checking if a dataset needs backing up or not.	2
10	enableBackup	bool	Enables/Disables backup. This is set to false in degraded mode.	true

[Router]

No.	Parameter name	Type	Description	Default
1	jobQueryBatchSize	int	The size of a batch of jobs to get from database. A list of retry list, processing list or unprocessed list or executed list.	10000
2	updateStatusBatchSize	int	The minimum size needed to update the status of a batch of jobs.	1000
3	readSleepInMS	int64	Time in milliseconds The time to wait for before the next iteration of the loop. In case the length of unprocessed and retry list is zero, sleep for 10 __milliseconds. Else add it to another list.	10
4	noOfWorkers	int	The total number of workers that are started to send events to destinations.	64
5	noOfJobsPerChannel	int	The number of jobs a channel in each worker can contain. So, each worker gets 1000 jobs.	1000
6	ser	int		3
7	maxSleepInS	int64	Time in seconds The time to sleep when response status code is not 200. This is to give time before the next retry.	60
8	minSleepInS	int64	Time in seconds Time to sleep when response status code is 200.	0
9	maxStatusUpdateWaitInS	int64	Time in seconds Time to sleep before ending one round of stat collection.	5
11	useTestSink	bool	Run internal tests.	false
12	maxFailedCountForJob	int	The maximum number of times a job can fail before marking it as aborted.	8
13	guaranteeUserEventOrder	bool	Maintain the user events order	true
14	retryTimeWindowInMins	int	Minimum retry window in case of errors like 5XX, 429	180
15	minRetryBackoffInS	int	Minimum time in seconds before the next retry in case of errors like 5XX, 429	10
16	maxRetryBackoffInS	int	Maximum time in seconds between errors in case of errors like 5XX, 429	300

[BatchRouter]

No.	Parameter name	Type	Description	Default
1	mainLoopSleepInS	int64	Timeout in seconds while running main loop.	30
2	noOfWorkers	int	The number of workers to batch jobs before deletion.	8
3	jobQueryBatchSize	int	Number of events picked up from batch router's jobsdb in each query	100000
4	uploadFreqInS	int	The Frequency with which batch router dumps events to storage destinations.	30
5	maxFailedCountForJob	int	The maximum number of times a job can fail before marking it as aborted.	128

[Warehouse]

You can ignore these settings if you don't have any warehouses (like redshift, bigquery, etc.) enabled in your config.

No.	Parameter name	Type	Description	Default
1	stagingFilesTable	string	Table name of staging files.	wh_staging_files
2	loadFilesTable	string	Table name of load files.	wh_load_files
3	uploadsTable	string	Table name of uploads.	wh_uploads
4	schemasTable	string	Table name of schemas.	wh_schemas
5	uploadFreqInS	int	Frequency of upload in seconds	1800
6	noOfWorkers	int	Number of concurrent writes to the warehouse	8
7	mainLoopSleepInS	int	Time in Seconds Time to wait between multiple warehouse writes	600
8	stagingFilesBatchSize	int	Batch size of staging files	240

[Processor]

No.	Variable name	Type	Description	Default
1	loopSleepInMS	int64	Time in MilliSeconds In case the length of user jobs process queue is zero or the sum of unprocessed and retry list is empty, sleep for 10 milliseconds is called.	10
2	maxLoopSleepInMS	int64	Max processor loop sleep time in milli seconds	5000
2	dbReadBatchSize	int	The total number of events to get as a batch from the database.	10000
3	transformBatchSize	int	Batch size of events added to request queue before sending to transformation server.	10
4	userTransformBatchSize	int	Batch size of events added to request queue before sending to custom transformation server. Only used when user transformation functions are connected to a destination.	200
5	sessionThresholdEvents	int	The minimum number of events needed to be process further.	20
6	sessionThresholdInS	int64	Time in seconds Minimum time needed before a new session is created.	10
7	maxChanSize	int	The maximum channel size for request and response queue in transformer.	2048
8	processSessions	bool	If processSessions is marked true, the status of the job in db is updated.	false
9	numTransformWorker	int	The number of go transform workers.	8
10	maxRetry	int	The maximum number of times a transformer can retry hitting the API in case of an error	30
11	retrySleepInMS	int64	Time in milliseconds Sleep time in case of an error by transformer while hitting the API, before trying to hit the API again	100

[BackendConfig]

No.	Variable name	Type	Description	Default
1	pollIntervalInS	int64	Time in seconds Frequency of updating data from config-backend .	5
2	configFromFile	bool	Reads backend workspace config from a JSON file instead of fetching form api.	false
3	configJSONPath	string	Path of the JSON file which contains the backend workspace config	./workspaceConfig.json

[RateLimiting]

No.	Variable name	Type	Description	Default
1	eventLimit	int64	Max number of events to be allowed in a time interval	1000
2	rateLimitWindowInMins	int32	Time in minutes Rolling time interval used to limit number of events allowed	60
3	noOfBucketsInWindow	int32	Number of buckets rateLimitWindowInMins is broken down into	12

[Diagnosis]

No.	Parameter name	Type	Description	Default
1	enableDiagnosis	bool	Boolean to send server diagnostics report to us. Disabling this will disable sending all diagnostics information.	true
2	gatewayTimePeriodInS	int32	Time interval to send gateway requests report in seconds	60
3	routerTimePeriodInS	int32	Time interval to send router requests report in seconds	60
4	batchRouterTimePeriodInS	int32	Time interval to send batch router requests report in seconds	600
5	enableServerStartMetric	bool	Boolean to send server start event	true
6	enableConfigIdentifyMetric	bool	Boolean to send workspace config received event	true
7	enableServerStartedMetric	bool	Boolean to send server successfully started event	true
8	enableConfigProcessedMetric	bool	Boolean to send workspace config details	true
9	enableGatewayMetric	bool	Boolean to send gateway request metrics	true
10	enableRouterMetric	bool	Boolean to send router request metrics	true
11	enableBatchRouterMetric	bool	Boolean to send batch router request metrics	true
12	enableDestinationFailuresMetric	bool	Boolean to send destination failures metrics	true