config.storage/etcd: connection alerts

[mandesero]

Related dev. issue(s): https://github.com/tarantool/tarantool-ee/pull/1663

Product: Tarantool Enterprise
Since: 3.7
Root documenst:

• https://www.tarantool.io/en/doc/latest/reference/configuration/configuration_reference/#config-etcd
• https://www.tarantool.io/en/doc/latest/reference/configuration/configuration_reference/#config-storage
• (maybe) https://www.tarantool.io/en/doc/latest/platform/configuration/configuration_etcd/

SME: @mandesero @georgiy-belyanin

Details

When connectivity to config.etcd or config.storage is lost, Tarantool set a named connection alert.
The alert is reported as warn for a short outage and escalates to error after the reconnect threshold is exceeded.

For etcd, the escalation timeout is reconnect_timeout * reconnect_max_attempts seconds.
With default values, this is 1 * 10 = 10 seconds.

For config storage (tkv), the escalation timeout is reconnect_after * 5 seconds.
For example, with reconnect_after = 1, the alert becomes error after 5 seconds.

The alert is dropped automatically when connectivity is restored.