Alerts on vector metrics

[nabokihms]

Hello, community! I have a question regarding internal vector metrics.

There are cases when:

1. Vector stops collecting logs from Kubernetes.
2. Vector cannot send logs because a sink
   2.1. is unavailable.
   2.2. blocked by a rate limiter.

The solution for 2.2 seems pretty straightforward. I came up with the following promql.

rate(vector_events_discarded_total[2m]) > 0

For the 2.1, it is possible to alert if there are no out events for the sink.

rate(vector_events_out_total{component_kind="sink"}[2m]) == 0

Yet it needs to be more accurate because the real problem is when the end of a sink responds with errors, but I need help finding a generic approach.

No idea how to alert on 1. There are many metrics, and it is not clear to me which one to pick.

I'd appreciate it if someone could take a look at what I'm doing and give me some hints hot to setup alerting rules correctly for vector.

[neuronull]

Hi @nabokihms ,

For 2 , when you say "Vector cannot send logs" , do you mean that the vector source cannot send events downstream? (My following notes assume this is the case).

For 2.1 , you can use https://vector.dev/docs/reference/configuration/sources/internal_metrics/#component_errors_total

For 2.2 , the one you identified is good. This is also helpful, if the sink has a buffer: https://vector.dev/docs/reference/configuration/sources/internal_metrics/#buffer_discarded_events_total

For 1 , component_errors_total is also what I'd recommend. This might also be helpful https://vector.dev/docs/reference/configuration/sources/kubernetes_logs/#k8s_watch_stream_failed_total

Hope this helps

[vparfonov]

Hello, @neuronull, could you assist me with an issue I'm having while using Vector metrics to detect a high rate of errors? Specifically, I'm using the following rule:

100 * 
(sum by(pod, instance)(rate(component_errors_total[2m])) 
/
 sum by(pod, instance)(rate(component_received_events_total[2m]))) 
 > 10

To simulate errors, I'm attempting to send logs to a non-existent Elasticsearch instance at a rate of one log per second using Vector. However, I've noticed that the number of errors is increasing very slowly, with the sum by(pod)(rate(component_errors_total[2m])) occasionally being as low as 0.003 or even 0, and I'm unsure why.

In contrast, the sum by(pod)(rate(component_received_events_total[2m])) has consistently been greater than 304.66, and the sum by(pod)(rate(component_sent_events_total[2m])) has been greater than 216.
Any help you can provide would be greatly appreciated.
Thank you!

[jszwedko]

component_errors_total indicates the count of the errors, rather than a count of errored events, so I'd typically expect it to be much smaller than the count of received events, even it the case that all events error.

You might instead by interested in the component_discarded_events_total metric which counts the number of discarded events.

[vparfonov]

thanks, going to try

[nabokihms]

@neuronull thank you for the answer. The component_errors_total metric works like a charm. I will stick to this.

The only behavior that may be not so obvious is that component_errors_total is a counter, but in combination with the expire_metrics_secs, it behaves like a gauge. If there are no errors, the value of the vector_component_errors_total metric will not be changed, and the expiration will be triggered.

Example of scrapes:

vector_component_errors_total 7 - 7 errors occurred between scrapes
vector_component_errors_total 7
vector_component_errors_total 7
-                               - expiration was triggered
-
-
vector_component_errors_total 3 - 3 errors occurred between scrapes
vector_component_errors_total 3
vector_component_errors_total 3
-                               - expiration was triggered again
-

This behavior makes the result of the rate PromQL function equal to zero. I decided just to use the vector_component_errors_total > 0 and rely on the expire_metrics_secs.

[neuronull]

@neuronull thank you for the answer. The component_errors_total metric works like a charm. I will stick to this.

You are welcome, glad to hear it.

but in combination with the expire_metrics_secs, it behaves like a gauge. If there are no errors, the value of the vector_component_errors_total metric will not be changed, and the expiration will be triggered.

Oh interesting!

[vang95com]

I found a way to detect issues with data sending.

We assume that if we are unable to send data in a timely manner (most likely), it leads to buffer overutilization:

sum by(vm_name,job,component_type,component_kind,component_id)(
  vector_source_buffer_utilization_mean{job="vector"}
)
/
sum by(vm_name,job,component_type,component_kind,component_id)(
  vector_source_buffer_max_event_size{job="vector"}
) * 100 > threshold

Additionally, the presence of errors can serve as a supplementary indicator for some sinks:

sum by(component_id,error_type,stage,component_type)(
  rate(vector_component_errors_total{job="vector"}[$__rate_interval])
) > threshold