kubernetes plugin - monitoring definition, bug?

YLesev — Thu, 23 May 2024 10:52:30 GMT

Took me long, but now i think i found strange behavior of the Kubernetes plugin - monitoring definition and especially matching criteria in the dynamic address groups.

TL/DR - having service in the kubernetes cluster, which does not have assigned ports - breaks the polling and watcher

Starting point was : working cn series as well the possibility to use the data from monitoring definition - both: to create dynamic groups and to use them in policy. Suddenly after while the monitoring definition processes (polling and watcher) stopped function (stuck in initializing state). Long story short - discovered in the kubernetes plugin logs the following suspicious lines

2024-05-23 13:21:39.868 +0300 ERROR: k8s-ret: skipping service, no port information found 'NoneType' object is not iterable 2024-05-23 13:21:39.869 +0300 ERROR: k8s-ret: skipping service, no port information found 'NoneType' object is not iterable

Looking at my kubernetes cluster - there were two distinct from the other services - without ports (PORT(S) column)

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE longhorn-system longhorn-engine-manager ClusterIP None <none> <none> 87d longhorn-system longhorn-replica-manager ClusterIP None <none> <none> 87d

Next - i just add some ports definition to them - as i want to see if that will make any difference to the panorama monitoring definition processes. And it does - the processes started in normal state (connected)

2024-05-23 13:24:20.327 +0300 DEBUG: k8s-ret: <labkube> Successfully retrieved Service information.

Still not sure for the root cause of this behavior, probably can be bug. As having services without ports seems just ok for me.
Hope that, this behavior is reproducible elsewhere and will be addressed?

Re: kubernetes plugin - monitoring definition, bug?

JayGolf — Wed, 29 May 2024 15:36:35 GMT

Hi @YLesev ,

Thanks for sharing your findings. I would recommend reaching out to support to share what you have.

topic kubernetes plugin - monitoring definition, bug? in CN-Series Discussions

kubernetes plugin - monitoring definition, bug?

Re: kubernetes plugin - monitoring definition, bug?