High Bandwidth on Broker VM: Cluster Mismatch (v29 vs v28) & P2P

[QuestionAb]

Hi everyone,

I'm facing high bandwidth usage on my Primary Broker VM. I need to validate if my diagnosis is correct:

The Setup:

• Cluster: HA Pair. Node 1 is v29.0.77 (Healthy). Node 2 is v28.0.99 (Service "Local Agent Settings" is Red/Down).

• Policy: Download Source = Broker VM (P2P is currently disabled).

My Questions:

1. Cluster: Does the version mismatch cause the load balancing to fail, forcing Node 1 to handle 100% of the traffic?

2. Agent Logic: If "Broker VM" is selected in the policy, do agents always prioritize the Broker over the Cloud (Internet), even if they have direct Internet access?

I plan to enable P2P and upgrade Node 2 to match versions. Is this the right path?

Thanks!

[susekar]

Hello @QuestionAb ,

Greetings for the day.

---

1. Cluster Load Balancing and Node Failure

Does the version mismatch cause load balancing to fail, forcing Node 1 to handle 100% of the traffic?

Native Load Balancing
The Broker VM cluster feature is designed for high availability and failover redundancy, but it does not provide native load balancing to actively distribute traffic across nodes. If active/active traffic distribution is required, it must be implemented using an external load balancer.

Failover Logic
When a node in the cluster is marked as Red/Down, the HA mechanism shifts all responsibilities to the healthy node. Since Node 2’s Local Agent Settings service is down, it cannot serve agents, which effectively forces Node 1 to handle all agent traffic.

Version Mismatch
Rolling upgrades are supported to minimize downtime, but running cluster nodes on different versions for an extended period can cause synchronization inconsistencies. Version v28.0.99 is associated with a known issue that causes repeated content downloads and excessive bandwidth usage. This behavior can further increase the load on the remaining healthy node.

---

2. Agent Logic and Download Priority

If “Broker VM” is selected in the policy, do agents always prioritize the Broker over the Cloud?

Priority Order
When Broker VM is selected as a download source, the Cortex XDR agent follows this hierarchy:
P2P → Broker VM → Cortex Server (Cloud)

Prioritization Behavior
If P2P is disabled, agents will attempt to download content from the Broker VM first. They will fall back to the Cloud only if they cannot connect to the Broker VM or if the Broker returns an error (for example, authentication, authorization, or SSL-related failures).

Direct Server Access
If Direct Server Access is enabled in the agent settings profile, agents may bypass the Broker VM and connect directly to the Cortex cloud if they detect connectivity or caching issues with the Broker.

3. Recommended Path

Is enabling P2P and upgrading Node 2 the right path?

Yes, this is the recommended approach.

Upgrade Node 2
Upgrading Node 2 from v28.0.99 is critical due to the known content re-download issue that causes excessive bandwidth usage. Aligning Node 2 with Node 1 on v29.x will also restore proper HA behavior and reduce synchronization risks.

Enable P2P
Enabling Peer-to-Peer (P2P) significantly reduces the load on the Broker VM by allowing agents to share content packages within the same local network segment.

Recommended Troubleshooting Steps

1. Verify caching prerequisites
   Ensure Node 1 has a valid FQDN and properly configured SSL certificates, as these are required for Broker VM content caching.

2. Check disk space
   Insufficient space on the /data partition can cause content download failures and abnormal bandwidth usage.
   Run:

   df -h /data
   

3. Monitor active connections
   Review active agent connections on Node 1. Note that the Local Agent Settings applet only shows connections that are actively transferring data at that exact moment, not total registered agents.

If you feel this has answered your query, please let us know by clicking like and on "mark this as a Solution".

Happy New year!!

Thanks & Regards,
S. Subashkar Sekar