Seconday Passive ES Health Red master_not_discovered_exception

[MatthewJenkins]

I have two Panorama servers 11.0.1-h2 in HA. Both are in Panorama mode and have 3x Disks (2TB each). Logs are really slow to view on the Active-Primary and never load on the Secondary. When diagnosing I came across this error on the secondary:

admin@Panorama2(secondary-passive)> show log-collector-es-cluster health

{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

I tried waiting 24 hours to see if it cleared up, but no luck. 

Does anyone know how to resolve this?

[PavelK]

Hello @MatthewJenkins 

PAN-OS 11 is quite new and at least based on list of known issues there are a few bugs related to ElasticSearch. As a next step I would try to restart ElasticSearch processes: debug elasticsearch es-restart optional all

Kind Regards

Pavel

[MatthewJenkins]

Doing that on both panorama servers resulted in both now having the error master_not_discovered_exception.

Reboots are not working either. 

[PavelK]

Hello @MatthewJenkins

I see, sorry to hear that. If even after rebooting you are getting the same error, I would open a TAC ticket. The output from "show log-collector-es-cluster health" should return status of ES instead of this error. This might required TAC's root access to resolve it.

Kind Regards

Pavel

[PStenstadvold]

Ha similar fault on 10.2.7-h3 log collectors in cluster/group

The command:

debug elasticsearch es-restart option all

fixed the issue for me.