12-22-2021 08:26 AM - edited 12-22-2021 08:31 AM
If the OP's situation and issue are like mine, the Panoramas (Primary-Active and Secondary/Passive) are in Panorama-mode -- meaning they function as Management and Log Collectors in what might be the only Collector Group. Whenever we upgrade the Panoramas -- we have a pair of M-600s, the ES cluster is red for 6-24 hrs... I think (and need to test this) the best method to avoid this is to stop the LCs on the Panoramas from taking in logs from the managed FWs just seconds before you begin the upgrade on the Secondary PAN -- the ES cluster appears to go red after the upgrade because the ES database on each PAN's LC is out of sync (for lack of a better word). If there is a way to disconnect the Managed Firewalls connections to the LCs, that is likely best overall -- they'll sit and queue locally on the FWs until they can connect to the LCs and forward them on. We've had 3-4 TAC cases about this and about the best advice they could provide was ... don't wait very long to begin the upgrade on the Primary Panorama once the Secondary is done -- the shorter the time period in which the ES cluster's nodes are out of sync, the less shards they have to process in order to get back into sync. (Green) There seems to be very little information out there regarding the ES cluster and issues such as this. Here's my current ES Cluster health:
(primary-active)> show log-collector-es-cluster health
{
"cluster_name" : "__pan_cluster__",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 6,
"number_of_data_nodes" : 4,
"active_primary_shards" : 2506,
"active_shards" : 4886,
"relocating_shards" : 0,
"initializing_shards" : 47,
"unassigned_shards" : 83,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 97.40829346092504
