Panorama Log Collector VM Cluster in 'Yellow' Status.

[Paul_Stinson]

We had some issues with licensing for one of the nodes.

We have since rectified this issue (switching to Panorama mode changed serial, licensed and changed back to logger mode).

On initial bootup Panorama was reporting log collectors connected and in config sync.

However there was a message around 'inter-lc-connectivity' not working.

We rebooted both log collectors within say 30secs of each other (as this has fixed this particular issue in the past).

On bootup I could see that Panorama was reporting "connected to all LC's in group" now with no issues reported. 

So Panorama looks pretty good and you would think no issues from this side.

**Edit: checking log collector detail shows a low number on amount of detailed storage "9days" etc which doesn't marry up with the list of "show log-collector-es-indices" which shows some traffic indices as 20Gb in size. 

Maybe the system is still doing something in the backend or we have instead some orphaned indices taking up space (any ideas on this welcome).

However checking the log collectors themselves I saw issues. 

When performing "show log-collector-es-cluster health" it moved from red to yellow as it checked local logs and 'active-shards....' increased until it hit around 99% mark.

It seems to be staying on yellow though I suspect due to 4 shards in an 'unassigned state' and 'active_shards_percent....' never reaches 100%.

Any ideas on whether there are any commands we can run or steps or process to deal with "unassigned shards' in Palo Log Collectors?

I see Elasticsearch doco has commands that they use to deal with unassigned shards but what can we do on palo firewalls to either clear these shards or get them re-integrated?

many thanks for your advice.

I'll log a ticket with tac if there are no ideas on this. 

[Paul_Stinson]

I have since found one of the commands shows more info around unassigned.

      ".pancache" : {
        "shards" : {
          "2" : [
            {
              "state" : "STARTED",
              "primary" : true,
              "node" : "Kpu5WVqMSLSQWzCSV42-dg",
              "relocating_node" : null,
              "shard" : 2,
              "index" : ".pancache",
              "allocation_id" : {
                "id" : "Z5XfmB_CRNavbcLzpT2Xlw"
              }
            },
            {
              "state" : "UNASSIGNED",
              "primary" : false,
              "node" : null,
              "relocating_node" : null,
              "shard" : 2,
              "index" : ".pancache",
              "recovery_source" : {
                "type" : "PEER"
              },
              "unassigned_info" : {
                "reason" : "CLUSTER_RECOVERED",
                "at" : "2023-09-13T05:18:17.409Z",
                "delayed" : false,
                "allocation_status" : "no_attempt"
              }
            }
          ],
          "1" : [
            {
              "state" : "STARTED",
              "primary" : true,
              "node" : "Kpu5WVqMSLSQWzCSV42-dg",
              "relocating_node" : null,
              "shard" : 1,
              "index" : ".pancache",
              "allocation_id" : {
                "id" : "cZkPb8isTiSH-H4hHo8ojw"
              }
            },
            {
              "state" : "UNASSIGNED",
              "primary" : false,
              "node" : null,
              "relocating_node" : null,
              "shard" : 1,
              "index" : ".pancache",
              "recovery_source" : {
                "type" : "PEER"
              },
              "unassigned_info" : {
                "reason" : "CLUSTER_RECOVERED",
                "at" : "2023-09-13T05:18:17.409Z",
                "delayed" : false,
                "allocation_status" : "no_attempt"
              }
            }
          ],
          "3" : [
            {
              "state" : "STARTED",
              "primary" : true,
              "node" : "Kpu5WVqMSLSQWzCSV42-dg",
              "relocating_node" : null,
              "shard" : 3,
              "index" : ".pancache",
              "allocation_id" : {
                "id" : "4BBrampGQL--XQbqQYiUCQ"
              }
            },
            {
              "state" : "UNASSIGNED",
              "primary" : false,
              "node" : null,
              "relocating_node" : null,
              "shard" : 3,
              "index" : ".pancache",
              "recovery_source" : {
                "type" : "PEER"
              },
              "unassigned_info" : {
                "reason" : "CLUSTER_RECOVERED",
                "at" : "2023-09-13T05:18:17.409Z",
                "delayed" : false,
                "allocation_status" : "no_attempt"
              }
            }
          ],

---

---