How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

[TommyHunt]

So my co-worker has implemented some containerized solution and deployed it to EKS.

I used my access key to create a defender.yaml for him.

twistcli defender export kubernetes \
--address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
--user ${PRISMA_ADMIN_USER} \
--password ${PRISMA_SECRET} \
--cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR}

And she deployed it.

kubectl create -f defender.yaml

When I visit her AWS account EKS Console, I can see the cluster and the daemon but is it healthy?

Why doesn't that daemon set appear in Manage/Defenders/Daemon Sets?

How can I determine the health of the daemon set?

Or, find the root-cause of it not reporting status?

[MDavis29]

.Tommy,

From the portal you should be able to view the defenders under Manage - Manage Defenders.  Please see the screen shot below. I also noticed that you have two other DaemonSets configured but they do not have any pods running.  Please check the health of the cluster, as well as run a kubectl get daemonsets --all-namespaces.  You should have pods running under desired, current and ready.  If you need more info on K8s daemonsets please review the links below, but it looks like the pods are not running at all.  If you need help with a sample daemonset I can send you a yaml example to test.

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kub...

[TommyHunt]

Thanks @MDavis29 I will read the docs to learn.

So I shared how I created my defender.yaml in the original post.

I understood that defender.yaml created with twistcli is good to deploy without change.

why do you offer to help with a sample daemonset yaml file?

am i missing something?  just trying to learn.

[MDavis29]

Tommy,

My apologies if I confused you.  The reason why I mentioned the sample daemonset was more for troubleshooting.  With k8s is can be easier to remove some layers of complexity and use smaller deployments or DaemonSets.   Looking at the screenshot you supplied of the workloads in the cluster, you can see that none of the pods for any of the DaemonSets are running.  We need to identify why? this is most likely why info is not showing up in Prisma.  Please run a kubectl describe on the pods in the DaemonSet and send the output, it should provide some clues as to what the problem is.

[MDavis29]

Tommy,

Checking the pods running in the DaemonSet is the first place to start.  Then perform a describe on any pods that are not running.

[USheikh]

Hello Tommy,

Can you please share yaml being used to deploy Prisma Defender?

Make sure to remove any personal identifiable information. The reason I am asking for yaml to look to see if all the relevant flags are turned on.

Regards, 

[TommyHunt]

Good morning @USheikh ,  attaching defender.yaml stripped of PEM data.  The portal doesn't allow yaml so I zipped it up.

[TommyHunt]

@MDavis29 Thanks for the advice.  Ya know I can't surface ANY pods related to ANY of the daemon sets.

$ kubectl get daemonsets --all-namespaces
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system aws-node 0 0 0 0 0 <none> 42m
kube-system kube-proxy 0 0 0 0 0 <none> 42m
twistlock twistlock-defender-ds 0 0 0 0 0 <none> 18m

There are some pods running but I can't corrrelate them to the twistlock daemonset.

$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-load-balancer-controller-5fbf77b8ff-drnk5 1/1 Running 0 38m
kube-system aws-load-balancer-controller-5fbf77b8ff-lr4d6 1/1 Running 0 38m
kube-system cni-metrics-helper-dcccc58d4-465ln 1/1 Running 4 (3m1s ago) 25m
kube-system coredns-697b6f9fb8-2xnbb 1/1 Running 0 14m
kube-system coredns-697b6f9fb8-pw28w 1/1 Running 0 14m

I can't do a "kubectl describe pod" because there are no pods associated with the twistlock daemon set.

I am in the process of deploying this microservice, https://www.eksworkshop.com/beginner/050_deploy/, hopefully that gives the cluster some work.  And then, I believe that the daemon sets will have pods, am i mistaken?

What do you suggest?

[USheikh]

Hello TommyHunt,

Looking at the yaml file you do not have "Nodes use Container Runtime Interface (CRI), not Docker" flag on.

Which version of EKS are you running? Is it running on daesmonset or CRI?

If it is CRI, then "Nodes use Container Runtime Interface (CRI), not Docker" flag needs to be turned on. 

[TommyHunt]

Kubernetes version 1.23

I understand this cluster is intended to run Fargate Tasks on EKS so I believe that Daemon set is the correct defender type.

IF "it" means defender daemon set then yes there is a "defender" daemonset but we know this from an earlier message.

I am still learning; pronouns may create confusion rather than clarity.

Can you please use proper nouns instead of pronouns?

There exist pods.

For some reason, I understood that the cluster requires some workloads so that pods are created with the defender daemon.

So, I deployed the k8s-dashboard and an AWS Labs microservice, ecsdemo.

But notice that none of my workloads, aka deployments, have the "Ready" state, all 0s just like the daemon sets.

Due to cybersecurity policies, we don't configure clusters with public network interface, no public internet ingress.

Maybe that's why these workloads won't run.

Based on what you guys have shared.

I understand that since there is no deployment running then there is no pod running with a daemonset.

@USheikh I cannot correlate your reference to "Nodes use Container Runtime Interface (CRI), not Docker" to any element in the yaml file.

Can you provide yaml path of the element that you find unacceptable?

[USheikh]

Hello TommyHunt!

Sorry I was OOO yesterday, please allow me time to review/test the scenario, and get back to you.

Regards, 

[USheikh]

Hello TommyHunt,

If I am not wrong, you are using yaml file to deploy a daemonset defender on EKS v1.23 that is running on Fargate, right?

Regards,

[TommyHunt]

Thanks @MDavis29 .

so we added a managed node group, pods are now running.

I can see audit, garbagecollector and clusterip assignment CloudWatch log events...

But Prisma Cloud discovery reports the cluster as Defended=FALSE.

I can see that the k8s service, twistlock/defender, is allocated an IP, and some garbage collection but no trace of dameonset api calls or errors.  Where do i go from here?

[TommyHunt]

Classification: Public


yes

[USheikh]

Hello TommyHunt,

Based on internal discussion, in order to install defender on fargate running eks v1.23, you would need to install app-embedded defender. Here is the documentation on app-embedded defender:

https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_def...

However, if you want to continue using the yaml method, please open a support case with us, so we can further investigate this and provide you with more details. 

Regards,