topic Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console? in Prisma Cloud Discussions

How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

TommyHunt — Wed, 02 Nov 2022 15:53:09 GMT

So my co-worker has implemented some containerized solution and deployed it to EKS.

I used my access key to create a defender.yaml for him.

twistcli defender export kubernetes \
--address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
--user ${PRISMA_ADMIN_USER} \
--password ${PRISMA_SECRET} \
--cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR}

And she deployed it.

kubectl create -f defender.yaml

When I visit her AWS account EKS Console, I can see the cluster and the daemon but is it healthy?

Why doesn't that daemon set appear in Manage/Defenders/Daemon Sets?

How can I determine the health of the daemon set?

Or, find the root-cause of it not reporting status?

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

MDavis29 — Wed, 02 Nov 2022 16:43:21 GMT

.Tommy,

From the portal you should be able to view the defenders under Manage - Manage Defenders. Please see the screen shot below. I also noticed that you have two other DaemonSets configured but they do not have any pods running. Please check the health of the cluster, as well as run a kubectl get daemonsets --all-namespaces. You should have pods running under desired, current and ready. If you need more info on K8s daemonsets please review the links below, but it looks like the pods are not running at all. If you need help with a sample daemonset I can send you a yaml example to test.

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kubernetes

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

TommyHunt — Wed, 02 Nov 2022 18:48:45 GMT

Thanks @MDavis29 I will read the docs to learn.

So I shared how I created my defender.yaml in the original post.

I understood that defender.yaml created with twistcli is good to deploy without change.

why do you offer to help with a sample daemonset yaml file?

am i missing something? just trying to learn.

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

MDavis29 — Wed, 02 Nov 2022 20:50:56 GMT

Tommy,

My apologies if I confused you. The reason why I mentioned the sample daemonset was more for troubleshooting. With k8s is can be easier to remove some layers of complexity and use smaller deployments or DaemonSets. Looking at the screenshot you supplied of the workloads in the cluster, you can see that none of the pods for any of the DaemonSets are running. We need to identify why? this is most likely why info is not showing up in Prisma. Please run a kubectl describe on the pods in the DaemonSet and send the output, it should provide some clues as to what the problem is.

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

MDavis29 — Wed, 02 Nov 2022 21:13:16 GMT

Tommy,

Checking the pods running in the DaemonSet is the first place to start. Then perform a describe on any pods that are not running.

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

USheikh — Thu, 03 Nov 2022 01:08:13 GMT

Hello Tommy,

Can you please share yaml being used to deploy Prisma Defender?

Make sure to remove any personal identifiable information. The reason I am asking for yaml to look to see if all the relevant flags are turned on.

Regards,

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

TommyHunt — Thu, 03 Nov 2022 14:37:37 GMT

Good morning @USheikh , attaching defender.yaml stripped of PEM data. The portal doesn't allow yaml so I zipped it up.

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

TommyHunt — Thu, 03 Nov 2022 18:39:40 GMT

@MDavis29 Thanks for the advice. Ya know I can't surface ANY pods related to ANY of the daemon sets.

$ kubectl get daemonsets --all-namespaces
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system aws-node 0 0 0 0 0 <none> 42m
kube-system kube-proxy 0 0 0 0 0 <none> 42m
twistlock twistlock-defender-ds 0 0 0 0 0 <none> 18m

There are some pods running but I can't corrrelate them to the twistlock daemonset.

$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-load-balancer-controller-5fbf77b8ff-drnk5 1/1 Running 0 38m
kube-system aws-load-balancer-controller-5fbf77b8ff-lr4d6 1/1 Running 0 38m
kube-system cni-metrics-helper-dcccc58d4-465ln 1/1 Running 4 (3m1s ago) 25m
kube-system coredns-697b6f9fb8-2xnbb 1/1 Running 0 14m
kube-system coredns-697b6f9fb8-pw28w 1/1 Running 0 14m

I can't do a "kubectl describe pod" because there are no pods associated with the twistlock daemon set.

I am in the process of deploying this microservice, https://www.eksworkshop.com/beginner/050_deploy/, hopefully that gives the cluster some work. And then, I believe that the daemon sets will have pods, am i mistaken?

What do you suggest?

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

USheikh — Fri, 04 Nov 2022 18:35:46 GMT

Hello TommyHunt,

Looking at the yaml file you do not have "Nodes use Container Runtime Interface (CRI), not Docker" flag on.

Which version of EKS are you running? Is it running on daesmonset or CRI?

If it is CRI, then "Nodes use Container Runtime Interface (CRI), not Docker" flag needs to be turned on.

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

TommyHunt — Tue, 08 Nov 2022 15:19:56 GMT

Kubernetes version 1.23

I understand this cluster is intended to run Fargate Tasks on EKS so I believe that Daemon set is the correct defender type.

IF "it" means defender daemon set then yes there is a "defender" daemonset but we know this from an earlier message.

I am still learning; pronouns may create confusion rather than clarity.

Can you please use proper nouns instead of pronouns?

There exist pods.

For some reason, I understood that the cluster requires some workloads so that pods are created with the defender daemon.

So, I deployed the k8s-dashboard and an AWS Labs microservice, ecsdemo.

But notice that none of my workloads, aka deployments, have the "Ready" state, all 0s just like the daemon sets.

Due to cybersecurity policies, we don't configure clusters with public network interface, no public internet ingress.

Maybe that's why these workloads won't run.

Based on what you guys have shared.

I understand that since there is no deployment running then there is no pod running with a daemonset.

@USheikh I cannot correlate your reference to "Nodes use Container Runtime Interface (CRI), not Docker" to any element in the yaml file.

Can you provide yaml path of the element that you find unacceptable?

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

USheikh — Tue, 08 Nov 2022 19:27:31 GMT

Hello TommyHunt!

Sorry I was OOO yesterday, please allow me time to review/test the scenario, and get back to you.

Regards,

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

USheikh — Wed, 09 Nov 2022 00:46:13 GMT

Hello TommyHunt,

If I am not wrong, you are using yaml file to deploy a daemonset defender on EKS v1.23 that is running on Fargate, right?

Regards,

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

TommyHunt — Wed, 09 Nov 2022 14:44:52 GMT

Thanks @MDavis29 .

so we added a managed node group, pods are now running.

I can see audit, garbagecollector and clusterip assignment CloudWatch log events...

But Prisma Cloud discovery reports the cluster as Defended=FALSE.

I can see that the k8s service, twistlock/defender, is allocated an IP, and some garbage collection but no trace of dameonset api calls or errors. Where do i go from here?

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

TommyHunt — Wed, 09 Nov 2022 16:04:17 GMT

Classification: Public

yes

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

USheikh — Thu, 10 Nov 2022 22:47:47 GMT

Hello TommyHunt,

Based on internal discussion, in order to install defender on fargate running eks v1.23, you would need to install app-embedded defender. Here is the documentation on app-embedded defender:

https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_defender/install_app_embedded_defender

However, if you want to continue using the yaml method, please open a support case with us, so we can further investigate this and provide you with more details.

Regards,

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

TommyHunt — Wed, 16 Nov 2022 15:04:20 GMT

@Umer agreed but that is half of the solution.

Re: How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

TommyHunt — Wed, 16 Nov 2022 15:43:12 GMT

So, here is how I solved it...

first, let's talk concepts...

EKS’ Clusters may be configured with Fargate Profile(s) and/or EC2 Managed Node Groups. These two kinds of EKS compute are separated and isolated. EC2 Nodes managed by customer; Fargate nodes managed by AWS. Because of this separation, a cluster daemonset cannot not protect Fargate tasks.

IMO, those concepts are poorly communicated here... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/defender_types

Prisma Defenders may be deployed to either or both types of EKS compute in the same cluster.

EKS on Fargate – a defender for this type of compute are embedded into the image at “build” time. Each image has its own defender. To configure, see https://wiki.comp.pge.com/display/CCE/Embed+Defender+into+a+Container+for+EKS+on+Fargate. You can confirm the defenders are protecting the task by navigating to either PCC/Manage/Defenders/Manage/Defenders and/or PCC/Monitor/Runtime/App-Embedded observations.
EKS on EC2 – defenders are deployed as Kubernetes daemon sets. Given the daemonset AND a managed node group of EC2s are deployed then you can confirm one twistlock/defender pod per node. To confirm that the EKS Cluster is protected, navigate to PCC/Manage/Cloud Accounts; filter on your account number then choose the Action, Account Details, to see the Cloud Discovery results rendering which clusters are Defended or not. To configure, improve this document here https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_defender/install_cluster_container_defender

also this document too needs to be corrected... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kubernetes

Bottom-line: The resolution to my original issue was that the daemonset.yaml produced by the command that I gave was missing two options, CRI and privileged. The daemon must use Container Runtime Interface and it MUST run privileged for it to protect the other pods' containers.

The command that produces a defect-free daemonset, defender.yaml is...
```
export PRISMA_KEY_ID=SECRET
export PRISMA_SECRET=SECRET
export PRISMA_CONSOLE=https://us-east1.cloud.twistlock.com/SECRET
export PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR=https://us-east1.cloud.twistlock.com/SECRET
export PRISMA_CLOUD_COMPUTE_SVC_ADDR=us-east1.cloud.twistlock.com

twistcli defender export kubernetes \
--user ${PRISMA_KEY_ID} \
--password ${PRISMA_SECRET} \
--address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
--cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR} \
--namespace twistlock \
--privileged \
--cri
```