How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

TommyHunt · ‎11-02-2022

So my co-worker has implemented some containerized solution and deployed it to EKS.

I used my access key to create a defender.yaml for him.

twistcli defender export kubernetes \
--address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
--user ${PRISMA_ADMIN_USER} \
--password ${PRISMA_SECRET} \
--cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR}

And she deployed it.

kubectl create -f defender.yaml

When I visit her AWS account EKS Console, I can see the cluster and the daemon but is it healthy?

Why doesn't that daemon set appear in Manage/Defenders/Daemon Sets?

How can I determine the health of the daemon set?

Or, find the root-cause of it not reporting status?

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

TommyHunt · ‎11-16-2022

So, here is how I solved it...

first, let's talk concepts...

EKS’ Clusters may be configured with Fargate Profile(s) and/or EC2 Managed Node Groups. These two kinds of EKS compute are separated and isolated. EC2 Nodes managed by customer; Fargate nodes managed by AWS. Because of this separation, a cluster daemonset cannot not protect Fargate tasks.

IMO, those concepts are poorly communicated here... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/defender_ty...

Prisma Defenders may be deployed to either or both types of EKS compute in the same cluster.

EKS on Fargate – a defender for this type of compute are embedded into the image at “build” time. Each image has its own defender. To configure, see https://wiki.comp.pge.com/display/CCE/Embed+Defender+into+a+Container+for+EKS+on+Fargate. You can confirm the defenders are protecting the task by navigating to either PCC/Manage/Defenders/Manage/Defenders and/or PCC/Monitor/Runtime/App-Embedded observations.
EKS on EC2 – defenders are deployed as Kubernetes daemon sets. Given the daemonset AND a managed node group of EC2s are deployed then you can confirm one twistlock/defender pod per node. To confirm that the EKS Cluster is protected, navigate to PCC/Manage/Cloud Accounts; filter on your account number then choose the Action, Account Details, to see the Cloud Discovery results rendering which clusters are Defended or not. To configure, improve this document here https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_def...

also this document too needs to be corrected... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kub...

Bottom-line: The resolution to my original issue was that the daemonset.yaml produced by the command that I gave was missing two options, CRI and privileged. The daemon must use Container Runtime Interface and it MUST run privileged for it to protect the other pods' containers.

The command that produces a defect-free daemonset, defender.yaml is...
```
export PRISMA_KEY_ID=SECRET
export PRISMA_SECRET=SECRET
export PRISMA_CONSOLE=https://us-east1.cloud.twistlock.com/SECRET
export PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR=https://us-east1.cloud.twistlock.com/SECRET
export PRISMA_CLOUD_COMPUTE_SVC_ADDR=us-east1.cloud.twistlock.com

twistcli defender export kubernetes \
--user ${PRISMA_KEY_ID} \
--password ${PRISMA_SECRET} \
--address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
--cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR} \
--namespace twistlock \
--privileged \
--cri
```

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

View solution in original post

MDavis29 · ‎11-02-2022

.Tommy,

From the portal you should be able to view the defenders under Manage - Manage Defenders. Please see the screen shot below. I also noticed that you have two other DaemonSets configured but they do not have any pods running. Please check the health of the cluster, as well as run a kubectl get daemonsets --all-namespaces. You should have pods running under desired, current and ready. If you need more info on K8s daemonsets please review the links below, but it looks like the pods are not running at all. If you need help with a sample daemonset I can send you a yaml example to test.

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kub...

TommyHunt · ‎11-02-2022

Thanks @MDavis29 I will read the docs to learn.

So I shared how I created my defender.yaml in the original post.

I understood that defender.yaml created with twistcli is good to deploy without change.

why do you offer to help with a sample daemonset yaml file?

am i missing something? just trying to learn.

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

MDavis29 · ‎11-02-2022

Tommy,

My apologies if I confused you. The reason why I mentioned the sample daemonset was more for troubleshooting. With k8s is can be easier to remove some layers of complexity and use smaller deployments or DaemonSets. Looking at the screenshot you supplied of the workloads in the cluster, you can see that none of the pods for any of the DaemonSets are running. We need to identify why? this is most likely why info is not showing up in Prisma. Please run a kubectl describe on the pods in the DaemonSet and send the output, it should provide some clues as to what the problem is.

MDavis29 · ‎11-02-2022

Tommy,

Checking the pods running in the DaemonSet is the first place to start. Then perform a describe on any pods that are not running.

How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

Show your appreciation!