How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

[TommyHunt]

So my co-worker has implemented some containerized solution and deployed it to EKS.

I used my access key to create a defender.yaml for him.

twistcli defender export kubernetes \
--address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
--user ${PRISMA_ADMIN_USER} \
--password ${PRISMA_SECRET} \
--cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR}

And she deployed it.

kubectl create -f defender.yaml

When I visit her AWS account EKS Console, I can see the cluster and the daemon but is it healthy?

Why doesn't that daemon set appear in Manage/Defenders/Daemon Sets?

How can I determine the health of the daemon set?

Or, find the root-cause of it not reporting status?

[TommyHunt]

@Umer agreed but that is half of the solution.

[TommyHunt · Accepted solution]

So, here is how I solved it...

first, let's talk concepts...

EKS’ Clusters may be configured with Fargate Profile(s) and/or EC2 Managed Node Groups. These two kinds of EKS compute are separated and isolated. EC2 Nodes managed by customer; Fargate nodes managed by AWS.  Because of this separation, a cluster daemonset cannot not protect Fargate tasks.

IMO, those concepts are poorly communicated here... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/defender_ty...

Prisma Defenders may be deployed to either or both types of EKS compute in the same cluster.

1. EKS on Fargate – a defender for this type of compute are embedded into the image at “build” time. Each image has its own defender.  To configure, see https://wiki.comp.pge.com/display/CCE/Embed+Defender+into+a+Container+for+EKS+on+Fargate. You can confirm the defenders are protecting the task by navigating to either PCC/Manage/Defenders/Manage/Defenders and/or PCC/Monitor/Runtime/App-Embedded observations.

2. EKS on EC2 – defenders are deployed as Kubernetes daemon sets.  Given the daemonset AND a managed node group of EC2s are deployed then you can confirm one twistlock/defender pod per node. To confirm that the EKS Cluster is protected, navigate to PCC/Manage/Cloud Accounts; filter on your account number then choose the Action, Account Details, to see the Cloud Discovery results rendering which clusters are Defended or not.  To configure, improve this document here https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_def...

   also this document too needs to be corrected... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kub...

   Bottom-line: The resolution to my original issue was that the daemonset.yaml produced by the command that I gave was missing two options, CRI and privileged.  The daemon must use Container Runtime Interface and it MUST run privileged for it to protect the other pods' containers.

   The command that produces a defect-free daemonset, defender.yaml is...

   export PRISMA_KEY_ID=SECRET
   export PRISMA_SECRET=SECRET
   export PRISMA_CONSOLE=https://us-east1.cloud.twistlock.com/SECRET
   export PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR=https://us-east1.cloud.twistlock.com/SECRET
   export PRISMA_CLOUD_COMPUTE_SVC_ADDR=us-east1.cloud.twistlock.com
   
   twistcli defender export kubernetes \
   --user ${PRISMA_KEY_ID} \
   --password ${PRISMA_SECRET} \
   --address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
   --cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR} \
   --namespace twistlock \
   --privileged \
   --cri