How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

L3 Networker

So my co-worker has implemented some containerized solution and deployed it to EKS.

I used my access key to create a defender.yaml for him.

twistcli defender export kubernetes \
--address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
--user ${PRISMA_ADMIN_USER} \
--password ${PRISMA_SECRET} \
--cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR}

And she deployed it.

kubectl create -f defender.yaml

When I visit her AWS account EKS Console, I can see the cluster and the daemon but is it healthy?

TommyHunt_0-1667403289928.png

Why doesn't that daemon set appear in Manage/Defenders/Daemon Sets?

TommyHunt_1-1667404246937.png

 

How can I determine the health of the daemon set?

Or, find the root-cause of it not reporting status?

 

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org
1 ACCEPTED SOLUTION

Accepted Solutions

L3 Networker

So, here is how I solved it...

 

first, let's talk concepts...

 

EKS’ Clusters may be configured with Fargate Profile(s) and/or EC2 Managed Node Groups. These two kinds of EKS compute are separated and isolated. EC2 Nodes managed by customer; Fargate nodes managed by AWS.  Because of this separation, a cluster daemonset cannot not protect Fargate tasks.

 

IMO, those concepts are poorly communicated here... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/defender_ty...

 

Prisma Defenders may be deployed to either or both types of EKS compute in the same cluster.

  1. EKS on Fargate – a defender for this type of compute are embedded into the image at “build” time. Each image has its own defender.  To configure, see https://wiki.comp.pge.com/display/CCE/Embed+Defender+into+a+Container+for+EKS+on+Fargate. You can confirm the defenders are protecting the task by navigating to either PCC/Manage/Defenders/Manage/Defenders and/or PCC/Monitor/Runtime/App-Embedded observations.

  2. EKS on EC2 – defenders are deployed as Kubernetes daemon sets.  Given the daemonset AND a managed node group of EC2s are deployed then you can confirm one twistlock/defender pod per node. To confirm that the EKS Cluster is protected, navigate to PCC/Manage/Cloud Accounts; filter on your account number then choose the Action, Account Details, to see the Cloud Discovery results rendering which clusters are Defended or not.  To configure, improve this document here https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_def...

    also this document too needs to be corrected... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kub...

     

    Bottom-line: The resolution to my original issue was that the daemonset.yaml produced by the command that I gave was missing two options, CRI and privileged.  The daemon must use Container Runtime Interface and it MUST run privileged for it to protect the other pods' containers.

    The command that produces a defect-free daemonset, defender.yaml is...

    export PRISMA_KEY_ID=SECRET
    export PRISMA_SECRET=SECRET
    export PRISMA_CONSOLE=https://us-east1.cloud.twistlock.com/SECRET
    export PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR=https://us-east1.cloud.twistlock.com/SECRET
    export PRISMA_CLOUD_COMPUTE_SVC_ADDR=us-east1.cloud.twistlock.com

    twistcli defender export kubernetes \
    --user ${PRISMA_KEY_ID} \
    --password ${PRISMA_SECRET} \
    --address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
    --cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR} \
    --namespace twistlock \
    --privileged \
    --cri

     

     

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

View solution in original post

16 REPLIES 16

L2 Linker

.Tommy,

From the portal you should be able to view the defenders under Manage - Manage Defenders.  Please see the screen shot below. I also noticed that you have two other DaemonSets configured but they do not have any pods running.  Please check the health of the cluster, as well as run a kubectl get daemonsets --all-namespaces.  You should have pods running under desired, current and ready.  If you need more info on K8s daemonsets please review the links below, but it looks like the pods are not running at all.  If you need help with a sample daemonset I can send you a yaml example to test.

 

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kub...

 

 

MDavis29_0-1667405330156.png

 

L3 Networker

Thanks @MDavis29 I will read the docs to learn.

So I shared how I created my defender.yaml in the original post.

I understood that defender.yaml created with twistcli is good to deploy without change.

why do you offer to help with a sample daemonset yaml file?

am i missing something?  just trying to learn.

 

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

Tommy,

My apologies if I confused you.  The reason why I mentioned the sample daemonset was more for troubleshooting.  With k8s is can be easier to remove some layers of complexity and use smaller deployments or DaemonSets.   Looking at the screenshot you supplied of the workloads in the cluster, you can see that none of the pods for any of the DaemonSets are running.  We need to identify why? this is most likely why info is not showing up in Prisma.  Please run a kubectl describe on the pods in the DaemonSet and send the output, it should provide some clues as to what the problem is.

Tommy,

Checking the pods running in the DaemonSet is the first place to start.  Then perform a describe on any pods that are not running.

MDavis29_0-1667423490068.png

 

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!