How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

How to triage an EKS Cluster with Prisma Defender daemon set NOT appearing in the console?

L3 Networker

So my co-worker has implemented some containerized solution and deployed it to EKS.

I used my access key to create a defender.yaml for him.

twistcli defender export kubernetes \
--address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
--user ${PRISMA_ADMIN_USER} \
--password ${PRISMA_SECRET} \
--cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR}

And she deployed it.

kubectl create -f defender.yaml

When I visit her AWS account EKS Console, I can see the cluster and the daemon but is it healthy?

TommyHunt_0-1667403289928.png

Why doesn't that daemon set appear in Manage/Defenders/Daemon Sets?

TommyHunt_1-1667404246937.png

 

How can I determine the health of the daemon set?

Or, find the root-cause of it not reporting status?

 

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org
1 accepted solution

Accepted Solutions

L3 Networker

So, here is how I solved it...

 

first, let's talk concepts...

 

EKS’ Clusters may be configured with Fargate Profile(s) and/or EC2 Managed Node Groups. These two kinds of EKS compute are separated and isolated. EC2 Nodes managed by customer; Fargate nodes managed by AWS.  Because of this separation, a cluster daemonset cannot not protect Fargate tasks.

 

IMO, those concepts are poorly communicated here... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/defender_ty...

 

Prisma Defenders may be deployed to either or both types of EKS compute in the same cluster.

  1. EKS on Fargate – a defender for this type of compute are embedded into the image at “build” time. Each image has its own defender.  To configure, see https://wiki.comp.pge.com/display/CCE/Embed+Defender+into+a+Container+for+EKS+on+Fargate. You can confirm the defenders are protecting the task by navigating to either PCC/Manage/Defenders/Manage/Defenders and/or PCC/Monitor/Runtime/App-Embedded observations.

  2. EKS on EC2 – defenders are deployed as Kubernetes daemon sets.  Given the daemonset AND a managed node group of EC2s are deployed then you can confirm one twistlock/defender pod per node. To confirm that the EKS Cluster is protected, navigate to PCC/Manage/Cloud Accounts; filter on your account number then choose the Action, Account Details, to see the Cloud Discovery results rendering which clusters are Defended or not.  To configure, improve this document here https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_def...

    also this document too needs to be corrected... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kub...

     

    Bottom-line: The resolution to my original issue was that the daemonset.yaml produced by the command that I gave was missing two options, CRI and privileged.  The daemon must use Container Runtime Interface and it MUST run privileged for it to protect the other pods' containers.

    The command that produces a defect-free daemonset, defender.yaml is...

    export PRISMA_KEY_ID=SECRET
    export PRISMA_SECRET=SECRET
    export PRISMA_CONSOLE=https://us-east1.cloud.twistlock.com/SECRET
    export PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR=https://us-east1.cloud.twistlock.com/SECRET
    export PRISMA_CLOUD_COMPUTE_SVC_ADDR=us-east1.cloud.twistlock.com

    twistcli defender export kubernetes \
    --user ${PRISMA_KEY_ID} \
    --password ${PRISMA_SECRET} \
    --address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
    --cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR} \
    --namespace twistlock \
    --privileged \
    --cri

     

     

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

View solution in original post

16 REPLIES 16

L3 Networker

.Tommy,

From the portal you should be able to view the defenders under Manage - Manage Defenders.  Please see the screen shot below. I also noticed that you have two other DaemonSets configured but they do not have any pods running.  Please check the health of the cluster, as well as run a kubectl get daemonsets --all-namespaces.  You should have pods running under desired, current and ready.  If you need more info on K8s daemonsets please review the links below, but it looks like the pods are not running at all.  If you need help with a sample daemonset I can send you a yaml example to test.

 

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kub...

 

 

MDavis29_0-1667405330156.png

 

L3 Networker

Thanks @MDavis29 I will read the docs to learn.

So I shared how I created my defender.yaml in the original post.

I understood that defender.yaml created with twistcli is good to deploy without change.

why do you offer to help with a sample daemonset yaml file?

am i missing something?  just trying to learn.

 

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

Tommy,

My apologies if I confused you.  The reason why I mentioned the sample daemonset was more for troubleshooting.  With k8s is can be easier to remove some layers of complexity and use smaller deployments or DaemonSets.   Looking at the screenshot you supplied of the workloads in the cluster, you can see that none of the pods for any of the DaemonSets are running.  We need to identify why? this is most likely why info is not showing up in Prisma.  Please run a kubectl describe on the pods in the DaemonSet and send the output, it should provide some clues as to what the problem is.

Tommy,

Checking the pods running in the DaemonSet is the first place to start.  Then perform a describe on any pods that are not running.

MDavis29_0-1667423490068.png

 

L3 Networker

Hello Tommy,

 

Can you please share yaml being used to deploy Prisma Defender?

 

Make sure to remove any personal identifiable information. The reason I am asking for yaml to look to see if all the relevant flags are turned on.

 

Regards, 

Umer Sheikh | Technical Support Engineer - Prisma Cloud Compute | PCCSE, AWS - Associate Architect

L3 Networker

Good morning @USheikh ,  attaching defender.yaml stripped of PEM data.  The portal doesn't allow yaml so I zipped it up.

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

@MDavis29 Thanks for the advice.  Ya know I can't surface ANY pods related to ANY of the daemon sets.

$ kubectl get daemonsets --all-namespaces
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system aws-node 0 0 0 0 0 <none> 42m
kube-system kube-proxy 0 0 0 0 0 <none> 42m
twistlock twistlock-defender-ds 0 0 0 0 0 <none> 18m

There are some pods running but I can't corrrelate them to the twistlock daemonset.

$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-load-balancer-controller-5fbf77b8ff-drnk5 1/1 Running 0 38m
kube-system aws-load-balancer-controller-5fbf77b8ff-lr4d6 1/1 Running 0 38m
kube-system cni-metrics-helper-dcccc58d4-465ln 1/1 Running 4 (3m1s ago) 25m
kube-system coredns-697b6f9fb8-2xnbb 1/1 Running 0 14m
kube-system coredns-697b6f9fb8-pw28w 1/1 Running 0 14m

I can't do a "kubectl describe pod" because there are no pods associated with the twistlock daemon set.

I am in the process of deploying this microservice, https://www.eksworkshop.com/beginner/050_deploy/, hopefully that gives the cluster some work.  And then, I believe that the daemon sets will have pods, am i mistaken?

What do you suggest?

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

Hello TommyHunt,

 

Looking at the yaml file you do not have "Nodes use Container Runtime Interface (CRI), not Docker" flag on.

 

Which version of EKS are you running? Is it running on daesmonset or CRI?

 

If it is CRI, then "Nodes use Container Runtime Interface (CRI), not Docker" flag needs to be turned on. 

Umer Sheikh | Technical Support Engineer - Prisma Cloud Compute | PCCSE, AWS - Associate Architect

Kubernetes version 1.23

I understand this cluster is intended to run Fargate Tasks on EKS so I believe that Daemon set is the correct defender type.

IF "it" means defender daemon set then yes there is a "defender" daemonset but we know this from an earlier message.

TommyHunt_1-1667573088732.png

I am still learning; pronouns may create confusion rather than clarity.

Can you please use proper nouns instead of pronouns?

 

There exist pods.

TommyHunt_0-1667572829051.png

 

For some reason, I understood that the cluster requires some workloads so that pods are created with the defender daemon.

TommyHunt_2-1667573414098.png

So, I deployed the k8s-dashboard and an AWS Labs microservice, ecsdemo.

But notice that none of my workloads, aka deployments, have the "Ready" state, all 0s just like the daemon sets.

Due to cybersecurity policies, we don't configure clusters with public network interface, no public internet ingress.

Maybe that's why these workloads won't run.

 

Based on what you guys have shared.

I understand that since there is no deployment running then there is no pod running with a daemonset.

 

@USheikh I cannot correlate your reference to "Nodes use Container Runtime Interface (CRI), not Docker" to any element in the yaml file.

Can you provide yaml path of the element that you find unacceptable?

 

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

Hello TommyHunt!

 

Sorry I was OOO yesterday, please allow me time to review/test the scenario, and get back to you.

 

Regards, 

Umer Sheikh | Technical Support Engineer - Prisma Cloud Compute | PCCSE, AWS - Associate Architect

Hello TommyHunt,

 

If I am not wrong, you are using yaml file to deploy a daemonset defender on EKS v1.23 that is running on Fargate, right?

 

Regards,

Umer Sheikh | Technical Support Engineer - Prisma Cloud Compute | PCCSE, AWS - Associate Architect

Thanks @MDavis29 .

so we added a managed node group, pods are now running.

I can see audit, garbagecollector and clusterip assignment CloudWatch log events...

TommyHunt_1-1668004779427.png

 

But Prisma Cloud discovery reports the cluster as Defended=FALSE.

TommyHunt_0-1668004338205.png

 

I can see that the k8s service, twistlock/defender, is allocated an IP, and some garbage collection but no trace of dameonset api calls or errors.  Where do i go from here?

Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

Classification: Public


yes
Tommy Hunt AWS-CSA, Java-CEA, PMP, SAFe Program Consultant
thunt@citrusoft.org
https://www.citrusoft.org

Hello TommyHunt,

 

Based on internal discussion, in order to install defender on fargate running eks v1.23, you would need to install app-embedded defender. Here is the documentation on app-embedded defender:

https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_def...

 

However, if you want to continue using the yaml method, please open a support case with us, so we can further investigate this and provide you with more details. 

 

Regards,

Umer Sheikh | Technical Support Engineer - Prisma Cloud Compute | PCCSE, AWS - Associate Architect
  • 1 accepted solution
  • 5789 Views
  • 16 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!